This is part of the game, called "influence". Your culture can eventually cause a planet to rebel and switch sides.
INFLUENCE is therefore countered with INFLUENCE. As well, planet RESISTANCE will help your culture resist theirs. Its very crucial in this game. Colored hexes around your planets are Zones of Influence. Each planet produces influence, and going unchecked by an opponent (or lack thereof) will cause ZOI borders to change. When a planet is fully surrounded by another's influence, it begins to rebel, which again can be slowed by planetary resistance modifiers.
The game is more than just science, production, and money - it lets you play and even win with the influence game. If you are playing passive and defensively, you certainly need to worry about influence, as this is the "peaceful killer".