There was some sort of bug that was making influence calculations wrong. From the 5.2 patch notes: "Fixed an issue where influence would be incorrectly calculated"
I also noticed that there are some diplomatic penalties planed for taking over/threatening planets with culture. I think they just haven't been fully implemented yet.
Hopefully the combination will make things more balanced.
-<DiplomaticModifier>
<UniqueID>CloseToInfluenceFlip</UniqueID>
<RelationEffect>-2</RelationEffect>
-<AIPersonalityModifier>
<PersonalityTrait>Expansionist</PersonalityTrait>
<WeightModifier>1.5</WeightModifier>
</AIPersonalityModifier>
-<AIPersonalityModifier>
<PersonalityTrait>Spiritual</PersonalityTrait>
<WeightModifier>1.5</WeightModifier>
</AIPersonalityModifier>
-<AIPersonalityModifier>
<PersonalityTrait>Xenophobic</PersonalityTrait>
<WeightModifier>2</WeightModifier>
</AIPersonalityModifier>
<Description>CloseToInfluenceFlip_Dec</Description>
</DiplomaticModifier>
----------------------------------------------------
-<DiplomaticModifier>
<UniqueID>SuccessfulInfluenceFlip</UniqueID>
<RelationEffect>-3</RelationEffect>
-<AIPersonalityModifier>
<PersonalityTrait>Aggressive</PersonalityTrait>
<WeightModifier>1.5</WeightModifier>
</AIPersonalityModifier>
-<AIPersonalityModifier>
<PersonalityTrait>Expansionist</PersonalityTrait>
<WeightModifier>1.5</WeightModifier>
</AIPersonalityModifier>
-<AIPersonalityModifier>
<PersonalityTrait>Spiritual</PersonalityTrait>
<WeightModifier>1.5</WeightModifier>
</AIPersonalityModifier>
-<AIPersonalityModifier>
<PersonalityTrait>Xenophobic</PersonalityTrait>
<WeightModifier>2</WeightModifier>
</AIPersonalityModifier>
<Query>false</Query>
<Duration>75</Duration>
<Description>SuccessfulInfluenceFlip_Dec</Description>
</DiplomaticModifier>