If the AI does not react when you build/extend your influence and start flipping its colonies, than it becomes too easy. Yes, it's one way to win, but the moment your influence becomes really strong, there's no way the AI can compete, even with massive build of cultural centers, there's no turning back.
And for a civiization like the Drengin, wich are better at war than spreading influence positively, if they were to simply stand by and fight back only with cultural buildings, they would lose 100% of the time.
Imho, the AI should react with war if you threaten its culture to the point of making it lose the game. Influence let you avoid some conquest, but not all of it, and it shouldn't be that way. As I said, if you play benevolent, it's way to easy to win a cultural war. The moment you have the perk that lets you flip planets&bases to your side, if timed properly, it simple spells the doom of your ennemies.