At present the only penalty for violating the AI space is a minor diplomatic penalty. As for you, the only reaction allowed is war. There is no way, in the beta, to warn them off.

It makes little sense to colonize deep into another race borders since the planet will flip in 10 turns. There is no reason they should have to declare war. They are not the offended party, you are. The game considers all uncolonized planets as fair game for anyone. In future releases the game will probably allow terse complaints to the AI. As for now war is the only option for you.