I've experienced this issue too. I have had a game with just the Iridium (the Trading Race) left on the map. I couldn't invade them for like the entire game. I ended up winning only because of influence and turning them that way (I had killed all the other opponents about 30 turns earlier).
Another game I was in, I couldn't invade two separate races. One of them was very powerful and in a good position, but with practically no defenses. The best play would have been to attack them / take over their position, but I couldn't invade them at all. Definitely annoying!