Spies on buildings are supposed to shut that building down until removed by sacrificing one of your own spies. I seldom had enemy spies on my planets, but when I had they worked.

If you have own spies available then you can click on a building with an enemy spy and should get the option to remove them.

Otherwise "unoccupied" spies (that are not placed on planets or are set to spy on enemy factions) should work as counter spies, supposedly reducing the chance that enemies steal your techs, but since there is no feedback whatsover we have no chance of knowing whether that has any effect. It would be better if we would not only get a message when a tech was stolen from us, but also when our spies prevented just that.