I find that it's more than worth it to build a Deep Core Mine or Space Elevator (occasionally both) on EVERY planet. They generally boost overall productivity of the planet by 50%, which is pretty substantial for between a 5 and 10 turn investment on the planet. Starports can be useful, but much later in the game unless you've got an luckily well-placed Tile bonus for them to use.
Both Computer Core and Galactic Mainframe are VERY much worth it, but they also have to be carefully placed, because their worth is extremely dependent on adjacency bonuses.
The various entry-level economic buildings are likewise dependent on proper adjacency placement; in general, I find myself not building them until about turn 50 or so, and then spend 50 turns getting a couple of worlds up to printing money. Similar for Factory - I use it mostly to boost the Space Elevator or Deep Core Mine's production, and as the first step to bigger manufacturing improvements. But you don't just scatter them around a lot - they're very situational.
The key to this game is world specialization - figure out what a given world would do best at, then cover at least 50% of all the tiles in that type of building. Adjacency matters a great deal here.
Yes, Factory is underwhelming and probably should be given a buff, as does the Xeno Research. I like the idea of giving them a small flat bonus as their primary trait. The others are just fine as they are.