So I made some changes and ran a soak on insane/abundant (with base 1600 around ~1000 habitables), spiral galaxy, 40 players, uncommon pirates, normal difficulty. Changes were:
* Made 'startstrats' last 50 turns
* Moved all the non-endgame and startgame strats apart from the 'expansionstrats' require turn 100 to start, so at level 50 they chose expand.
* Removed the tech age requirements from strats
* changed the governors to concentrate on their speicalism during fillers.
* Removed loads of useless defense, influence, tourism and wealth buildings from industrial and research planet governors
* removed 'growth' world governors completely
* Changed blueprints to be less obsessed with life support, move keen on engines
* moved all AI strategy spending priorities to be more industrially focused.
* tweaked the faction defs to give more races 'expansionist' (though this doesn't actually seem to do much).
The effect of this was enormous. I ran the soak to turn 70ish.
The first thing I noticed was that the Thalan and Terran factions no longer sucked. The Terrans were about the 3rd largest empire, comparable in size to the Krynn and Altarians but quite a way behind the Drengin. The Thalans were a ways behind, but that was mostly due to their starting position; they still had 15+ planets (in my previous game with the same settings but no AI adjustments, they had 6 by turn 150). This was a game with 5-6 big empires and 20-odd medium ones, as opposed to 38 tiny empires and the Krynn.
In terms of planets, the AI was a lot more focused. It was generally possible to see what an AI planet was for. Average industrial production and research were way up, and that effect was felt across the board - even the most viable non-modded planets are only slightly above where the average planet now sits. With no ridiculous 'Growth' worlds dragging their approval down, the AI was able to concentrate on useful, productive planets.
The empires were much wider, and expanded much more rapidly - due almost entirely to the blueprint changes imo, as now they stack engines more effectively, and make good use of sensors (which the AI does need on normal). The also had more and better starbases.
The level of military was far more subdued, however. This wasn't a problem with few pirates and such a large map, but may cause issues in smaller or more pirate-intensive games. Given the AI's vastly better productivity, I expect them to militarize rather swiftly post-turn 100, but they're sticking rather strictly to the colony-spam strategy for now.
Sadly, there doesn't seem to be a means of telling the AI to use different strategies based on things like "do you have close neighbours", "are pirates threatening our colony ships", etc. Its strategic choices are largely not based on what's happening in-game.
For the next set of experiments, I'll try adding in more strategies. If that's successful, then I'd propose we do the following:
Create a number of strategies, each lasting roughly 10 turns and divided by personality and ideology. There should be 4-5 different options for each 10-turn chunk of the game. These will vary between going all-out research for 10 turns, to simple outright colony spam for 10 turns, to dedicating itself to building a small combat fleet.
This will last until around turn 50, where we switch to 20-turn chunks on the same pattern; then at turn 150 we do 50-turn chunks up to turn 300ish, and then put them into endgame mod.
This would require around 80 different strategies to be written, with the AI switching between them rapidly enough that it never gets 'stuck' with a strategy which is truly terrible for it's given situation for too long.