Really, 90% play BELOW normal? Or is that how many of those that finished a game? Are those the folks that have 200+ games in their inventory and only play about 30 hours and move on?
As to changes that would effect lower levels, make certain subroutines only kick in at normal or whatever level you deem it to be.
That the AI doesn't have local focus explains a lot about why it is doing some weird things. That alone would go a long way. However, as you said, that is the hardest.
Right now from watching soaks, the AI sets too many rally points. It rarely seems to remove them. The AIs were battling over 1 planet and it swapped hands several times. There was a rally point next to that planet, so when the enemy took it, the ships were being fed one at a time into it. In the meantime there were several 2 ship fleets in the area and there was plenty of logistics to make them larger. It looks like were essentially sentry/guarding the area. I was thinking if I were controlling my faction, I would have combined those mini-fleets and wiped out the area. Then after invading the planet moved forward.
Does the AI only have one target location when at war?
Personally, I rarely guard transports. I secure the area around the planets, move my transport to a position at its maximum distance it can reach the planet in one turn. Then invade the next turn. My transports don't have sensors and I only have enough range to get where I want to go. Everything else goes to engines. I can often just speed my way out if there is trouble.
May I suggest that ships are give a "global" target goal of take "planet X" or whatever, but as the ships move to that area, they combine with nearby ships/fleets as logistics allow. Also, they will look nearby before they move to see if there is an enemy. If there is, then THAT becomes their temporary goal before continuing on.
If enemy fleet < x hexes and (my fleet strength/enemy fleet strength) > .8 then attack enemy fleet. This would need to take account if it can reach that fleet in this turn or has more speed than the enemy fleet, otherwise it would chase it for no good reason. Also attack fleets with transports whenever possible.
AI ships need more priority on speed of the ships and as was mentioned take speed into account when assembling fleets.
Glad to see you are looking at getting the AI to prepare for war first. This is my usual MO.
I almost always scout an enemy before I declare war. That way I know where their ships are, what type of ships and where they are the weakest. I then target 2-3 juicy targets, move my fleets and transports up close, but not in their LOS. I have a couple of scouts to inform me if something is coming that might see my fleets. After assembly, I move into position. If they get angry, who cares? THEN I DoW and start rolling them. Ideally losing some good planets will make their cash per turn negative, hurt approval and thus affect manufacturing and research. All within a few turns.
As marigoldran said (although waaaaaaaaaaaaaaay too much. Multiple posts saying essentially the same thing is annoying) the AI needs to be be sneaky. Hidden modifiers and false information on the +/- relations would help. Caesar in CIV V was never to be trusted. The problem was, I never know when he has troops coming by my border if he is going around or going to try a sneak attack. That forces me to tie down troops by him just in case.
The AI needs to take a "crabs in a bucket" approach. By that I mean if a faction is starting to pull away, they gang up and drag that faction back down. This has to be carefully balanced or perhaps an option in the game setup. Those who want to have a peaceful build and make alliances, etc. might not like it. Perhaps some factions will do this only to certain ideologies. I think that is some of the idea of ideologies, but pragmatic and especially malevolent should only be a temporary thing. When it comes to "You are weak" and right next to me, you are dead.
Really the diplomacy screen gives the player way too much information. You should only know what you have seen. I shouldn't know all their planets, ships, star bases, etc. unless my scouting shows it at some point. Knowing ship composition (weapons,defense, etc) let's me know what kind of ships I need to wipe them out. That gives the player a huge leg up.