The problem here is simply the AI's inability to set production at a planetary level. It will never be able to keep up with the player late-game, even with the huge bonuses on offer from higher difficulties.
Say both you and the AI have 3 planets, each with 100 production. You give each 900%'s worth of buildings, one specialized on econ, 1 on industry and 1 on research.
The player specializes each planet to use 100% on the correct output type. So you have 1 planet producing 1000 industry, 1 producing 1000 research, and 1 producing 1000 econ.
The AI uses the empire-wide wheel, and set itself to 33% on each. The industry planet produces 330 industry, 33 econ and 33 research; the econ planet produces 330 econ, 33 industry and 33 research; the research planet produces 330 research, 33 industry and 33 econ. It's total production from the 3 worlds combined is 396 manu, 396 econ and 396 research.
If it tries other settings, then the results are not improved. If it goes for 50% manu and 25% elsewhere, we get the industry world on 500 manu, 25 econ and 25 research - a nice increase for industrial output. But the econ world is now producing just 250 econ, 50 manu and 25 research; the research world is only producing 250 research, 50 manu and 25 econ. It's total output is now 600 manufacturing, 300 econ and 300 research. It's still stuck on 1200 output, where the player is pumping out 3000 total output.
You're producing more than 2.5 times as much output as the AI from the same buildings, planets and population. Even with the tripled production that the AI enjoys on Godlike, it is only just able to match the basic specialization that the average player manages using the wheel alone (even without any advantages from optimal improvement placement etc). This means it starts falling behind the moment the player begins specializing planets.
This is actually exacerbated by some of the other features - so, for example, the AI avoids going into debt if at all possible, by setting the econ slider to match outgoings (when set to breakeven, or when reaching the 'breakeven' point on a spendtilbroke setting). This is entirely useless on manufacturing worlds, which would be better off sticking with industry and just pointing it all at an economic project. The AI here is doing the right thing - avoiding bankruptcy - but going about it in the wrong way, by re-allocating highly productive workers into tasks they lack the infrastructure to do usefully - it's making the guys on the factory floor sit down and do the accounts while the machinery sits idle. Then there's some basic other problems - the way the AI picks which build queue to use on planets is based largely on what special the planet has, and research specials are rare, so it under-produces research-dedicated worlds; it's research choices are (in most cases) sub-optimally weighted; many of its blueprints are outright broken; the entire BestDefense thing is conceptually flawed.
As to the bonuses the AI is granted; the scripted nature of the strategic AI, combined with the lack of difficulty triggers (so the Easy AI uses the exact same script as the Godlike one), means that the AI doesn't make efficient use of the handicaps. It cannot mass-produce colonies in an effective way on Godlike without scripting it to rely on bonuses which do not exist at lower difficulty levels, and so would be completely useless on Normal or Easy.
This is very frustrating, because the actual scripting of the AI is not difficult. Most players could likely figure out how to write a decent AI script given 30 minutes or so and a very basic thread outlining what the different script options do. That means that, as a community, we could probably produce challenging AI scripts for 500+ turns on all different difficulties in the space of a week; and Stardock could take this work and use it in an official patch - but we are unable to do so based purely on the primitive strategic AI triggers in the xml. Just adding in a difficulty check would mean that we could make them take advantage of the bonuses more effectively.
TL;DR: The AI will continue to perform poorly until it understands planet-specific economic settings, and handicap bonuses do little to help the situation because it cannot be made to understand that it has them. If the AI is given a means to specialize planets, then we will likely immediate see it performing 2-3 times better than it presently does.