I experienced something similar with the Krynn. At first contact, they declared war and I took around 20 planets from them and won every single fight. They begged for peace ~5 times and I declined it every time. After getting enough planets, I wanted peace to stabilize my approval rate. Despite them begging for peace, if I finally offer them peace, they just say "You must be joking". After kicking their butts, I can't even demand a single thing for peace or give it to them for free...and the AI was on 'genius'.
Most of the time you can't even speak to the different races and diplomacy itself is rather simple. The AI is demanding baseless stuff and you can't even ask for simple things like "ey, don't cross my territory, we don't have a open border treaty". "I swear, one more ship in my territory and it is time for a war" etc. But the AI is sending me that kind of message all day long complaining about close borders and me building ships etc. I really hope a lot of aspects in GalCiv3 will get some love now that the game is released.