The number shown by the Victory Condition is the % success you have, achieving the 76%.  I'm at 90% right now.  And it states in the window, i need to spread my influence to 3% more of the galaxy.  At 100% victory condition, I will have 76% of the galaxy under my influence.

Concerning the drastic change, sounds like you may have lost a planet with a LOT of space in between.  Were you invaded by the enemy?

Correction..I just got an influence victory, and the number on the left is your progression for the victory, which happens at 100%.  When your progression hits 95%, you have influenced 76% of the galaxy.  The last 5% of the progression is the 10 turns you have to hold it. 

the influence graph isnt the output of your influence, its the number of hexes that are in your ZOI.  Is no one conquering planets?  Just "waiting for your influence to out do theirs will take a long time.  And their influence increase may be matching yours.  Conquering a planet is much faster, as your remove that planets influence and add it to your own.

if you want to play passive and friendly, and just try to let your culture take over the galaxy, you will need MASSIVE amounts of influence buffs.  it could be done I suppose.  To flip a planet requires atleast 4X (pun intended) of influence.  This works our well when the AI trys to colonize a small world in your empire core, that you havent gotten around to, or a nice huge planet that they finally have the tech to settle, but you never had.  They will be out influenced in a heartbeat, and it will flip quite quickly.  But to use influence to push borders and slowly flip their planets, you need a lot of influence and the other races would have to neglect their influence.  it can be done, but they usually go to war with your, when your influence is pushing them back too much, and they consider you weak militarily. 

Your style would work great if you are outmatched militarily, and have excellent planetary defenses to make them invasion proof.  influence would be the silent killer, but its still a slow process as you have to fight their influence.  Make sure to use Influence Adjacency bonuses well on influence improvements, starbase cultural improvements, high population approval, but its still a nurturing process.

I just finished an Alterian game.  Influence was great, but I decided to put more focus on research of military and win the old fashion way.  I happened to get an influence victory first, but taking control of enough of their space (and they surrendered to me in the end).  It might be fun to try to just play defensive and try influence only, but the AI can have VERY long range ships, and you will have to fight space battles eventually.  else, starbases and shipyards will get wrecked.