Tons of stuff. It could be something in particular you are panning past, like particle effects or shadows. It could be your PC lagging preloading graphics. It could be your virus checker. Your hard drive could be a bit too fragnmented so when your game is loading things it takes a bit more effort.
There's a tweaking guide for XP over at tweakguides, it is a good place to start. Get rid of all your spyware and your demanding background applications, defragment, turn of anything that would update automatically while you play, and generally try and make sure it is the game itself lagging. Then hit the forums there and see what the issue is.
I've been playing Oblivion. I feel your pain.