I mostly use IE, sometimes Firefox and Opera. depends on what I'm doing. Yesterday I ran Spybot and Ad-Aware and both came up clean, and that's after a week of browsing. I do have MS Antispyware and Spybot-SD running in the background though.
All browsers have vulnerabilities. The more popular the browser the more vulnerabilities and criticism it will attract. Firefox has it's share of zealots in much the same way the Mac does, perhaps it's the underdog insecurity psychology at the root -dunno. To hate something (like IE) is not productive. Both do a job, personal choice is down to you