Well, some may not have a problem with them, but in a test game I have been playing on a large map (normal) with just the Drengi and common pirates, it took me 150 turns to get them out of my half of the map. It may be partly me. I tend to play a massive force combat game and I only lost about 4 ships to pirates. I was sending over-sized fleets with 4-600 HP to kill them off.
I ended the game with an Alliance V just to see the stats. They had been giving the Drengi pure hell. The Drengi were way behind and had lost over 75 ships to pirates and they were pretty much stagnated. The Drengi had killed way more than I had but it was obvious they were taking them as they came and not trying to finish them off.
I was progressing slowly, and by the time I ended it, I was at a point I could play normally, but they had been so distracting, I had a hard time developing my empire properly and I was very unhappy with the overall game, and with what I had accomplished. I was more or less disgusted with myself and the damned pirates.
I don't know if anyone is playing with the bastards on common or more, and liking it, but my hat is off to anyone who does. 
