I seems to recall that the phantom warriors were more like 3 or 4 swords each, not 1.
Moreover, if you were fighting in a sorcery node, they get +2 ATT/DEF as all sorcery creatures in a sorcery node.
I use them often against other units and find them usually more cost effective that a direct damage spell of the same cost.
Note that they *will* fight worse against spearmen, as those usually come in squads of 8 for most race, so they need 8 hits instead of 6 or 4 to kill, and they get 8 units to counter attack agains their 0 def and 1 hit point.
Phantom warrior are a very good way to kill or at least damage a heavy squad or hero, they might not win everytime against spearmen, but they will do as much damage on swordmen, pikemen, or heroes and monsters as long as they don't have immunity, so you might say that they scale with the defence of their target.