Dominated ships have the tendency to follow the enemy's retreating fleet for some reason, even though they are not a part of it. After making it yours, give it a "stop" order- Works most of the time.
That's not the only wierd behavior of dominated ships: They quite often form a single-ship fleet, the minute they are captured. Never figured out how that happens.