How could you make a fleet big enough to defend all those planets though?
Don't need to defend all of them. Just the borderworlds will do. Station your defensive fleet near several chokepoints, put up a bunch of hangars, turrets, repair bays, and jump inhibitors, and you've got yourself a pretty solid quagmire for your enemies to wade through. Then they can either stay and fight your defensive fleet, or try to push through while getting attacked by bombers and your fleet catches up to finish them off.
The key is PJIs - without them, the enemy fleet would just rush in to your capital and kill off your empire from there. But you need to protect the PJIs, so you build a bunch turrets to keep the enemies at bay, some repair bays to heal the PJIs, and a few backup inhibitors so the enemy can't just rush them and move on. Then you need hangars to protect the turrets from LRMs. Now if the enemy wants to get through, they have to bring flak/carriers to kill the bombers, LRMs (or just lots of firepower) to kill the turrets, and a bunch more stuff to kill the PJIs and repairers, which should be built as deep in the gravity well as you can, preferably on the other side of the planet. Even better if you have a frigate factory or two and some spare fleet supply, so you can build whatever unit would best help defend, right on the spot. By the time they get through all that, your fleet should be there and in position to intercept the enemy and do some damage if they attempt to run away.