New players don't quite understand the importance of population. I'm going to take a few shots in the dark here. The transport that you start out with is loaded with 100 million colonists to start your next colony (I believe?), but you can fill a transport with as many as 1 billion(?). You will find yourself seriously lacking in power if you max out transports and send them to other worlds in the very beginning. Your home world will end up with too small of a population to produce large amounts of money, production or research. It doesn't make any sense to build a bunch of factories and research centers on your home world, and then send off all of your workforce to planets without any improvements. For this reason, your home world should be your most powerful planet for the first 100 turns (at least), so it needs the economy (i.e population) to support it's industry and research. I find sending out transports at 100 million (1/10th capacity), is a pretty good rule of thumb. This way you can colonize lots of new worlds without destroying your economy at home. Try to find a balance that works for you.
Another tactic related to population, is that the AI will try to keep taxes low to keep their people happy. This increases the birth rate. Make sure to try and research population increasing technologies in the very beginning, as this is the biggest factor (IMO) for building a powerful empire. Trade for them as well. You need a strong economy on your home world, and a large group on one planet will reproduce much faster than small groups over many planets. I forget what the "optimum zone" is but I'm sure someone here can tell you. I believe it used to be 3-5 billion or something? You can hold out on farms until you reach your population limit, but don't wait too long!
You said you skip hospitals and farms?! This is a huge mistake. The hospital is the most important improvement you can place on a new world IMO, and farms can double the performance of a planet that has reached it's population limit. Always build a hospital on a new planet, in fact, I usually just buy them! It's cheap and increases your production/research through population much more than any specialized improvement will. Get the population moving first, then give them a place to work! You can always build over your hospital once things have gotten off to a good start. I just keep them around until I need the space.
Protip: I also tend to build an extra production facility on a planet that has an extra space, and then just build a farm over it when the time comes. Several extra spaces? Build markets, as many as you want, they will increase your flow of money until you are ready to build over them. Don't leave empty spaces if you have a healthy economy!
Also
1. Increased population = 2. more money = 3. less taxing needed = 4. happier people = 5. faster population growth = Increased population (go back to 1).
If you want to see just how important population is, try setting your taxation at 100% for 20 turns deep into a game. This is especially meaningful if you have a dozen planets under your control and a decent size military. This will show you exactly what role population plays when 80% of your population leaves. I did this once in Galciv2 by accident and it changed how I played forever. I had to turn global production down to 0% and I was still losing money every turn. The game was over. Now imagine what 2x population can do for you!