Hey, I know it's been months, but I've found a way to change heightmaps for a custom planet. I always forget to take notes when I'm modding and I quickly forget how I get to do things, but here are some explanations I remember:
1. It is impossible to mod heightmaps through the mod folder. All new heightmaps must be placed in your regular game folder, where the original are.
2. The game will not recognize any new heightmaps by itself, you need to give it a little push:
- Under your game's properties, in steam, you need to modify launch options and add the line : /atlasgen
- Once you have played the game at least once after using /atlasgen, it won't be necessary anymore. Not unless you bring new changes to your heightmap.
I think that should help. If you have any other questions, send me a private message. I'm not too often on the forum.
Good luck!