Should I remove all the branches growing and leave just the main vertical branch?
I would do the opposite, remove all vertical branches. You want an open vase with a peach tree. Here's one of mine.
Drew, Could looking tree, Is that paint you have on the bottom trunk?
Wow, Just realized I wrote COULD instead of GOOD, Time for a nap.
I still understood what you meant! Yes, it's in full sun, it protects the trunk from sunburn, and also stops the trunk from heating up during the winter. If cells in the trunk get warm enough during a sunny winter day, and the temps drop at night, you could have cell damage. I had one peach lose it's central leader last winter. It was not painted. Luckily the very base of the plant was still alive above the graft.
Instead of developing a new central leader, I just let it form new scaffolds. It's super low, but what the hey!
All branches are new growth from this year. Some water sprouts (vertical branches) I need to prune out.
I don't know how much it will matter being in a pot like that. I would just let it be...