My practice is to retain all leaves, and partial leaves, that have green on them as they are still contributing to the plant.

As for brown leaves, remove at will.

Bagworms or tent caterpillars? Just curious.....jean is right about letting the plants recover on their own. Plants have a built in ability to balance their carbon resources at the pace that is best for them.

Their recovery mode has already kicked in by the defoliation by caterpillars. They need every little bit of foliage in order to manufacture the food that will fuel their growth and storage responses.

Tentworms, generally early enough that trees can releaf, seldom require any pruning. Bagworms, mid to late summer cousins of the Tentworms, may damage branches enough that they do not leaf out again and that can mean some pruning is needed. You need to look carefully at the affected branches to see if there is new growth coming in.

according to your story.. you have solved your problem..

now put down the tools.. and step away ...

and no.. i would not be fertilizing pots in fall ...

ken