Scrape away a little of the bark and see if the stems are dead or the leaves are just brown. If the stems are dead, pruning them off won't affect the plant at all.

If the stems are alive (green cambium layer), leave it be and let the dead leaves fall off on their own.

If you want to prune back stems all the way to living tissue, wait until it blooms. Well, if they do this year; I'm not sure sure they will.

The ends of the branches are dead. They will break off easily. I have found that about 10 inches into the limbs there are green limbs. I found about 4 or 5 new green leaves starting to grow in one area that was more protected. There probably won't be any blooms but will wait a little longer to prune it back to the living part of the limbs. Thanks for the advice. Wasn't sure about when to prune it.