I'd wait a little and only cut back what you know is dead...Mine actually has I think less growth then yours and has just first started putting on leaves this week. If I recall, it blooms on second year wood so cutting all the stalks will give you lots of growth from the base maybe but not a lot of flowers if any! I bet that's why you didn't get many flowers, if you were trimming it back. My dad did that one year and I was devastated! We did get a few but not nearly as many as previous years....
~Michael :)

Thanks Michael. I'll let it go as is for now. I will be visiting a garden center this weekend and will take a photo to show them and see what they say as well.

Mine are in worse shape here in NEPA. This winter was harsh on them. If there is wood you KNOW is dead (very brittle, no green, no leaves,) you can cut those down to the ground or once you see green in the stem closer to the bottom.

I'm not cutting anything that still has life left in it. We haven't had a winter this harsh as long as I've been gardening, so I don't know how the live but stunted branches will fair out. I do know, however, that if I cut them all back, I'll have a very disappointing bloom. I'd rather deal with some ugly sparse wood for a few more weeks than risk losing this years' blooms entirely.