hello, I am just an occasional lurker, but see that you haven't had a response, so thought I would jump in. I have no idea what it could be, but I did a search for pests and came up with a site with some info about something called "cranberry fruitworm" which can be a problem with this shrub. The link is below. I hope that helps, but if you don't think that is it, then maybe you could take a sample of the worm (in a sealed container of course :)) to a nursery around you and see if they can identify it and suggest something to control it.
gg

Here is a link that might be useful: cranberry fruitworm info

Google viburnum borer. Its another possibility.

tj

Here is a link that might be useful: Borer