Anthony, what kind of damage, the type you would see from deer or animal? Or bug?

Deer eat rhododendrons, squirrels eat the buds and those two things could get into other areas of your garden as well.

The only insect I know (not being a bug expert) that would bother both vegetables and rhododendrons are climbing cutworms.

Damage to rhododendron from climbing cutworm is large pieces of individual leaves eaten, sometimes to mid vein of the leaf, entire leaf sides may be gone. Root weevils leave notches along outer leaf edges.

My best advice to you is to go out at night with a flashlight (you may have to make more than one trip) and identify the problem...then post back.