Deer will eat spicebush, like they'll eat sassafras, although I wouldn't say either one is a favorite, like hydrangeas or day lilies are. Either deer or rabbits ate some of my sassafras coming up from an existing root, then left it alone long enough for it to grow real tall real fast. They've eaten the few spicebushes I've tried to grow, which may be a good reason I was never successful at growing one. I don't know about swamp milkweed. I don't think they like buttonbush. Mine has gotten huge and colonized, and I can't remember it being browsed by deer. That's no guarantee that yours won't though - deer can and sometimes do eat the most unlikely things. I planted two Cherokee roses up front, by the road, thinking deer would never eat them, since they're so thorny. Well, something did, either deer or rabbits! Not down to the nub, just a light grazing, but it was still surprising!
Sherry

Deer eat my spicebush and swamp milkweed. They have yet to touch my button bush.

KC

I had not gone out in my backyard for a few weeks. Figured I'd check on things. The last few years, I had my cherry trees surrounded by green plastic fencing attached to garden stakes. It had worked out. Then last winter, the deer wanted in and broke most of the stakes. So I replaced them with steel stakes and fishing line per a suggestion from my neighbor. Just found that most of the fishing line was broken and that my tallest tree has the bark chewed off of it. It is the tallest because the taller trees all died last year from getting their bark chewed off. I still need a mountain lion.