It won't do any harm in your gardens. It's the lawn where it's a problem since the mower will send the gravel flying and can break/damage windows. If you really want to get it out, it's a long manual process with rake or shovel or you can use a Power Broom - which is not something most of us have access to. Gravel driveways just aren't a good idea in snow areas, but if you have one need to shovel carefully - as you know now! :)

I looked up power brooms and saw some videos on youtube. None of them would serve my needs.

I have these beds that I have put a lot of time into planting ferns and mosses. That amount of gravel would show through the moss and wouldn't do the ferns etc. any good either. If I go through the effort of scooping out the gravel, which is my plan unless someone can tell me a better way, how do I separate it from the soil?

I found a screen that worked well. Trash-picked it from somewhere. Funny thing is, that was in New York, and I left it there when I moved to NoVA, but the other day I saw a nearly identical one lying in someone's yard.

It was a wood frame, about 2' by 3' made of 2x3s with 1/2" wire mesh stapled to it. I used mine for returning gravel to the driveway in the spring, too. Sit it on the wheelbarrow and shake - soil goes through, rocks stay in, toss rocks, dump soil. Pretty labor intensive.

Where you are, with luck, it will be a few years before you have to do it again.

If the beds are already planted - you're out of luck. You'll have to pick the gravel out by hand.