If you want to add annuals, I would designate a bed for annuals or keep them to the edges (as you suggested) allowing the perennials to remain relatively undisturbed.

While a lot of folks like boxwood for edging, I personally can't stand it. Boxwood always smells like cat pee to me - so in my opinion, you're not missing much!! ;)

Here's a few that I've grown that are at least zone 5 hardy - you'll have to check if they work in zone 4. You'll also have to review sizes since some are large for smaller gardens. All perennial unless otherwise noted.

Echinacea - Medicinal, beautiful, native, pollinator attractor
Lemon Balm - Medicinal, tea, culinary, you can't kill it, deer can't kill it.
Mint - take your pick of peppermint or spearmint. Some of the flavored mints are more tender to check zones.
Bee Balm - Medicinal, tea, native, pollinator attractor
Chili Pepper - Medicinal, culinary, pretty plants and tasty, healthful fruits
Rose - Medicinal, tea, culinary, aromatherapy, as beautiful as useful
Catnip - Medicinal, tea, and fun for many cats, short-lived perennial
Sage - Medicinal, tea, culinary, wonderful herb
Marshmallow - Medicinal, fuzzy leaves, large plant, hollyhock-like flowers
Garlic (hardneck) - Medicinal, culinary, a wonderful allium
Chives - Culinary, another wonderful allium
Thyme - Medicinal, tea, culinary, yummy!
Parsley - Biennial, medicinal, culinary, tasty
Borage - Annual, culinary, pollinator attractor
Calendula - Annual, medicinal, culinary, my garden always has them
Lovage - Culinary, a stately beauty with a the max'd out celery taste
Fennel - Biennial/perennial, medicinal, culinary, feathery fronds are very attractive, swallowtail butterfly host plant (parsley too)
Comfrey - Medicinal (cautioned), pretty
Violets - Medicinal, culinary, who doesn't love them?
Angelica - Biennial, medicinal, culinary, a stately and dramatic plant
Sorrel - Culinary, yummy, sour lemon taste
Wild strawberry - Medicinal, culinary, tea, a native plant prized and celebrated by the Native Americans of my region (Haudenosaunee aka Iroquois)

There are others to be sure.

FataMorgana

FataMorgana- Thank you so much, for the response...and the information, on the boxwood :)

I have several of these herbs, already in the fairy garden...and love them! The woodland strawberries are hidden under taller, deer resistant plants and they make a wonderful ground cover. I'm also planning to move some violets from another location and add those to the garden, too.

So far, the spearmint and the pineapple mint have been very hardy...and the sweet woodruff. I'd like to add some angelica, but I understand that takes more shade, in the summer. Do you have yours in a shady area?

In the kitchen garden, I'd like to have some of the larger herbs, mixed in with the perennials, along the border. I have beds all around the garden, with raspberries, blueberries, strawberries, smaller shrubs and perennials and some annuals. I'd like to add some peonies and bulbs for more spring color...and the dill is usually hidden in with the cosmos, zinnias and marigolds, to keep the deer from chewing on it.

For the middle beds, maybe I should concentrate on culinary herbs...then have the tea/potpourri and medicinal herbs, mixed in with the border. Still trying to figure out what should go where, but thank you so much for the list. There are a lot of great herbs that I'd like to add :)

I've grown angelica in partial shade to full sun. It does fine for me. I plant things close together so the soil is always shaded. I think that and giving them the water they need in the hottest summer weather we have, is key to having them do well. The herb gardens at the nearby Cornell Plantations usually has angelica in full sun so I'm not the only one.

FataMorgana

I once saw a photo of a bed edged with curly parsley. Personally I prefer low, creeping plants like thyme to edge beds. It looks good all the time (no pun intended). I like the addition of nasturtium, esp Alaska which has variegated leaves, to the garden but don't plant it too thickly since it can get large. Not sure if deer like these plants. We have deer but the repellant spray I use around the edge of the garden seems to keep them away. Calendula self seeds in my garden so I usually let it grown where it is. I like the color that the blossoms add.

What about scented geraniums? I am too lazy to over winter them indoors so I buy new plants each spring.

I have been able to over-winter rosemary indoors by keeping it in the bathroom with a north window. It's on the back of the toilet. I think the humidity from the shower helps plus I don't usually forget to water it. It's not terribly happy indoors but it survives. Our current bathroom also has a west window.

I think keeping a map is a good idea. Plant markers are nice but they are in the way when you rake or disappear over winter.