We grow things deer don't like outside the fence. We have an outside perimeter border about 2 ft. deep, for allium family vegs.: onions, garlics, scallions.

Since you appear to have an abundance of space, I would personally grow flowers and ornamentals on the outside of the fence. (I'm jealous. I have been thinking all winter how I can add just a very few flower areas to my potager.) This would give your potager depth and add to the structural bones of your area. (Adding some evergreens to the bed would add even more. For instance, add a small conifer to each corner.) If you can focus on plants deer don't like to eat, you MIGHT turn them away from the area where they like everything! :)
I can tell you that I have never had deer bother zinnias, salvias, amsonias, or ornamental peppers. Ditto for daffodils and camassias. There are plenty more. Typically, they don't like anything with "smelly" foliage so that might mean herbs are safe too. I grow my herbs in pots, so cannot speak from experience.

Ah, thanks for getting some threads going, lavender. It feels good to type about garden stuff again!

I like the idea of growing outside the fence. I only have a butterfly and bee garden in the front. I grow butterfly bush, catmint, lavender, beauty bush, grasses, potentilla, forget me nots, chives, alliums, yarrow (sometimes gets chewed), hyssop, bee balm, lambs ears, sage. I continued the bee/butterfly garden inside the fence in the front with stuff the deer find tasty: more yarrow, daylilies, roses. There's so much stuff in there right now. This year everything needs dividing.

I'm also planning on making a bed of leeks along one side of the fence that has nothing. I'm also going to move my blueberry bushes outside the fence either to a new bed in front or along the back of the potager.

I think beds around the potager add to the continuity rather than having a naked fenced rectangle with gorgeous stuff inside.

Last year I had catnip, garlic chives, tarragon, lots of basil, vining nasturtiums, cosmos, yarrow and a few leftover pepper plants all outside the fence. This year I want more color and will include a bunch of beautiful purple shrub sized Dahlias that I have extra tubers of.

I garden around the perimiter of my city fence. I have to agree with Donnabaskets, growing perennials is in may ways so much easier than annuals in a tight space in the city (not that you are in the city). That's simply because they like to spread around by seed and I've not left them much room to do that..so they can flop. It's also harder in some areas where the fence casts shade at all times except for the early morning. Herbs have been my best bed, loads of thyme and low growing oreganoes, also carpeting phlox. I will make "pockets" down in between these plants to put other things. Along the chain link fence it has been vegetables but I don't eat them, they are just pretty (kales in winter) and I had lots of extras. It's not great, just OK right now. I'm looking forward to getting my wooden fences in!

My fence exists to keep out dogs, not deer. So I keep my tender plants that can be dug up inside the fence and plant fruit trees and and tough plants (like very thorny roses) outside the fence. I love my dogs, but I hate it when they kill my plants.