Mine lose some leaves and may have some branch dieback at around -10C (14F). I have lost smaller plants at -11C (12F)but thats the lowest my area has gone in the last 10 years.

Osmanthus heterophyllus has some cultivars that should be OK in your area, but o. fragrans, even in a protected area, would likely be a losing bet.

Besides, its best bloom is in winter -- grow it in a container and bring it inside where you can enjoy the fragrance without long underwear!

Jim

About 400 large bushes of Osmanthus have been planted in in pots at Imperial Gardens, Beijing, China where it is very cold. They move the pots in shelterd area during the Winter.
You may find the cultivars that are four-season type or early bloomers, so you can enjoy the fragrance during warm seasons.