Try GARDENIAS...ha ha ha! They definitely aren't invasive, heck you have to pray every day that they don't die on you.

gardenias will probably be marginally hardy in tennessee but a protected location can help. they do well for me in partial sun.
many of our native azaleas are fragrant and also like a site with some shade, particularly afternoon. they make fine woodland garden subjects.
carolina jasmine (vine) has a nice scent and can take sun to partial shade.
i really like the fragrance of dianthus. likes sun but maybe 4-5 hours would do. 'bath's pink' performs better in the heat of my zone.
i also like moonflower vine for night fragrance. those large, white flowers can pick up the moonlight and stand out in a white garden.
daphne odora would also need a protected site in zone 7 but has a wonderful winter fragrance and prefers partial shade in the south.
sweetbox (sarcococca) prefers shade. it's evergreen and blooms in winter.

You can check your state's native plant society:

http://www.tnps.org/

for more fragrant selections; in addition to jeff's excellent ideas I'd add fothergilla (witch alder) itea virginica, and, if you've got room, magnolia virginiana (sweetbay).

Jim

Oriental Hybrid lilies!