Some of my favorites that get "what's that?" comments from visitors are:
Lovage - gets large and smells/tastes similar to celery
Pineapple Sage - great fruity scent, returns each year for me, which it shouldn't considering my zone
Lemon Verbena - super lemon scent, mine comes back every year which is unusual for my zone
Wormwood Absinthe - large bush with feathery grey foliage, very ornamental
Artichoke - very ornamental, large plant, fun to let the flower open up instead of eating it.

They're not terribly uncommon, just most people don't see them everyday...

I also grow "deadly" plants as conversation pieces:
(Disclaimer: I don't have children nor outdoor pets)
Datura (awesomely fragrant & large night blooms)
Deadly Nightshade
Castor Beans (Red variety is very ornamental, 2nd most toxic plant poison on Earth)
Rosary Peas (for a long time these were used to make rosaries, and they contain THE most toxic plant poison on Earth, go figure)
Monkshood
Foxgloves

I have seeds for stinging nettle, I couldn't believe these were commercially available. I Got 'em mainly to show folks that you could probably find packaged dandylion seeds too, LOL.

People usually ooh and aah over stevia, don't know why because it's becoming quite common. The lemon myrtle is a talking point, as are the arrowroot, galangal, turmeric, cardamom and ginger. You won't get these to do well in a coolish climate, however. People like the assorted mints - chocolate, basil and Japanese menthol especially. And they like the Chaste-tree.

How about some liquorice? Alexanders? An assortment of dye herbs (alkanet, madder, weld, woad, indigo etc)? Asafoetida?
Medicinals like Astragalus? Withania? Schizandra? Bayberry? Boldo? Calamint? Centaury? Epazote? Fumitory? Hawthorn? Mushroom Plant? Jacob's Ladder? Pasqueflower? Squill? Viper's Bugloss? Could you grow things like Ginseng, or Arnica, or Goldenseal?

I bet nobody you know grows zedoary! (much like ginger).

Got room for climbers like vanilla or pepper or loofah?

If you go for poisonous, you could try the henbanes and hemlock. Personally, I won't grow these nasty ones in my garden, but some are pretty, if dangerous even to touch.

Thing BIG. Think trees. Bay, neem, cloves, eucalyptus, any fruit trees, curry leaf tree, ginkgo biloba, oak, white willow etc.

Check through the sites at this google search for rare herbs for inspiration. You'll obviously need to do your homework as to your climate and plants suitable for it. Also you may need to check with local authorities about the legalities of growing certain herbs which may be environmental threats etc. You have about 20000 herbs to choose from!!

Here is a link that might be useful: rare herbs

Well thank you! I guess I need to do a little more looking. I was at the greehouse this morning and they have a bunch of funny kinds of mint. I bought some of those. I'll for sure check out some of the ones you both listed. Thank you again!!!

Mullein
Verbascum thapsus.
I grew a row of these in my garden. They were nice to look at.

In addition to all the great plants mentioned:
rose scented bee balm, rose scented thyme, lemon savory, rhodiola, maral root, osha, Oregon grape root, coptis, butterbur, angelicas (archangelica, Korean and Chinese), boneset, vervain, tangerine southernwood, astragalus, epazote (even though I think it smells like kerosene), black cohosh, blue cohosh, Solomon's seal, lungwort, liverwort, bloodroot, gotu kola, ginkgo, mitchella, pipsissewa, Chinese forsythia, etc. Welcome to the world of unusual herbs!

You live in America, and you have a lot of herbs in America that we don't have here in Australia, and some of them I'd really like to get hold of!

Think of Sweet Joe Pye, Chapparal, Mesquite, Partridge Berry, Creosote Bush, and many more.

http://library.thinkquest.org/03oct/00175/naherbs.html

http://www.galileo.org/plants/kainai-html/index.html

Here is a link that might be useful: native american herbs

Since you're not afraid of poisonous, how about a Pokeberry bush? They are very cool looking, with showy white blossoms in the Summer and gorgeous purple berries in the fall. Watch out though. They can be invasive!

Here's a pic!

Here is a link that might be useful: {{gwi:892848}}

As nice a color as pokeberries have I can't find enough words to say how much I hate this plant. There were a few in my yard when I bought my house. For as many as I've dug, chopped and beaten to a pulp more come up every year. The tap root is difficult to get rid of. The plant gets quite large, 4-5 feet in some cases, and I swear the seeds go everywhere. It may become a talking point but not in polite society. ;^)

One of the herbs that I love but haven't found to buy in the garden centers is costmary. The scent is just out of this world. I have some plants that a friend shared with me, otherwise I haven't seen this herb in any of our local garden centers.

Stacie

I've got costmary. In my climate it hasn't grown much at all since I got it about 5 years ago, but it's healthy. It has a lovely smell to it, but the taste is a bit robust for me.

Another one you might have trouble getting is Balm of Gilead. Lovely perfume. You won't get the one mentioned in the Bible, however. The one you'll probably get is Cedronella canariensis.

Patchouli (make sure it's Pogostemon cablin, not P. heyneaus) has an Eastern sort of perfume to it. Nice, if you like it, which I do.

Here is a link that might be useful: balm of gilead

Rue is a beautiful plant not often seen, easy to grow from seed and also easily obtainable.

My costmary is about 5 feet high and is getting very bushy. Another name for costmary was ale cost and was used in beer brewing in ages past.