If you haven't checked the following website yet, I suggest you do so: www.oregonolivetrees.com A big site with tons of valuable info on olive tree cold hardiness. Conventional wisdom has it that olives can't stand temps below 15F, and no subfreezing temps at all that last for any duration. But I used to live in Turkey and saw an occasional olive tree in mountainous or high steppe areas with very harsh winters that were far beyond the milder coastal areas where olives are successfully cultivated. Perhaps microclimates or just the occasional exceptionally hardy tree? You might try a few, if possible, in large containers that could be brought into some place where the trees would get the winter chill they need for fruiting, but would not freeze. Olives are not deciduous, by the way.

Peaches and many other fruit trees need winter chilling for best fruiting while grapes are extremely hardy plants. They can grow almost anywhere. I'm in zone 8 in southern NM, olives manage to survive here although most of them froze to the ground last winter. It was an unusually cold one for us but it wouldn't have been where you live.
Olives are pretty much a "Mediterranean" plant. If you have a microclimate or very protected area you might can grow them.

I agree with nmgirl, there have been rumors on this forum of olive trees in protected microclimates in Albuquerque, which is pretty incredible, but that is still a good zone or two warmer than 6.

Potted plants may be the only surefire way to go, especially if you want fruit... I had some small potted olives in a sunroom for a while, but they never flowered, showed slow growth, and were very susceptible to scale. Other plants, like pomegranates, citrus, guava, mango, have done much better under the same conditions.

Incidentally, pomegranate is another Mediterranean fruit that would have a better chance for you outside in the ground. It's much hardier than sources say, sprouts back easily from the roots, is smaller and easier to cover for protection, and is deciduous, which means that it goes fully dormant in the winter. A nurseryman in zone 6 Moriarty, NM is trialing some pomegranates that survived last winter's record low of -34 there!

Back to the olives on a different note: if you read accounts of the early Spanish settlement of Santa Fe, you find that their plans included olive groves on every land parcel. Obviously they tried it and quickly discovered that olive trees could not survive here. I can see say with confidence that there are no true olive trees grown outside here in Santa Fe to this day. Similarly, almonds were initially grown here by the Spanish, and supposedly there were still almond orchards in Northern NM until the trees were killed by some intense winters sometime in the mid 1900's. I'm sure you could still find the occasional backyard almond tree, but they are apparently even more susceptible to late frosts than the locally widespread apricot.

Hey, I don't think you're a zone 6 anymore. Check out the new USDA zone map that just came out, I'm pretty sure you've been bumped up to 7a. That's only half a zone colder than Albuquerque, so maybe that gives you more hope?