One advantage of your having a cool climate, even if there's not enough summer heat to maximize growth and flowering of tropical species, is that you can grow plants like T. papyrifera (Chinese rice-paper plant) without worries about invasiveness (this plant is problematic in gardens in the deep South due to rhizomatous spread and supposedly is banned in Hawaii).

As for the "fear of frost" thing, I find that's more prevalent in areas where frosts are uncommon but do occur from time to time. Once gardeners in temperate climates get their susceptible plants indoors or mulched/protected for winter, they can relax - whereas someone gardening in a location where most but not all of winter is frost-free is obliged to scurry around when cold weather looms to cover or bring in susceptible plants (remembering for instance the semi-panic in the Houston area that ensues when a "freeze" is coming).

Agreed about the fear factor, but I've dialed that way down these days in my own garden by eliminating any plants that are zone 11 tropicals anymore. I also tend to stick with plants that will easily recover from a freeze and not take years to recapture their good looks, but sometimes can't resist planting exceptions to that rule such as Kentia palms or some new plant interest I stumble across while browsing nurseries in southern California. What I have absolutely given up trying to grow here in the SF Bay Area is any plant that demands hot weather to thrive. Too frustrating to see things just sit there all summer long doing nothing, only to give up the ghost the following cool summer back to back. Still, it is a lot easier gardening where it rarely gets above 80F, no biting horseflies or chiggers to worry about, and hardly ever too humid to want to be outdoors. I do miss those warm summer evenings to be out in the garden, hardly ever happens here more than 7 days in a year. And Tetrapanax is no less invasive here in northern California, I only plant it with a bamboo rhizome barrier to contain it. I've seen it spread to take over an entire back yard locally.