I have a passiflora vine on the north side of my patio and it does well there. It's on a trellis, but it easily sticks to the stucco and climbs the walls on the sides (despite me moving it when it does). It gets sun just like you describe, no direct sun in the winter (it goes dormant), and just morning/late afternoon in the summer.

Hey Andy, any idea which variety of Passion Fruit vine that you have growing?

I love iAndy's suggestion of passion vine, so fun. I've seen a lot of east and north-facing walls covered with creeping fig, Ficus pumila. If you don't mind it clinging to your walls, then that's a good option for you too. Let us know what you select and how it works out. LOVE Palm Desert--you've got some fun nurseries nearby! Happy gardening!

Here is a link that might be useful: Pics from my little garden, May 2014

I started 3 different varieties from seed, my first one was passiflora edulis (started 2 years ago), and that's the big main vine by the porch that seems impossible to kill.

The other variety I started is Passiflora Foetida and I'm pretty sure it died over the winter.. It was in a pretty bad spot, one with hardly any sun all year, so I may start another one and put it in a better spot.

The last one is a maypop. I haven't figured out where I want to plant it yet so it's just sitting in a container not really growing (they need a lot of root room I think, so containers are a bad idea I believe). I'll need to find a spot for it soon.