How about some sort of cytisus, they don't mind heat/dryness that much, nor does osmanthus fragrans.

Cytisus is perfect for your situation except that it is extremely invasive in California. Why nurseries are still offering them here is beyond me. Osmanthus fragrans will get serious sunburn especially in the seedling stage. You can try mediterranean plants with scented foliage such as lavenders and rosemaries. You can also try roses if you do not mind the work. However, if you would like to try some other exotics such as michelia and boronia, you have to mitigate the afternoon sun situation in some way such as shading or misting. Most of the fragrant plants like it hot but not dry. Hot and humid evenings are the best times for most fragrant plants to show their worth.

I'd consider:

Pittosporum (several)
Lavender
Roses (Chrysler Imperial, etc.)
Viburnum (several)

You might also stick with the more standard sun/heat-loving foundation shrubs and bring fragrance in with a vine (jasmine, etc.)

Also, some daylilies are fragrant. They combine nicely with agapanthus to liven up a sunny foundation. Rosemary's another option, though it's the leaves that are fragrant. That might be a great spot for a culinary herb garden with roses and lavender mixed in!

Angela

Star jasmine will grow anywhere.

My Gardenias in full sun in Dallas TX area do just fine--our temps are high 90's and sometimes 100+.