You need to thoroughly mulch and water your Sweet Olive. They sometimes burn in full sunlight. I have my shrub in light partial shade. I would also recommend that they get good drainage and good fertilizing at half strength in the in the dormant season.

Thank you for your Info.
I do have my Sweet Olive where it gets morning sun and afternoon shade, drainage may by an issue as I have it near a down spout for my gutters. It just may be getting too much water. Do you know if they move well? I had to move a Banana shrub last season and I do believe it now hates me!

It's too hot now. Wait until winter. If there's no taproot, you can move them. You might want to just redirect the gutter, it makes a great irrigater for other plants.