Smell? Flavor, if you've tasted a leaf? Savory has a nice spicy aroma, and flavor. It does have the look of savory, though my winter savory grows much bushier than that. I've never had much luck with summer savory, and what I've grown hasn't been that full growth--the leaves are much more spaced out. But you would have planted annual summer savory anyway and I would assume known what it was.

Maybe Pittosporum undulatum - Victorian Box They were commonly planted in Southern California some years ago.

That was my original thought when I first researched I just wanted some more opinions. Thank you!

Also could be that it's planted too deep. I can definitely see that from the first picture. That can cause the bark to start peeling. dig out the base of the trunk until you reach the root flare. Those are the large roots that flare out from the trunk. Any trunk material above that should not be underground. If you see fine roots coming out of the side of the trunk, they are aventitious roots which are abnormal roots growing out of the trunk because it's buried too deep. Keep digging down until the large roots at the base of the trunk are flaring out. Expose those in the area closest to the trunk. You might have to go down 6" or more. Lots of times trees are planted too deep by the nursery. I think once you expose the root flare and leave it exposed, the trunk should stop peeling, and the peeled areas will eventually heal over.