When I moved into my home I was happy that there was a 10ft orange tree in the backyard that had big oranges that I shared with my family. Five years later, I'm nervous that the orange tree is dying. For the last two years the fruits has become smaller and less of them have ripened. There are barely any leaves on the tree too, over the years the tree has been less lush. I water it regularly (3 times a week or when I see it really dry.) I live in Southern California were we don't get much rain and my dirt is very very hard and clay-like. I feed the tree hands-full of fertilizer at least 3 times a year too.
Any advice would be greatly appreciated. I'm fairly confident and willing to learn and try to save my tree.

Thanks!