Is there any way that you could take a picture or two and let us see these plants?

Mine is planted in unamended red clay soil and has grown about 6 feet in three (maybe four) years. It was given to us as a small but healthy three gallon plant. We don't have an irrigation system, so all new plantings are hand watered for a period of time. We fertilize sparingly and infrequently. It has always bloomed prolifically off and on during the years.

Please don't add phosphorus to your soil without having a soil test that determines a severe deficiency. In most cases, adding phosphorus to the soil is not just unnecessary, but harmful...to the plant, the soil ecosystem, and even the environment. It's a myth that won't die.

One thing you can do to improve the root/soil system for your Osmanthus fragrans is to maintain a 2 to 3 inch layer of mulch in a large area around the trunk BUT NEVER PILED UP ON THE TRUNK. If you improve the soil environment, the roots respond by increased growth and development. Once that happens....the top growth will respond the way you want it to.

Avoid excess Nitrogen, too. I mentioned that we didn't not amend the soil when we planted. That's an important factor, even in clay soil. Also, plants should probably be placed a bit higher than the natural soil grade.

You said you were sure that you've done something wrong. How about telling us just what you've been doing (and not doing) and we can be in a better position to help you. You can see from my own experience that you sure don't have to do much, even in red clay soil! If I can, you can.

i have a small one i have in a pot that i keep in the glasshouse but it has dropped leaves all winter...when i look up to see if it wants to be moist or dry-ish, i get conflicting info. has anyone here kept one happily in a pot?
many thanks!

Dabney,
I did.
I used to live in zone 6 many years ago, and I had a sweet olive in a clay pot for years.
They don't like wet feet, but they do enjoy water.
They prefer moist soil.
I kept it in the house in a sunny window in the winter and it was fine.
I brought it outside in May and it would bloom.
i did flush out the soil in the spring to flush salts out of the soil and always fertilized with a small amount of Miracle Grow in the water when inside, then increased miracle grow when I took it outside.
They grow slowly, not a real fast grower, but will outgrow it's pot usually in 1 season.
Foliage Pro is excellent for potted plants and my tea olive loved it.
I bought Foliage Pro online.