Gardenias can be persnickety. They typically need shade at least in the afternoon, although you will occasionally see an old one growing in full sun. (Go figure.) They do not like competition from tree roots. This will make those leaves yellow up very fast. They do like rich, fertile soil and a steady level of moisture (not wet, mind you). Chances are, the problem with the three that are not thriving is they are not liking that afternoon soon. I'd advise you to move them. My best ones grow on the north side of my house. All gardenia blooms turn yellow as they age. Pull them off. If you think they need fertilizer, try giving them a cup of Holly Tone.