My yuzu is about the same age as yours. It is growing like crazy but only made 6 fruits this year. Sadly, we accidentally cut off 4 of them when doing some grafting work. OOPS! Well, 2 yuzu is better than none. I figure it will take off next year, just like my calamondin and clementine did this year after a couple years of squat.

I wonder if it was too warm in the winter.
I see you're in New Jersey, so I'm sure you bring in your citrus in winter.
The Yuzu is a very cold hardy citrus, (survives down to 10*F) and I believe it needs a bit of chill in winter to produce fruit.
Not much chill, mind you, not like an apple tree, but constant warm temps inside may affect it. Do you have a cooler room or garage you could keep it in? It may make a difference.
When we had a warmer than usual winter my Yuzu didn't fruit as well. We usually get 40*F nights, and occasional frosts down to 28*-30*. When we get those temps, I have to cover my other citrus, but my Yuzu likes it. It's normal for it to drop it's leaves in the cold, it leafs out quickly in spring.
Maybe try it give it a cooler place this winter, and see how it does next year.

This post is late. I also have a yuzu tree from Four Winds. It bears fruit every other year. Apparently, from what I read on the internet, yuzu bears fruit every other year. I don't know if all yuzu is that way or just the variety we have. I will give Four Winds a call.