Many of my fruit trees aren't setting fruit, even though they had plenty of blossoms. If you were a bee, would you go out in that 40 degree rain? I didn't think so. Or another possibility is that it was too cold for the plants to pollenate properly.

One advantage is that plants that don't set fruit will use more energy making new growth. But you need to prune them so they don't get out of control.