The two main things that would be affecting the fruit production of your plant would be too little direct sunlight or too much Nitrogen in your fertilizer. Stop fertilizing. In-ground fig trees rarely need fertilizer unless the soil is very sandy. Good luck!