My fig tree is beginning to form small fig fruits and my leaves are just starting to open. It wintered over outside in a pot, and I am late topping up the soil and adding worm compost. Is it too late to do that now? Will doing so somehow ruin the leaves or fruits, since I keep it in a large pot?

Any suggestions are much appreciated. Thanks.