Hi,
I guess first you will need to determine what kind of worms those are. Have you noticed any gnats around your tree lately? You can stir up the soil a little with your hands and see if any fly out. Sometimes they will surface during watering as well.
If so, the worms could be the larva of fungus gnats which will eat the roots of the tree which may cause it to eventually wilt and die as the roots can no longer take up nutrients/water. During the spring is when they seem to be the biggest problem here, when things are damp and warming up outside. They get into the house as well and cause havoc on the houseplants. If you think it is gnat larva then you can use a product called Knock Out Gnats fom Gardens Alive
that contains Bt/H-14 (bacillus thuringiensis). It is a bacteria that kills gnats and other tiny flying insect larva. It is not supposed to be harmful to pets, fish, plants etc. You will also need to use a yellow sticky trap to kill the adults.
I think a gnat life cycle runs about 7 days (adult), eggs take @4days too hatch, then will become larva/pupa and take @3 days to become adults. So you may need to reapply a few times to get all the gnats in their different life cycles.
Another cheap way to kill the larva is to break off a 1/4 size piece of mosquito dunk (Bt) (found at walmart or H.D.) and soak it in @ a quart warm water overnight and then add
enough water to make a gallon. Water your tree well. Repeat in a few weeks as needed. Some people just buy the granules and sprinkle them on the soil. Walmart also sells those pull out yellow sticky fly traps for @ a dollar as well.

Also, sometimes I find little worms that are earth worms when I am repotting. I think they must crawl in the pot while it is outdoors on the soil and they make babies. One time I found dozens of little baby earth worms and several large ones. I just let them be apparently they were happy there lol. Good luck, Hope this helps

P.S Another interesting note : Since I have repotted all my citrus this spring in a mix with cedar shavings (as per instructions from 4 winds nursery which says you can amend soil to loosen up the soil with cedar shavings or redwood shavings), I chose cedar shavings that you can get in a huge bag at Lowes as dog kennel bedding for @ 5 dollars. I haven't had any gnat problems in my soil. I know that cedar is also used in pet bedding to deter fleas. So maybe it has some gnat repelling qualities as well. So now all my citrus get mg rose soil (acidic) mixed with perlite and cedar shavings and they are loving it and best of all no gnats!!
~SJN

Here is a link that might be useful: knock out gnat