exact same happened to my 2 year old sweetheart in the ground here. I think it just set too much fruit initially and as they get bigger, which they are well on the way to golf ball size, some get starved off by the tree

so you think this might be sort of natural occurance? Some will make it to mature fruit, some wont...

For what its worth...wind does play a big role on lychee trees...check lycheesonline.com. Also when its windy the faster your pot dries. Check for moisture with a thin wood dowel...stick the wood dowel carefully close to the bottom of the pot and see if the stick is moist...if not water it. Lychee trees require a lot more water than other fruit trees...learned this one from Harry and others on here!

Also, since you just purchased the tree recently, that could very well be the main reason. I'm sure its going through some adjustments from the nursery to your house.
Good luck!

Here is a link that might be useful: lycheesonline.com