If you tree is small, it may grow sturdier if you remove the fruit and let it grow larger. Many nurseries like to sell trees with fruit on them because they attract more buyers.

If your tree is a few years old, it may be okay with a few fruit. Navels are ripe in the winter. In future years, you will not have to thin. Oranges set out many more blooms than the tree can support, and some fruit that sets will fall off when they are tiny, so the tree is self-thinning.

Thank you very much for your response. :)

It is about 3-4 feet tall and maybe 2 years old (I don't know a ton about citrus trees but it's trunk is maybe 1.3 inches thick.

Anyways, the fruit is growing fast.... if I leave it, will it be bad fruit since we are heading into summer? Should I remove it anyways?

The fruit won't be bad. It takes longer than you think for it to ripen. The quality of the fruit will improve as the tree matures, so if the first few years the fruit is so-so, it will get better. Fruit production typically won't be much until the tree is more than 3 -4 years old.

Here is a link that might be useful: Growing citrus