"Sounds" like two issues rather than one. Investigate spider mites, a dreaded pest of indoor plants in general, not just citrus.

The little flies might be the very common fungus gnat. They are almost ubiquitous in the typical, peaty, dense potting mixes often found at the big box stores. It's a challenge to water those media properly, creating an environment for fungus gnats to lay their eggs.

Thanks bud :)