You can harvest them any time. I wait until the fruit comes off the stem pretty easily when I twist it a bit. But it probably doesn't matter, you can use them any time.

I'm guessing your response refers specifically to Rangpur LImes? I believe I recall reading that Bearss Limes are ripe when they turn yellow...I don't have much knowledge about how taste/acid/sugar is effected by length of time on tree. Anyway, I was trying to find info that was recently posted that was a guideline for various types of citrus,I believe it listed oranges, grapefruits, mandarins, etc. It seemed like handy information to bear in mind. I'll try to find it again - maybe I was looking up an old post...