Are you treating your plants for the caterpillars? Bt is a very effective control, so effective that worm holes seldom cause a problem for me. Other controls are also available to eliminate the problems so you don't have to throw fruit away.

I lose a few to BER sometimes very early in the season but that's about it.

Dave

At this point, I'm just trying to catch them all. Check every day or so and catch the couple I can find.

Well your pic shows tomato fruitworm damage (holes eaten out from the inside). You will never see or catch tomato fruitworms until after the damage has been done.

So why not prevent the damage rather than throw away fruit?

Dave

Here is a link that might be useful: Tomato fruitworms