The black specks are worm poo. As far as getting rid of them....if you have time in the morning, check the undersides of leaves for tiny sparkly round things attached...those would be moth eggs, most likely. If you get rid of those you'll have prevented the hatching of more worms.

The least toxic (to people and plants and animals)solution to any caterpillar problem is BT spray. I am going to misspell it, but it's a liquid with bacillus thurengesis in it. These are bacteria that will kill the caterpillar about 24 hours after they take a bite of the leaf. Assuming they do so as baby caterpillars, you won't see much damage, and eventually the population will drop off to just a few every now and then. Follow the directions on the bottle. It is supposed to be safe to harvest same day. I still rinse my maters well, just to be on the safe side.

BT is generally considered acceptable by most organic gardeners...(certainly not all).

Another thing you can do is to encourage birds to pick off the insects and worms...but if you do, you'll need to pick your maters at first blush and let them ripen in a window indoors.