There is a treatment for catapillars that you can buy that is very effective. It's called Dipel, or Bt (Bacillus Thuringiensis). It is completely organic and comes in spray or dust. You have to get the whole plant. When the caterpillar ingests some it kills them within 4 hrs. Spray your plant before any holes appear in your leaves so you get the worms when they are babies and you probably won't ever see any large caterpillars/worms on your plants.

Dipel also doesn't kill beneficials. Very specific to worms and caterpillars. Mine is called "Worm and Caterpillar killer", you have to read the ingredients as I haven't seen it sold as "Dipel". I sprayed a monthish ago and it's time to spray again, though I don't know what the actual intervals you are supposed to spray to keep the plants worm-free.