I'd be tempted to remove them by hand and toss them where birds are likely to find them.

Me too, hahaa. I don't mind a few raggedy leaves, but I'm just worried that the population of 'bad' caterpillars will somehow go out of control and start swarming my poor plants.

That could be any one of about one hundred moth larva that do chew on leaves and these can be controlled in the early stages of life, if dunking them is too much, (first three weeks) with sprays of Bacillus thurigiensis - Kurstaki. After that time Neem Oil sprays or pyrethrin dusts keeping in mind that both of these are broad spectrum poisons and should only be used as a laste resort.

Actually, Bacillus should probably be used only when the caterpillar population threatens the plant (or crop). Excess use of this bacterium threatens non target larvae of Lepidoperans.

If merely a few caterpillars are present, they could be bird food by the time someone mixed up a batch of Bt-K and applied it.