Don't think it's ants,they usually eat the aphids and such.More than likely it's either slugs/snail or those little green worms.If you see a glistening trail on or around them,then that's 'SNAIL SNOT" and those are the culprits.JMO.Others may have other ideas as well.
Kathi

Could be earwigs too. They climb up to eat at night. Go out tonight with a flashlight to see if you can find any culprits.
Renee

Ants don't eat aphids or plants. Slugs and snails do. and rabbits. and birds.

Ants "farm" aphids and eat their poop. Seriously.

We do have slugs, but I haven't seen the slime trail near my coleus plants - which are up on a table. I went out again last night and the only thing I found was one ant on a stem.

I'm still betting on earwigs. But if you suspect caterpillars, you can spray with BT, which is only toxic to caterpillars.
Renee

If your plant is on a table the pest could be a more mobile one such as grasshoppers. You won't catch them but maybe if you have a cat or dog he/she could help. When my dogs were young and had the eyesight they loved to help in the garden.

could be thrips. Spray for thrips