It does sound like a hornworms, even though I've never had problems with them eating my fruit (maybe something else is eating the fruit, maybe not).

When it has white sacs on its back, those are the cocoons of parasitic Braconid wasp larvae--and you should not kill those worms. The wasps will kill the caterpillar as they feed on it and then hatch out to reproduce. If you kill that hornworm, you will be taking away natural hornworm predators that are only helping you. As long as they don't have the white eggs on their back, you're doing the right thing by removing them off the plant. You can feed them to birds or pets or whatever as long as you aren't spraying the plants with anything toxic.

Hornworms are the caterpillars of moths, two species in particular (Manduca quinquemaculata or M. sexta). You might want to consider what is attracting the moths to your garden at night, or look into natural predators. You can attract more Braconid wasps by growing small-flowered nectar plants, but you may be able to find them commercially as well. Another organic solution would be to buy BTK (Bacillus thuringiensis var. kurstaki) to spray or dust on the tomato leaves and fruit. The bacteria will kill any feeding caterpillars, but is otherwise safe.