I really doubt you had aphids on your tomatoe plants. If you did, it is extremely bad news for organic growers since a very effective method to control aphids is to use tomato leaves!
Tomatoes as members of the nightshade family, contain toxic compounds called alkaloids in their leaves. When the leaves of tomato plants are chopped, they release their alkaloids. When the alkaloids are suspended and diluted with water, they make an easy to use spray that is toxic to aphids, but still safe around plants and humans.

Aphid eggs aren't in soil.

Organic management of aphids: Squish on sight, or blast off with a harsh water spray, then determine what went wrong in the care of the tomatoes.

The cause of "lots of yellow leaves" include disease, insect attacks, and/or poor cultural conditions. We would need images and a detailed description to help you determine what might have occurred.

Thanks for the replies. They were most definitely aphids. I had them identified at my local nursery but I knew by looking at them what they were. I've seen more than my share on my house plants. The main difference were these were a reddish color.

As for the yellow leaves (and stems), I'm pretty sure the cause was too much rain and too little sun. Also being in pots probably doesn't help. I have no choice about the location and pots because I live in a condo. Just wondered if there was something that could be done to help leaves stay green. The yield was fair, with the greatest amount being now despite the poor condition of the plant itself.

"They were most definitely aphids"

Yeah probably an aphid that is immune to the toxin. I would think water wash would be too difficult to do on delicate tomato leaves. Maybe a garlic solution, or summer oil, or insecticidal soap, all of which should not bother beneficial insects.
Yeah we had a lot of rain, and fungal diseases were all over. I lost one plant of 6. But harvest has been good. Jasper f1 is an excellent cherry and didn't catch any diseases in a very bad environment this year. They taste great, super sweet. I will grow these again.

i did not read all the comments but i just wanted to say that YELLOW LOWER LEAVES, FALLING LEAVES are often associated with the potted plants of various kinds. It is just the plants mechanism to maximize its resources. In that respect , lower leaves do not make a significant contribution to the plant. So they are aborted.

Why The Leaves/Plant Yellow ?
The answer in "too much water.

Because the container plants need to be watered more often, the nutrients are leached out by watering. You add too much rain on top of that, the matter can get worse. That is why you have to feed container plant almost every time you water, BUT with a VERY WEAK solution (say 1/4 of normal recommendation)
With extremely small containers when the plant becomes rout bound, in effect one is growing hydroponically.