I found some clear discs (like pie pans) in the Walmart garden section that have raised ribs so that when placed under your pots still allow water to drain out, yet also form a barrier to keep the ants out. Just buy some that are 2 -4" larger diameter than the bottom of your pots.

PS They're cheap too.

They probably won't hurt your pepper directly. Might even make your soil a bit more well draining. However, they farm Aphids and that is not a good thing. If it was me, I would probably try to get rid of them, but I would try to find a natural deterrent before throwing on a chemical pesticide. I am not sure what natural deterrents there are, but if you google it, I am sure you will get several suggestions.
Bruce

I noticed in my pots that treating the soil with BT (for fungus gnats) really knocked down the ant population as well - in the range of about a 90% decrease. The BT is not made for ants at all, but I suspect that it did a number on their larvae nonetheless. Worth a shot at least, plus it's organic (and fairly cheap).

Jay

Thanks for the tips. I'm afraid of killing the plant if I am too aggressive in trying to get rid of the ant colony. I will see what I can find online.

If they're fire ants, there's plenty of controls that you can apply outside of the container that'll get rid of the nest.

I'm pretty sure they are not fire ants though I haven't looked to closely. My last ant bite swelled up my foot so much it was hard to walk. I don't generally react well to insect bites. I'm not even sure how much of the nest is in the container and how much underground underneath the container.

Seriously, the "pie pans" worked for me.

I had ants take up residence in a flower pot next to my pepper plants in pots, I noticed dirt piling on the outside from underneath, I slowly tip the pot over and used a broom and swept them away and kept tapping the broom at the bottom as more kept coming out. Sprinkled cinnimon powder all over the ground under where the pot sits. Also a little bit of windex sprayed on the bottom of the pot. No ants to be seen now.

Mark

Move the container. Apply control. Amdro kills the queen.
Lather, rinse, repeat...

The ants would be really easy to kill. There are dozens of effective pesticides out there that will kill them or drive them away over night. If you don't have a problem, the just get some Sevin or similar product and apply it. Personally, that is what I would do, but there are a lot of people who don't like pesticides. Just be careful to wash the peppers when you harvest them.
Bruce

I've moved the effected containers over concrete so the ants can't burrow underground anymore. I will work on killing the ants in the containers after I have a chance to go to the store and see what is available.

Just leave the container on the concrete and surround it with Amdro. You don't have to put it in the container. They'll leave eventually without damaging your plants.

Amdro works if the ants are the type that take bait. Not all do.