I don't think the ants themselves are necessarily bad, but they can nest in your pots. And I think that can cause fungus.

What you have to worry about are the farmer ants that put aphids in your plants and milk them like cows for the honeydew they produce. They protect the aphids from predators.

I think that's interesting, but not when it's in my plants.

Here is a link that might be useful: ants farming aphids