Trim them off. Once the plant is about a foot tall, you can start to trim off the lower leaves, bearing in mind that the plant needs its leaves for nurture. If the plant is tall though, I always trim off the lower leaves to about 4-6 inches depending on the size of the plant. This helps with air circulation, and helps to avoid the development of mold on the leaves. It also makes it easier to water the plant without getting so much water on the leaves,- again helping to reduce the chance of mold.

If they are brown, cut them off. The aren't doing the plant any good anymore.