Yellow leaves usually means too much water, indeed.
Peat holds a lot of water and doesn't dry well, so overwatering is generally a given.

Josh

I'm about to pot by peppers up into 16 oz cups with generic peat-based potting soil... I was wondering, would mixing a large proportion of pumice be good for the seedlings? They are very small right now, only 2 or 3 true leaves each.

Pumice is an excellent additive, although it should be rinsed before using (to remove the fine dust).
Pumice will improve drainage, it will displace excess water volume, and it will provide a durable
open structure allowing air to return to the roots. Also, pumice is abrasive, which encourages root-branching.

If you're unsure, maybe experiment with just a few seedlings.

Josh