don't know about peppers, but people sometimes remove the first buds from flowering perennials so that the plant's energy will go into stronger stems & roots for that important first year, & they sometimes cull small fruits from peach trees & such so that the remaining fruit will be bigger.

Aren't peppers annuals?

If so, you might be discarding its entire crop!