Your experiment is valid. I suspect that the brown seeds are also smaller and more shriveled than the black ones.

I have found out that green seed collected from the calyx will act like your brown ones. However, if you harvest green seeds which have some ripening initiated by exhibiting some brownish areas, these are much more likely to germinate, since the seed has begun to develop the necessary enzymes. Then the odds of germination improve the longer they stay in the calyx and the browner they are prior to harvesting.

Thnaks Rich - I was sure you would know. Actually the brown seeds were exactly the same size and smoothness as the black seeds, so it was not so obvious to me. But my gut feeling told me that they may not be viable. I purposely left out pale brown ones or anything that didn't look mature.

Collecting seed can be quite laborious as you are often picking out individual seeds. So I think I won't worry about the paler looking ones in future.