Sounds like the pelletized fertilizer that is often added to potting mixtures. Search for osmocote to see pictures.

I had another look and they're not fertilizer.
They're a lot smaller than osmocote balls/pellets.

Some of the white spores are almost grey.

Maybe it is perlite. Are they a bit crunchy?

I doubt that they are spores. Some of the largest spores are produced by fungi in the genus Gigaspora and even those don't get much more than 50 µm in diameter. There are 1000 µm in 1 mm, so that is about 0.05 mm or a little less than the width of a human hair.