Looks like worm castings to me.

Otherwise it could be sweat bees. Watch and see if you see any bees coming and going.

I have bumble bee (no sweat bees that I've seen at all) invasion on my back porch. Would they do this as well? Also, would they make the hundreds or thousands of those in my backyard?

I vote for June bugs or Japanese beetles depending on where you live. Most places don't get both. Did you have unexplained lawn problems last season?

I see you have oak leaves.

I have both June Bugs and Japanese Beetles and I have seen both emerge from the soil and they do not leave mounds like that. However where I have shredded leaves into the soil I do get worm castings that look like that.

Thanks kimmsr. I'll update my mental image database. What do the bug/beetle emergence holes look like?

Just a small round hole in the ground, smaller than one a wolf spider would make.

I vote worm mounds. I would venture to say if you take a blower and blow all the leaves from the area you will find leaves being pulled down the holes. With Organic fertilizers and mulching you can expect large worm populations.