Additional picture

Try looking at night with a flashlight.

It might be more than 1 pest & perhaps disease as well. Some pests introduce diseases when feeding. The rough patches & distorted leaves look like the damage of some sucking insects.

The dark blobs appear to be caterpillar frass (feces)

FWIW & if this is any consolation, plants can usually withstand a loss of up to a third of their foliage

Looks like spider mites, perhaps. They are difficult to see. You can take a leaf and shake it over a white piece of paper or use a jewelers loop or a magnifying glass. Look under the leaf. Good luck

I don't think spider mites can poop that big.

Take a sample to your county ag agent. They most likely will be able to identify the culprit. And provide a remedy. Good luck. Let us know.

Thanks everyone.

Right now, I am spraying it with neem.

Have you noticed any galls? Those are caused by mites. But the bucida caterpillar (characoma nilotica) moth lays her eggs in them. The caterpillar can chew up leaves, feeding externally, or feed in the galls like a boring caterpillar, causing a lot of frass (poop) that is seen. The frass can also cause a lot of staining on a sidewalk or drive. Check your tree really well for galls and break one open. You could find the caterpillar in there since you're not seeing it on your tree. What can you do? The window for control is when the caterpillars are small and feeding on the flowers. Try using BT then. I hope my post helps you.

Oh yeah, BTW, that small black fly could have been a tachinid fly. Tachinids attack caterpillars, adult and larval beetles, as well as some other insect pests. They also attack cabbage loopers, which makes them friends of mine even though they're a danger to my swallowtail caterpillars. So that fly wasn't part of your problem, but maybe your friend too.