Hi,
Yes stink bugs sure do cause that. I have been having the same problems with my p. vines this year-loads of stink bugs then bloom drop. At first I just picked off the stink bugs and threw them in soapy water-they sink like a rock. If there aren't too many ,this could work. But I had too many so I used NEEM on my vines that have edible fruit. You could try a homemade oil/soap spray as well.
If the dew (we have alot of dew here near Erath LA where I live) washes it off, you will need to reapply every other day until you get the prob under control.
For my passies that dont produce edible fruit I used ortho systemic and that flat out took care of the bugs on those vines. Don't use this if you want butterflies to lay eggs on your vines.
~SJN

Thanks SJN! They were DEFINITELY the problem. I tried picking them off but when I saw that they had laid eggs I sprayed. None of my passions are fruiting (I have a ruby glow, a blue-eyed susan, and a macrocarpa). I think in the process I killed off my caterpillar population but hopefully they come back and the stinkbugs don't! Again, thanks for your help!