I wait until it takes almost no effort to separate it from the plant or when it drops. I haven't gotten much fruit from my vines and quite a bit of it has been empty, just little sacks of pith.
Karyn

I havent gotten any fruit ever it is alway hollow inside but hope this year I will get some seems the GF butterflys arnt here at all so my passifloras are blooming and Im hand pollenating so hope I get some fruit I have never even ate one no clue how they taste.

Think Hawaiian Punch and you'll have an idea of what the fruit tastes like. On the plants that I've had palatable fruit the smell is actually better than the taste. Many of mine are fairly acidic but that might be because they are over-ripe. I was trying to get viable seed.
Karyn

Just wait for the fruit to drop. That's the best way to tell--especially with some incarnata who's fruit stays green the entire time.