I don't have a tamarillo but every tree I've owned that dropped flowers was too immature to set fruit. Sometimes it's maddening because even somewhat large trees I own like my Jackfruit, Canistel, and Persimmon have dropped flowers.