It's not neccsarily temps per se, but rather frost. Any extended temps under 50f will cause the banana to go into dormancy. It won't hurt it at all. Light frosts will nip at the leaves and cause some browness and deadness and the end of leaves. Prolong frosts will damage and kill leaves. Freezes will then kill the psuedostem. If the freeze reaches the corm in the ground, the banana is dead. I would bring them in when temps start getting to the 40 degree mark and lower at night.