When the leaves are off.

Umm. That's an evergreen.

Instead, during the fall -- at the very least, during much cooler weather than we now have.

Another excellent argument for using botanical names :-) Japanese aralia = Fatsia japonica.

I still say fall.