Hardiness should not be a problem with Clethra alnifolia in Toronto. Literature indicates zone 4 and in some cases zone 3. There are Clethra alnifolia growing here in Kingston on the RMC campus that are thriving.

It does best in wet or moist soil on the acidic side but will tolerate occasional dryness. It languishes in perpetually dry soil (ie. under a silver maple)

Simon

Mine has survived and I am in a little bit colder zone. It is in a slightly sheltered spot. If you try it again, you may want to protect it in winter.