Yes, knock it down for safety reason, if you think it's dangerous from getting stung, do it in the dark with a flashlight quick. They should all go somewhere else in day light,..give it a couple of days. The queen would probably start a new colony, worker bees might be joining other colonies like honey bees do, or at least live their lives as solitary bees, still helping to control pest insects and or pollinating etc.

I only see one bee no others yet.He/she keeps coming out thru its tunnel looks like a tornado. I see it fly away alot is that a good time to do it. Also its late may here in CT can they start another house?

It's probably the queen bee just starting, yes, still early to start over a new nest.

whats odd I have not seen the bee at all in 3 days? Did she leave the nest and go elsewhere or what. Was going to knock it down but was going to do it when I saw her leave but haven't seen her at all?