Some gardeners cut them back when the flowers begin to appear, so that they can be more visible.
Others, myself included, wait until the new foliage begins to appear, before removing the old. This permits the old foliage to photosynthesize and provide the plant with nutrients, without interruption.
If the old foliage is tattered and/or turning brown, I sometimes remove those.
Whichever method you choose, probably won't affect the plant very much.
Enjoy! :Rb

I just cut most of mine back today. Not all of them were leafing out yet, but some had started, and the new foliage was showing some signs of fungus (or whatever turns the leaves brown and limp; I assume it's fungus). There were a lot of wet maple leaves around the crowns, and the emerging foliage seems to be particularly susceptible.

I was a little worried about exposing all the volunteer seedlings in these beds to the inevitable cold snaps that are likely to be ahead, but figured it was less of a risk than having them rot under the leaves. The beds look pretty naked now, but I know they'll be full of new foliage and flowers soon.