You should remove the spent flowers only, both for appearance and to prevent wasting energy producing seed. Leave the plant as is until the foliage yellows in the fall. I like to cut them down to about 4 inches, leaving the stiff stubs to protect the new buds forming for next years bloom, from digging animals over the winter. Al