We keep agapanthus in the ground here year 'round. They do die back almost all the way during the winter. Wilmington is zone 8 and I bet you don't get much colder.
Unless someone closer to you says different, I'd say go for it.
Of course, I am answering this over a week after you posted it, so maybe you've already taken the plunge...
Barb