That is most likely the herbaceous root stock sprouting from the nurse root of your tree peony which was likely not planted deep enough.

If you want to fix this, I would suggest pulling out all the herbaceous stems, then digging out the entire plant and planting it at least 1ft deeper. This can help prevent herbaceous sprouts from coming out, and it will give a chance for the woody tree peony part of the plant to produce its own roots which will eventually result in the tree peony growing independent from the nurse root. Your tree peony will decline in a few years if it doesn't produce its own roots. If you plant it deeper now, it should start growing some root hairs this fall, but it might take until the fall of 2013 to really produce a good root system.

I've attached a blog of exacly what I am talking about.

Here is a link that might be useful: blog from Cricket hill garden