You may have gone from inside conditions to sunny outside to abrubtly. Keep them in the shade until they adjust. Do not fertilize until they are growing well again. Al

Even very hardy plants that have been grown inside need a period of time to adjust to the different light and temperature conditions outside. The bigger the difference, the more time they need to adjust. This is called hardening off. (BTW, the same priniciple can apply when moving houseplants back inside in the fall. Softening off?)

Next time, start by putting your potted plants outside in the shade for an hour or two in the daytime. Next day about an hour longer. Work your way up to all day outside, and inside only at night. Then you can leave them outside overnight when it doesn't freeze. By then you should usually be able to plant them in the ground, unless there are hard freezes still possible. Hardy plants will probably withstand a light frost by then, but if you're nervous, cover them with an old sheet for the night when frost threatens.

Your peonies will probably survive, but will not perform as well this year. You might want to try covering them on frosty nights but I'm not sure how much difference it will make at this point.