If you have a lot of peat moss in there or the soil is not well draining, then that might be the cause of decline. I really doubt any kind of peony would have troubles with hardiness even if it is in a pot in your climate.

Some peonies will die back every year, while some types will grow a shrub or small tree. They also do better in full sun however blooms usually last longer when there is shade.

What kind of peony did you buy?

Here are some of my 2nd year tree peony seedlings looking fine right after a pretty cold winter which bottomed out at 2F with many days in the single digits. They are definitely hardy enough for my climate as they are coming out unfazed after a very long winter which still doesn't seem to be ending.

Peonies are very hardy as long as they are completely dormant, or when the buds are still red.

Here is a link that might be useful: 2nd year tree peony seedlings