Our last frost date is usually May 15th (I'm 3-4ish as well). The forcast is looking pretty good for the next week. So I'd go ahead and plant out the peonies if it were me. My peonies that are in the ground are a good 6 inches tall and just starting to leaf out, so that should be a good sign for you.
If you would get a night where the temps would drop down to low 30s you might want to consider throwing a sheet or plastic over them for the night just in case.

As for the tiny seedlings, sounds like they are ready to go out too. Having them outside for the week should have hardened them off nicely.
Once planted out, be sure everything gets enough water for at least the first month. I tend to baby my newer plants for the first season (watering) just to give them a good start. After that, things are usually on their own in my garden.

Columbines are surprisingly frost tolerant. You will have no problem with them, even if we still get a freeze. Not sure what kind of poppy you are talking about.

Well, I put a couple tiny colombine seedlings in and they seem ok so far - despite how windy and cold it was today.
I'm not sure what kind of poppies I have either - I planted about 4 different kinds - some of the red orientals, another perennial one, the carnation type (annual I think?) and watermelon - but then my goat got out when I had them outside and ate more than half, including the markers, so I'm not sure what the ones that are left are!