Soak them in soapy water for a few minutes (easiest done in a tub like for parties)- the bugs will flee out. Make sure you let them drain well before you bring them inside. If you want to spray with something nontoxic for foliage, etc, either use insecticidal soap or very diluted oil spray.

Will that take care of the pill bugs and other bugs in the soil? Do I immerse the whole pot and plant in the soapy water?
Thanks!

Yup- i usually do the whole pot, a bit above the rim. Whatever's in the soil will come running out so be prepared. Lots of soap bubbles will kill the bugs- if you don't want to kill them but just flush them out, use less soap. If i use much soap i rinse with a hose afterwards, which helps gets any bugs that may be in the foliage and any dirt that gets stuck in it. Some of your soil will float up, too (especially perlite). I usually scoop out the solids (bugs, soil, whatever) in between plant dips with a fish net. Otherwise it can get really messy. I can usually do 2 or 3 plants at a time in one of those party tubs. I have a ton of plants, so when it gets to dirty, i dump the water and start another round.

Now that i keep a lot of my stuff in a cool greenhouse for the winter i don't fret as much about the roly polys and roaches in the soil (since they'd reaquire them anyhow), but i do the oil spray to deal with aphids, mealies and the like above ground. I still dunk anything that's coming into the house for the winter.