I have been growing plants in both peat / perlite and hydro side by side without any issues. The peat is sterile before plants are planted in it, and that is part of the reason why I have no issues.

The biggest problem is of course bugs. And once you get them, they can hide in the dirt, and then are almost impossible to get rid of.

One of the things I did was buy a bug zapper (outdoors model) and hung it up in my grow room. It only comes on at night, and has a black light. This gets rid of any larger flying insects that get in through the doors and windows.

Last year I bought some hot pepper plants from a nursery and washed off the roots and put them into the hydro garden. To have the plants adapt to the grow room, I leave them in the pots for a day or so. I am sure you can guess what happened. I lost all my plants due to numerous bugs.

Introducing plants from the outside in dirt is worse than having plants already inside and in good health and adding hydro to it.

I would say go for it, just be careful.