Any downsides would be the same as with any other perennial :-) Ongoing maintenance - the need to cut them back annually, divide occasionally; an ability for some to spread or reseed more than desired; the possibility of some just not thriving, etc. OTOH, routine maintenance like deadheading (unless the freely seeding type), watering after establishment and fertilizing is generally not needed.

I tend to prefer the evergreen species as these generally require even less maintenance and attention than most but a tall clump of a well grown Miscanthus in full fall bloom is hard to resist :-) My current garden just doesn't have enough sun or enough room to support one of these so I tend to focus now on the carexes, which are perhaps my favorite genus of all OG's. But I do use the full assortment of available OG's when I design gardens for clients and in much the same manner you intend - incorporated into a low-maintenance mixed border of shrubs, grasses, perennials and groundcovers.

I agree with gardengal, grasses have some of the same maintenance requirement of other perennials.

In addition, I'd advise to plan and place according to mature size. The mistake I see most often is the placement of a small 1 gallon grass in a location that is totally inappropriate for that mature grass.

Also, choose the right grass for the right location. Check zone hardiness carefully, sun/shade and water requirements. Is it a clumping or running grass? Does that particular grass self-seed?

I'd recommend Calamagrostis Karl Foerster, as well as Panicum varieties in your area. Check out the Karl Foerster plantings around BWI.

I love ornamental grasses and have lots of them, especially alot of native prairie and woodland grasses. Overall I've found them to be easy to grow and relatively low maintenance.

The downsides I can think of - some of them get HUGE, can outgrow their position, and become very difficult to transplant or divide. Others get floppy and fall all over their neighbors - this past Spring I gave away 2 large Miscanthus 'Variegatus' because they had gotten huge and would get so floppy they were covering the surrounding perennials, and I was sick of them (although they have pretty foliage). I like to leave the foliage standing through the winter, very ornamental, however the fronds of some will degrade and fall apart, they can get messy and blow around the yard. Occasionally they have gotten attacked by rust.

I echo Grass Guys suggestions with the addition of Little Blue Stem - a great grass for a naturalized landscape. Easy care, super drought tolerent & gorgeous fall colour.
If you want some landscape ideas check out some of the pictures in Rick Darke's 'The colour encyclopedia of ornamental grasses', available in most libraries. He has some great examples of natural plantings. AND I highly recommend looking at Ohme & vanSweden's web site for some ideas. In my opinion, they are THE master's of using ornamental grasses to create 'better than natural' looking landscapes!
Oh, and on the maintenance downside - if you are going to have a lot of grasses you had better figure out what you are going to do with all the trimmings in the spring when you cut them back. It will amount to quite a bit of material! there are lots of options for that problem but, that is a whole other subject. (which has been commented on before if you care to search the forum)
a.

Here is a link that might be useful: Ohme & vanSweden