I haven't had that problem too much. Up until recently, I've always just used 2" net pots so the 2" size always worked fine for me.
Can you buy a 3" hole saw? I ended up having to late pot some tomato cuttings and couldn't get them into a 2" pot.

I own a full set of hole saws all the way from about 1/2" to about 7". They are available at the box stores. If you buy a set, be sure to get an arbor that will let you change sizes without having to use lots of tools. Mine use spring-loaded detents that slip into holes on the back side of the saw bits.

By the way, if you get into anything larger than about 2", run the thing backwards first and get a good little groove going before you try hogging off lots of material or it'll try to break your wrist!

I use a drill called a hole hog and it has a full 2.3 shaft horsepower! I own a nice gold crown courtesy of the thing -- be careful!

I bought a nested set that goes to 3" at Menard's (a hardware store). They aren't all that sturdy, so I only use a somewhat weak electric black and decker drill for it. Using the corded drill with much higher torque will probably break them and my wrist both. I agree with Freeman on running it backwards to start with on the bigger diameters, but keep in mind that all things need to be tight as possible. You don't want the saw portion coming loose. I did that. Once was all it took. I felt a little on the stupid side and the first thing I did was look up to see who saw me do it.

I've found that starting a hole and using a dremel with the mini jigsaw attachment works excellent, too. That way you can make the diameter slightly larger and still catch the lip of the net pot. It also works great for cutting siding and drywall holes.