Are you interested in plants or garden design? For garden design, I'd say Kykuit in Tarrytown is pretty much a must see. It's the old Rockefeller house, and tours are arranged through Historic Hudson Valley. I don't know how booked up they get. I went when it was first opened to the public and tickets weren't easy to come by.

For plants, Stonecrop in Putnam Co. is another excellent garden. It's the home of the Garden Conservancy. The Cabot's also owned Wave Hill, just south of Westchester.

If you are going to be there during a weekend, do check out the Garden Conservancy's Open Garden listings.