Here's my yard (via google earth)

The blue line is our fence (chain link) and we do have a gated driveway. Anything with red x's through it is no longer there. All placements are approximate but do give a good idea of where things are, except for the azaelea hedge on the eastern foundation, I drew that as growing through the bedrooms when it actually forms a right angle with the hedge that runs east-west. Oops!

The tan amoeba-like shapes are mature oak trees, each one about 50+ feet. The two on the west are deceptive, I mistakenly outlined their shadows as well. Sorry.
There's also a long-leaf pine on the west side between the two oaks that my children named grandpa tree after grandpa moved away (yes they give it hugs and kisses and no I don't know why they chose to honor it so) that gives us almost enough pine straw to mulch both beds with each year, a 30' tall 40 year old magnolia in front of the picture window and a 20' tall 10 year old phallic "cedar of lebanon" (looks like a plain old arborvitae to me) about 25' due east of the magnolia. Those are the tan circles. The large trees are staying. They were here long before the house was, and I can't see cutting them down now, but I'd happily axe the arborvitae. The previous owner removed a "sick" maple (it's leaves turned purple in the fall so it must have been phosphorous deficient like tomatoes are when they turn purple, right?) and replaced it with the arborvitae. *sigh* Goodbye, Norway Maple, I wish I could have met you.

About 15' east of the "cedar" is a baby fig that I planted this year, it's not on the map because it's knee high. :)

The bright blue circles are dogwoods and crepe myrtles that are also mature at around 15'. The green circles and squiggles are azalea hedges/individual shrubs that are about 10' tall. I'm amenable to most of the shrubs being removed (especially those on the east side) but the ones along the fence will remain although they will be trimmed to fence height.

The red triangle is my flower bed (16x20 on the straight edges), and the red box is my 8x4 raised bed. The raised bed, even though it is under the canopy of the largest oak tree in my yard, gets a solid 9 hours of sun in the afternoon. The flower bed is shaded by the house in the morning and receives full sun (at the point) to ten hours at the back. The magnolia tree doesn't produce much shade, the previous owner gave us the parting gift of lopping every branch off at the trunk, cutting a chunk out of the trunk, and drilling two holes completely through the tree the week before we closed. It's healed the trunk damage and has started putting out new branches, gaining about two feet in circumferance per year, and deserves to stay after recovering. It's flowering for the first time in the 4 years we've lived here. If you look closely you can see the tiny shadow of the magnolia between it and the driveway, it must have just been planted. I cried when I saw what had been done to the tree. Does anyone else get emotional over their landscaping, or is that just my own particular brand of insanity?

The two yellow boxes on the NE corner are two sheds that share one concrete slab, that corner of the yard has a wooden privacy fence that extends about five feet past the sheds on either side.

For scale, the driveway (those two dark blurs are cars parked in it) when measured in a straight line from the edge of the roof to just shy of the street is 55' long, and the magnolia is 22' from the edge of the roof to the trunk and the fence is about 10' beyond that. The area outside of the fence is drainage ditch that the county clears out biannually, anything planted more than 5' from the fence line is removed.

Judging by the shadows of the trees, the picture was taken mid-afternoon, I'd say about 4, and it's nearly 40 years old. Surprising how little has changed, really.
So, what do you think? Is there potential? Would a yard sized potager be too much? What would you do? Help?