I feel your pain! I just moved from a home, where I grew lavendar with great sucess and am about to try it for the first time on a large deck on the 2nd floor of my apartment. I have no idea what will happen, and like you, love this flower.

The only thing which I can think of is that you aren't getting enough sunlight. Lavender does require quite a bit of it to grow properly, from what I remember. It's natural habitat is the mediterrean, which gets loads of sunlight, think the Provence area of France for instance. If your balcony is on the shady side, it's no wonder you aren't having sucess with it.

Light could be a problem, also make sure that you pick a species/cultivar that's hardy enough. look for zone 6 hardy (one zone colder than you) - then wrap/mulch your pot in the winter.