Four branches are doing fantastic, tons of new growth, and growth of leaves at the base of the plant, but the majority remains empty. More sun perhaps?
The first thing that comes to mind is a case of badly congested roots or a N deficiency - a fertilizer with too much P can do it, too. How long have you had the plant? fertilizing with what (NPK %s please)? Did you repot or at least pot up if this isn't the first year you've owned the plant?