I would remove now. Whatever weeds are growing will flower and produce more seeds if you let them grow until fall. Grass is a real problem once it has invaded a garden bed.
At times I have had to dig up a few good garden plants in order to remove something invasive around it. I then re-planted the perennials.

Thanks Carol - oh my aching back, I can feel it crying out in pain.

It is never too late,or too early, to remove any plant you do not want growing where it is growing, a "weed". However, you need not, necessarily, dig those "weeds" out and a layer of newspaper covered with a good mulch to hide that newspaper is a good way to do that and much less work and an added benefit of a mulch, besides "weed" suppression, is adding needed organic matter to the soil, aiding in maintaining soil moisture levels, and aiding in keeping the soil cooler.

Thank you "Kimmsr", my neighbor does the newspaper and mulch and I have seen it work for her.