What you have is not a worm, but the larva of a gnat like fly. There is not much you can do once the larva is in the blossom, any control must be done before they get that far. Look for egg cases on the leaves, underside usually, and squish them. If you can see the adult flitting around you can spray with Insecticidal Soaps, but be careful about spraying any broad spectrum poison which will also kill off the beneficial insects that will eat these pests.

I have been finding these caterpillars eating my rose buds.
{{gwi:270863}}
{{gwi:248899}}

That looks the larva of one of the Sphinx moths. When they are less than half that size a spray of Bacillus thuringiensis - Kurstaki (BTK) will do a lot to control them, now a pair of pruners will dispatch them very efficiently.