I've always been under the impression that rosemary isn't a host plant for pests (like caterpillars, moths, etc.) because it never seems to be listed. It seemed that the strong fragrance might be a deterrant.

So imagine my surprise to find a looper caterpillar sitting inside my rosemary plant. The next day - looking deeper - I saw an adult/moth, and a pupa cocoon. They all seem to match up with soybean looper pictures. Since there really aren't any flowers around, I bet the perpetual fragrance stands out, and attracts them.