It's not clear from your post what kind of vitex you're dealing with. If it's a vining type, like the notorious beach vitex, then its tendency to spread indefinitely is part of the characteristics of the plant, and so there's always going to be a certain amount of regular maintenance to keep it away from other plants and other places it's not wanted. If it's one of the more common upright shrub/tree types of vitex, though, then it's a little more complicated. They grow vigorously and flower abundantly on the new growth of the season, so they can (and perhaps should) literally be cut all the way back to a stump every year in March or April, which will help to limit its size. It's good to deadhead them in mid-summer (around now), so that's another opportunity to keep its size limited. And almost any amount of trimming at any other time of year as necessary is unlikely to ever harm the plant, so theoretically you should be able to keep it at any size you want if you're willing to prune enough. But there's no getting around the fact that they're big, robust plants by nature, and for that matter, so is the cryptomeria. So, if they're conflicting now, it's not likely to get a lot better on its own. If you have the opportunity to move the vitex now, and have a spot for it that's very sunny and not too damp where it can be allowed to develop more fully while still being aesthetically pleasing, then you probably should do that. But, if like a lot of us, you have a yard that's a little too small, a little too crowded, and a little too shady, then you might have to make-do with it where it is. Either way, they're wonderful, beautiful plants, and very forgiving of just about any treatment.

Kristan, I don't think you'll be happy if you decide to main this plant to suit a small space. Can we assume that you are talking about the ornamental Vitex agnus-castus? Find a location where your plant can be trained to a tree-form, but left to get as tall as it wishes.

I surely wouldn't move it now, as it is likely blooming and is in full foliage. The transplanting of deciduous trees and shrubs is best undertaken in the dormant season.

Let me know if you need some good links to proper transplanting procedures.