My guess is deer are nibbling off the tips of your azalea branches some time during the winter. Take a close look at the branch and see if there is a leaf or flower bud at the very tip.
The American Azalea Society suggests the following as possible reasons for lack of bloom.

Many factors affect the quantity of blooms on an azalea, beginning with heredity. Some azalea varieties are normally covered with blooms, and some are normally "shy bloomers". Next year's flower buds begin to form within a few months after the plant blooms. If an azalea used to bloom well, and doesn't now, it may be due to one or more of a number of reasons:

pruning after the buds form, which removes the buds for next year
on some varieties, the buds are less cold-hardy than the plant, and may freeze, turn brown and not open
lack of moisture during the late spring and summer reduces bud formation - mulch helps retain moisture and also keeps the soil cool
less than 3 hours of sun reduces the number of buds
a phosphorus deficiency, characterized by dull, dark green foliage with reddening underneath, reduces the number of buds
poor plant nutrition reduces the number of buds