It looks like lace bug to me.

Whitish specks on the upper surface of leaves and dark spots varnish-like on the bottom are symptoms of Rhododendron Lace Bugs, Stephanitis rhododendri, and Azalea Lace Bugs, Stephanitis pyrioides, small insects with transparent wings on under-surface of leaves.. This insect hatches early in spring as the new foliage begins to mature and its numbers may build to damaging levels with successive generations. Lace bugs reach their peak in late summer and do their worst in sunny, exposed sites. Spiders are important predators of lace bugs and since they shy away from sunny, hot places, plant your azaleas where there is some shade. Insecticidal soap, horticultural oil, or a systemic insecticide may spare your azaleas the damage if applied in spring when the first hatchlings are noticed. Care must be taken to spray the lower surfaces of the leaves where the lace bugs live. Moving a plant to an area with more shade may alleviate the problem. Lace bugs are more prevalent on certain varieties. The following azalea cultivars have resistance to azalea lace bug: ´Dawn,´ ´Pink Star,´ ´Ereka,´ ´Cavalier,´ ´Pink Fancy,´ ´Dram,´ ´Seigei,´ ´Macrantha,´ ´Salmon Pink,´ ´Elsie Lee,´ ´Red Wing,´ Sunglow´ and ´Marilee.´

The ironic thing is that lace bug is more prevalent with too much sun. The normal advice is to move into more shade.

If this is the only problem, then moving into more sun would make the problem worse. If you have other problems, what are they?

Here is a link that might be useful: How to grow azaleas and their problems.

Only other real issue is lack of any real growth over the last 2-3 years... The plant existed when we bought the house, was in another shady location at the time under a white birch and shaded by the house... Maybe I'll try treating for lace bugs next spring before doing anything else... (We do have plenty of spiders in the vicinity of the current location)

I'll also have to keep a closer eye on the plant next season to see when the spotting starts to appear and see if I can spot the lace bugs...

As for the lack of growth, I do remember some growth early this spring that was easily spotted by the light green. It's possible that the lack of growth was due to improper pruning...

You didn't mention if you were on the West Coast. If so then it could be thrips. Thrips are characterized by a silvery white discoloration on the leaf's upper surface and silvery on the bottom with small black frass specks. The damage is similar to that from lace bugs. Thrips are a problem in warm and dry climates like California and New Zealand. Most of the standard insecticides and insecticidal soaps can be used. Control for thrips whether an insecticide or insecticidal soap is a contact control and must be applied on the adults. This is most likely done in May in California when they are visible. Locally, biological controls may also be available. It is best to remove infected flowers.