You didn't dig too soon. Sounds just right.

There is a method that is explained in the "Dahlia Society" website which entails removing all the tubers from the clump and wrapping them in saran wrap AFTER allowing them to dry out for a day or so.
Moisture should not be cocooned with the tuber as it can set up mildew over the span of storagetime.

I have yet to try this method....I shrink from removing the tubers from the clump considering the loss of 'eyes' will prevent their re-appearance next spring.
The clump that is uplifted I treat gingerly so as to protect all the tubers from falling or being knocked off.
I then allow a couple days of storage before they are given a peat moss cover and stored in my coldcellar where they remain cool, dark, and dry.

Shrivelling is part of the tuber's drying out and unless it shows signs of softeness, or mold, I trust it is in good shape for storage.

The time of digging is as you point out....letting the plant be touched by a killing frost which blackens the foliage. Then, all that is removed and the stem is cut back, except 6", serving as a good handle to lift them out when unearthed.

You use a refrigerator to store the tubers.....so I assume then you remove all from the clump, saran wrap them, then store them.

Question: have you had passed success with them doing this way?

The tubers that were planted last spring grew in numbers over the summer. What clump went into the ground at that time, is not the same size that came out. Do you not have large volume of tubers doing it this way---the refrigerator.