The stunting, drooping and dropping imply a water mold problem or other issue with the roots. While very shade tolerant, often able to be grown in some pretty rooty and dry places skimmia actually likes fairly good living with decent soil, some moisture and good drainage.

After you pop it out you might want to remove and replace the soil around the hole. Shaking the dirt off of the carcass onto ground you want to use for growing other plants should probably be avoided.