My lime tree has two main branches. One looks healthy and the other was been looking worse and worse over the past few weeks. I checked it today and it felt rotted in places near the base of the branch. I picked the 40+ limes off of that branch and cut it off. About 1/5 of the cross section of the branch looked healthy. The rest was dark or dead. There are ants around the rotted part of the branch and there's still some of this extending further down into the tree. Does anyone have any idea?

I have some white mildew stuff on the leaves as well. I don't think it's related but worth mentioning.

I've posted high res pics here:
https://www.dropbox.com/sh/wflbnt0jcsy6k5p/AACpHHSoqDoIrCxL7z-FxXana?dl=0