It would really help us help you if you posted pictures of the tree. Close up pic and one of the whole tree, especially showing the trunk near the soil where it would have been grafted.

If your tree is grafted, and 80-95 % of citrus trees sold are grafted, then your story unfortunately sound a lot like the top of the tree, the "sion", the part that give you the "good tasting" orange, probably died at some point, and the root stock has taken over.
For the most part, even though there are different kinds of root stocks, most do not produce good fruit, bitter fruit is pretty typical of them.
If that's the case, you should just pull it out and plant another tree, and keep a close eye on the graft line so that it doesn't happen again. Anything growing below the graft line should be pulled or cut off, anything growing above the graft line is the "good fruit".

Lack of water can affect the taste of the fruit, but I don't think it would affect it enough to make it "bitter".
If they're juicy and the tree is healthy, there was probably enough water. But if the tree is also sickly, then lack of water could be a issue. Citrus trees do need regular water to fruit well.

But pics would help a lot.

This post was edited by BarbJP on Tue, Jun 24, 14 at 13:39