There are many reasons for fruit drop. The first consideration is pollination. Many avocado trees require two different cultivars to insure pollination because they are either type A or B flowering. Having a second plant of the opposite stage increases the potential for pollination. There is a need for both a source of pollen and the receptive female phase of the flowers, plus a pollinating insect to achieve fruit.
Small fruit, subject to low temperatures, will likely cause the developing seed abort. In most cases the fruit drops, but in some instances the fruit continues to develop without the seed enlarging - this fruit is termed "cukes" by growers and sometimes sold as "cocktail avocados" by enterprising produce people.
Strong winds produce a wind chill effect on the plant and result in damage to small, developing fruit by lowering its temperature below what an air temperature reading reports.
Try constructing a 3 sided plastic wind break around the tree and see if this helps. You might also see if there are sources of named cultivars of avocados in your area and graft several different selections to replace some branches.
Good luck.
Claude Sweet
San Diego, California
USA