From your description it could be any number of things. What I'd advise you to do is use a natural pest control method to try and rid your trees of any pests, especially around the time fruit is ripening. Mix a few drops of dish soap with a full spray bottle of water and spray the fruit, branches, and trunk. Should get rid of whatever's coming by.