Go to nursery, they point me to different directions depending on whom
I ask, they are few time more expensive than the tree itself.

Well, I don't know the answer, but at the risk of stating the obvious,
your choices include:

(1) go to someone who is likely to know more and less motivated to
sell you something. Probably a county extension agent (since you
say California, not sure what the equivalent, if any, outside the
US is).

(2) Ignore it (at least in the sense of not trying to intervene) and
see if the tree recovers. This will tell you whether the problem
is a serious one or not (some problems which look kind of bad will
solve themselves, others won't). But this may be nervewracking.

(3) Browse through some lists of diseases like
http://www.nysaes.cornell.edu/pp/ext.../disindx.shtml
http://www.nysaes.cornell.edu/pp/ext...fruitkey.shtml
(there are many more like this, including with photos; I don't
know which are bets).

(4) Post (via one of those photo upload web sites) photos of your
yellow leaves, the holes in your leaves, the dark branches, the
young flies, the fly droppings, the adult flies if you can, etc.
Perhaps someone else (more knowledgeable than I) will recognize
what's going on, more easily from photos than from a description.