The symptoms you describe sound alot like dogwood anthracnose. A dead giveaway
is brown spots on the leaves with purple edges around them, and fungal spores
on the one and two year old branches.

Toad