Your neighbor has the right to fence his property. He probably has the
right to put the good side facing him. After all it is his fence and he
paid for it. You do not have the right to cross the property line or to add
dirt to his property. If your dirt is washing into his yard he probably is
very unhappy. You are crossing the line and liable if he wishes to take
action. You may put up your own fence on your property if you don't like
his.