I agree with Alexander and others. Have a survey done, and if there is
encroachment, draw up a piece of paper which states the exact details of the
encroachment, attach a copy of the survey, and make the neighbor sign it.
That way, if you go to sell your house and the bank or buyers make a stink
about the missing bits of land, you can pull out the sheet of paper and let
them know that everyone agrees there is encroachment - therefore, if the
buyers need those extra feet, they have the right to make the neighbor tear
the fence down.