Who owns the fence? If the posts are his side, which it
sounds like they are - then maintaining it is his problem
and not yours.

Not necessarily so. As you say talk to him.