My question is who has the final say on whether or not the fence is
replaced?

Most deeds have some wording to the effect of "land and all improvements". The
fence is an improvement and therefore belongs to the property that it sits on.

Gary Brady
Austin, TX