yes, the soil will probably rot the fence unless the wood is weather
treated. I have picket fence that connected to only a little bit of
soil and after about four months began to rot.

I have a similar situation with people throwing soccer balls and other
things into my flowers. I solved it by moving my nice flowers
elsewhere even though it was on my property. This is really
irritating as the kids that threw the balls just didn't care and the
police were too busy to deal with this. I also stopped from growing a
veggie garden because of this--it's just going to break.

Definately check with city code enforcement and first make sure that
everything that's been done is up to code. Ultimately, if something as
minor as this were to go to court for any reason, you'd be protected
because you adhered to code. My city (Escondido, Ca) has a website
with all their codes in building specifications. When I needed to
build a retaining wall, this was the website I downloaded the
information from. Code violations can be really epensive
(experience!).