I recently removed a section of my asphalt driveway to make way for a
small garden. My plan was to remove about a foot or two of the very
poor looking dirt under the section of drivway and replace with
topsoil.

This dirt is mostly rocks, clay and sand, but contains small pieces of
asphalt and asphalt dust. I thought I might spread it over a section of
lawn that I was planning on turning into a raised bed. I would then
cover it with topsoil.

My question is, will this layer of "stuff" hurt the plants that will
grow above and though it? Could it hurt the rest of the world below it?
Is this a bad thing to do?

-Rick