To put it in differently, without the worms your lawn will compact and then
start to hold water. Moss will move in and the grass will die. Your
beautiful lawn will look like a patch of dying mud, which is what it will
be.

Leave the worms alone and if the casts offend you, buy a lawn broom and
spread them around. Failing that the best solution is to give up on the
lawn altogether because without worms it won't be worth the trouble.

Paul DS.