I wouldn't remove the stones and rocks, it's a unique landscape that can grow plants and look nice in time. The garden at my currrent house is very standard carefully landscaped bowling green striped lawns and curved flower beds without a stone in sight, and really it's a rather artificial and unexciting landscape to work with.

Look for plants that like poor soil and would grow well there. A few bushes like budlia may work, and you can put down poppy seeds as they grow like weeds in even the worst conditions. I find that the right planting on waste land can attract a lot of nature like birds and butterflys. But, if you want to turn it in to a suburban garden for partys and playing tennis, this isn't the way to go.