I was told by someone in my area that clay soil actually retains water
better than any other type. The soil here is mostly clay so I add a lot
of leaf mulch and manure to it.

A loamy/sandy soil doesn't retain water well but has some benefits when
watering. That is it absorbs the water much better than a clay soil. By
the time the soil needs watering here the surface absorbs water so
slowly that a lot is lost to runoff and evaporation.

So for you I would say yes to adding some clay to your soil. A good mix
of clay, manure, compost, wood chips, leaf mulch, and the already
present sand would do just fine (and maybe some lime if your soil is
too acidic).

Lumps of clay are fine as long as they aren't too large. My garden has
lumps about the size of a marble after some deep and thorough tilling
that I did this spring when I added to my soil.