You are correct that in soil science they define these things by the size of the particles. But that doesn't exclude what I said, at least in a practical and broad sense.

Most rocks contain a lot of silicate minerals and/or a lot of silica grains, so in practice the sand in your garden soil will be mostly silica, or at least a source of silica. Things might be different if you are gardening on a volcanic soil, or the spoil heap of a mine, etc.

Silt can be formed by mechanical weathering, but to get clay we need chemical weathering. So clay (meaning small size soil particles) will reliably be clay (meaning one of a range of aluminium phyllosilicate minerals).