R ponticum seems to like clay. It grows precisely in pockets of clay in places like the North Downs - eg there are large stands of it in clay pockets on the North Downs and Chilterns.

In general, rhodies don't like clay, because they need good drainage. They are shallow rooted, however, so you can create a patch of well-drained soil above the clay, and it doesn't have to be a mountain, a foot or two will do. So I think what you have to do, somehow, is create a well-drained acid bed. You can do this as a raised bed. Or alternatively at the top of a slope if you have any slopes in your garden. The soil you import to make this bed should have plenty of humous, eg, mix in bark chippings, compost. It needs to be on the acid side of neutral, so you need sand (sharp gardeners sand - because it is acidic, avoid builder's sand it can be salty), ericaceous compost, perhaps some perlite (more acid than vermiculite). Now the problem of doing this is that it will tend to dry out, so you will probably have to water it a lot, and for that you'll probably have to collect rainwater so you can avoid watering them with tap water, especially if you are somewhere like Croydon that has very, very chalky water. Finally, choose some rhodie varieties that are less fussy about being very acidic than others, any specialist should be able to tell you.

The reason you have to go to this fuss is because, as the evidence of your neighbours' gardens is telling you, you are not in an area suited to growing them. But, by doing something like this, you can defeat the reality.

Fwiw, although I can grow a few rhodies, I have more difficulty growing hydrangeas. I grow the rhodies because my soil is very well drained, and I can modify it to be sufficiently acid locally. But hydrangeas are gluttons for moisture, and I just can't keep them sufficiently wet.