Sometimes plants that have to try hard (but not too hard) are encouraged by the situation to put all their energy into flowering and fruiting, because they don't expect to live long. Move them into a comfortable situation, they'll know they have longer to live, and put more energy into surviving and less into reproducing.

The situation is well known in figs. You have to confine them and put them on a pile of rubble to encourage fruting. It is not because they require these conditions to grow well. Give them an easy life and they will take to it, grow vigorously and healthily, but barely fruit at all.

A plant growing in a crack in a rock next to a stream may be getting natural trickle irrigation deep down, and may have access to particular mineral nutrients. It can be very difficult to reproduce sufficiently similar conditions in the garden.