few female flowers, try hand pollinating. the immature fruit turning
yellow & dropping sounds like lack of pollination

I'm not sure that is the problem - I see loads of bumble bees, wasps and
ants in every flower.

The trouble is - there are no female flowers. I see the miniature version
on the plant now and then, but they never seem to get big enough to actually
flower. They get yellow, wither, and fall off. (including the fruit part)