It might be that you need to get a bigger pot or move it into the earth.
We have a lovely olive tree in the ground and they thrive in our soil in
Twickenham.

hmm, maybe, but the pots are decorative - to go round the pool in summer -
and i'm not sure about the soil here in e. yorks, lots of clay content ...
d'you think this might be okay?

anyway, after what Alan Gould has said above, i'm thinking i should maybe
put the tree back outside - it's in a conservatory right now - although i'd
be worried that this new growth, being quite delicate at the moment, might
be killed if there is a frost ...