You don't say where in UK you are,
You will find Olives growing quite close to Granada in Spain and they have
quite cold winters there (Av Jan Temp 6C)
Some say you don't get good oil from Olives till they have been frosted.
You get different directions for growing olives from different places. as in
the 2 following extracts..........
Soil:

Olives generally prefer a free draining soil with reasonable moisture
retention, but some cultivars are more tolerant of heavy soils. Local
rainfall patterns tend to simplify or compound problems associated with
difficult soil conditions. Overseas heavier soils are usually associated
with Mediterranean climates which suit the Olive better. This results in the
moisture patterns required for optimum production. These heavy soils also
tend to hold trace elements better than many of New Zealand's lighter soils,
so trace element requirements need to be considered on a site by site basis
in New Zealand. Olives require the full range of soil nutrients but not too
much nitrogen. Careful selection of seaweed fertilizers will address the
trace element requirements as well as the general nutrient needs.

Microclimate:
Every area of the country is often considered to have it's own general set
of growing conditions but within these areas there are localized
microclimates. Even within the average garden there are microclimates that
will suit some plants better than others. When choosing a spot for Olives,
the basic requirement is warm dry summers and cool winters, that are not too
wet. If high rainfall is to be expected, then good drainage becomes more
important. Most of the country experiences enough chill to satisfy Olives,
but late frosts at flowering time can be a problem.
Shelter is important so that trees can establish well and do not damage as
they mature. Some air movement is needed for pollination as well as to
dissipate frost pockets during winter.

Culture
Olive trees have been grown since the Sumerians and were know in Egypt from
the VIIth dynasty (2400 BC). They require a mediteranean climate of long
dry, hot summers and mild winters with a lot of sunlight and temperatures
above 12°C. The ripening of the olives requires strong sunlight. Winter rain
does not both the robust tree which needs at least 200 mm of water. The
olive tree thrives in most soils except those that are humid. Nevertheless
they prefer stony soil where they give the best oil. Of the 5.5 million ha
of olive trees that are cultivated throughout the world, a third can be
found in the Mediteranean bassin, mainly in Italy, Spain, Greece, Turkey and
Syria.


--
David Hill
Abacus nurseries
www.abacus-nurseries.co.uk