I'm in zone 9 and we only experience a few days subzero each winter so
you'll have to bear this in mind. My olives in pots (Olea europea)
survive very happily in an unheated greenhouse.

Olea europaea is supposed to be hardy to USDA Zone 8-10. Frost Protection:
Hardy to 10°F ( -11°C). If it is exposed to much frost, the leaves bronze.

I wonder if their growing conditions, from germination onward, helps them
to become a little hardier?

Probably not. It's genetic.
Iris,
Central NY, Zone 5a, Sunset Zone 40
"If we see light at the end of the tunnel, It's the light of the oncoming
train."
Robert Lowell (1917-1977)