Hmm ... clay isn't usually crumbly (or able to be crumbled). What sort of colour is your soil? Remember loam is (roughly) clay + humus - is it possible your soil is better than you think?

Replacing soil with better soil can't do harm, but isn't always necessary. Mixing in some good soil or compost at planting is good enough. And a 4ft sq hole is quite large - you'd be better to dig a bit small and use your effort to go deeper. You need to bear in mind that the roots won't spread that fast, and replacing too big an area will benefit the plants around long before the apple tree gets there.

But the trouble with digging a deep hole in clay is that you run the risk of creating a reservoir - very wet good soil, and no way that the water can drain through the sounding clay. I know you say it's well drained, and on a slope, but if you dig into that slope and into underlying clay (if you have underlying clay), will it still be well drained?

No answers, I'm afraid, just some things to think about!