If I were to buy a little bit of an adjacent field from a farmer to
extend my garden would I therefore need planning permission to grow
anything but grass ?.

The idear might be, it seems building land is expensive, agricultural
land is cheap, buy a home with a small garden next to agricultural
land, extend size of garden.

Paul
Know somebody who did exactly that in Little Kimble. After a couple of years
they had a beautifully planted garden plus summerhouse in the corner of
their paddock.
The council served an enforcement order and the whole thing was ripped up
and restored to long grass and hogweed.