I'd just want it as a garden though! :-)

So am I correct in thinking that the legal reason is that turning a
formerly farmed field into a garden counts as a 'change of use', even if
it
wouldn't involve any actual building, the same way as converting an urban
garage into an office does?

Exactly! The change of use from agricultural use to domestic requires
planning permission. The are numerous pieces of legislation affecting
planning and it is not practical to quote them but it is the same basic
stuff as if you want to build a house. If you want to do it go and talk to
your local planners who will give you an idea of the likelihood of the
request being granted. Unless you are wanting to take a large amount of
prime agricultural land then DEFRA will not be involved.