The bird nesting point is important - it is an offence under the wildlife legislation to disturb a birds nest, and that includes "recklessly", ie accidentally because you were heedless of the time of year that you were doing it. And if you come across a nest, either occupied or being built, you have to stop work.

My neighbour has two old horse chestnuts which have been cut back, so that they're about twice the height of the adjacent cottages. It's a few years since they were last done, and they're looking very attractive - normal shape, just not as tall as they'd otherwise be. I'm not so sure about the idea of trimming back the sides drastically - but I suppose if it's on the neighbour's side and you can't see it, it's up to them.

But definitely work for qualified arboriculturalist if you want it to look good on your side

Since you're happy for the tree to remain, if the neighbours are being difficult, you could ask advice from the council tree officer. It's hardly your fault if you inadvertently trigger a TPO. (though that's probably not likely if the tree is visible only to you and your neighbours)