No one should come to your front door and be swearing at you, especially over a fence! I really hope you sort the situation out. To put it in to some context, our neighbours just replaced our adjoining fence. They didn't talk to us about it beforehand. The first we knew about it was noisey workmen in our garden (without permission) for two days erecting it. It's a totally different fence with metal posts and horizontal wooden slates with big gaps between. What did we do, nothing, we just let them get on with it. Ultimately it's just a fence and better than the rotting one already there. Sounds like there may be more issues here than the fence and others recommendation of a mediator sounds like a good one.