I live directly across the road from the Gala site just up from Harris Boys, can see it from my front window, and the festival stopped before 11 pm and everything was quiet.
I haven't enjoyed the last few days, it's been unbearable in my flat and I went over the hill to Honor Oak last night and could still hear it there, but it certainly didn't go on until 3.
It's actually not as bad today because it sounds like they only have one stage and I've just got one reggae song playing in the distance and it's quite nice. The thing that has made it most like torture is the cacophony of all the stages mixed together for three days. If it was just one stage at the sound level it is right now I'd probably be ok with it.