Having experienced East-Dulwich and Gipsy Hill, I feel equally safe in both places, I like them both for different reasons and really feel you are talking about another place, you must have bumped into the wrong person and that could happen anywhere, even in E-D. E-D has a more exciting nightlife, but that can also keep you from sleeping at night, whereas in GH you are more likely to be woken up by a melodic birdsong. These two places are my london favourites.