A good point which only a few people have mentioned is that many people only have a few experiences with bronies, and they're almost never good.
See, some types of MLP fans are nice, quiet people who keep to themselves. Other types are the loud, obnoxious idiots who spam every site they find with ponies and nothing else. Because the good bronies are the quiet ones, and the bad the loud ones, most non-bronies only come into contact with the worst kind of bronies.
However, (a good point that Geek0zoid made) brony hate is diminishing. This is mostly due to the popularity of MLP in recent years, and the fact that the fandom is spreading to many other demographics. Tons of my friends watch it, my mom loves it, and I myself am a young pegasister. The few times I've gotten into an internet argument with an anti-brony, a bunch of people backed me up, saying that MLP is an awesome show and no one should be ashamed for watching it.
Go Bronies!