Having enemies is a waste of time. Just go on with your life, really. It is, after all, just a show, and what other people's opinions are on it don't matter.
Hell, other people's opinions on anything don't really matter at all in the end, so just carry on and do what you enjoy and don't worry about what other people think.
I don't think there's really any "threat" to the fandom, it seems strong enough to me that
1) haters can't do anything, which covers a lot of specific points, including the rage against cloppers (it's going to keep happening, avoid it if you don't like it and ignore it if you stumble across it and you'll be fine), annoyance at "love and tolerate EVERYTHING" bronies (who simply need to learn that negative emotions are okay to have, and that the best thing to do is simply not be bothered by people vocalizing their disproval), and other such grievances.
2) if the show were to end, the fandom would continue for a long time, I hear Firefly has a small but very dedicated fandom, bronies are a very large and dedicated fandom, I see little to sway a loss of interest for everyone all of a sudden.
I suppose the biggest threat would be some of the popular names/voices in the brony community suddenly losing interest, seeing as a large amount of people seem to care about what their favorite brony artist thinks. However, I am certain that if this were to happen, you'd still see plenty of strong, active brony communities.