Exactly! I like the Freemason analogy--just because they are a group of people organized as a 'secret' society, the general public singles them out and concocts ludicrous theories about them. In a similar vein, 'bronies' are wrongly generalized by a handful of ignorants as pedophiles, rapists, et cetera, simply for not fully understanding a shared cultural interest.
There will always be backlash towards certain groups of people within defined societies, though. Humans will always be different, no matter if we speak the same language even. It is just human nature to formulate culture, and thereby opinions will tend to differ as long as we live in modernized societies. And I KNOW that we always will. (I am looking at you, John Zerzan!)
Anyway, all I say is, 'Buck it!' Who cares what people say? Do your own thing, and just be proud of whom you are. You are unique, and as long as people are willing to listen to your opinion, listen to theirs, too.
P.S. Should we create a group of ponies à la 'Freemasons'?
P.P.S. The 'brony' episode sounds awesome! I cannot wait!