Edit: Before you read the rest of this, I feel this quote from earlier describes a lot of what I didn't already:
Ok I'll try and explain this, bare with me.
We are all fighting over what it means to be a "Brony", since the definition is so obscure that my cat could be a "Brony." The problem being some people don't want to be called a "Brony" for one reason or another. Why is this a problem, you might ask? Because it indirectly reinforces multiple negative stereotypes AND it divides the community to some degree.
The only reason this is an argument is because some people put the community, and everyone in it, ahead of themselves, so they adopt the name to be a positive influence. People really care about this community, because it's fun, generally nice, and networking/making friends is easy. Some would go to ends of the universe to keep the community going, and it's these people who willingly use the name "Brony."
With that out of the way...
Before I was in this fandom, I was in another fandom. Furry, to be exact.
I did not deserve then to feel like hell for being a furry. There was nothing wrong with me being one. But at the same time, the image that that title brings to mind brought me so much internalized self hatred, and so much guilt, that I predominantly identified as and tried to push the title of Anthro. It didn't change the fact that I went to the same sites that furries did, it didn't change that I was into the same things. But it was an attempt to disassociate myself from all the negative connotations that being a furry brings with it.
Did it change anything?
No, because to the outside world, I was still a Furry, which is the catch all term for anyone into anything involving talking cartoon animal people. That may not be fair, but much of the world will not care to differentiate one tribe of a particular interest or community from another tribe. If they already don't like X, and you identify as something like X, they will simply still go after you for being X.
This fandom faces the same kind of division and self loathing that furries do, and weaboos / anime fans to a lesser extent. We try to distinguish ourselves from each other. Those from 4chan's /mlp/ and other chanboards call themselves ponyf*ckers / horsef*ckers like it's a badge of honor, but to the rest of the world? They're still bronies. Bronies, being the catch-all for an adult fan of my little pony. Some may allow you to differentiate yourself from the negative image of being an adult mlp fan, but the majority simply won't just care, and will call you what they desire, and will do what they please with you in their treatment.
So, ultimately, I would say the answer is try to come to terms with this the best that you possibly can. If you can adopt a "f*ck all of you, I'm me and I at least deserve my own respect", that's great, but I know not everyone can. I've tried. I tried with Furry, and I tried with Brony. And really, I am still a Furry, as much I don't want to admit that. But I shouldn't be. I know I shouldn't be. But like with so many other things, the world will still try it's damnedest to hurt you for anything. Even being a harmless fan of a little girl's cartoon.
The best thing I can possibly say here, is to be comfortable with yourself, however you can.
Beyond that? The best way I couldn't put it myself just yet, has been put here:
http://juniorbizarre.tumblr.com/post/45695139663
http://juniorbizarre.tumblr.com/post/124518900194
It's your skin. You should be comfortable with it.