That's actually a really good point.
In a perfect world, we'd all be able to watch whatever the f*ck we want to and nobody would criticise us for it. But this isn't a perfect world, so bronies are always going to be criticised, to some extent, for watching a little girls' show.
So really, we have a few options. Pointing out all the reasons why it's 'a show for men' is one, but it's really not helping anything. As you said, it's contributing to the problem.
Not giving two flying f*cks is really the best option. You don't have to explain yourself to anybody just because you enjoy a show. So what of it's for young girls? It's 2014, stranger things have happened.
People will become more accepting over time, but in the meantime, why bother trying to make out the show to be something it's not? OP is right, it's contradicting the cool part of being a brony, giving the middle finger to society's expectations. I mean, I'm a 14 year old girl, so I suppose this doesn't apply to me as much as it does everyone else on this site. Oh well.