Mate. The definition of the term "Brony" is quite clear. It is used to describe a fan who is older than the target demographic for the show, and generally used for males.
So if you call yourself a fan, fan = brony, so you can't escape the inevitability of being reffered to as a brony.
It's like, someone who is black will probably be referred to as "Black". Nothing mean or racist at all, we all do it. The criteria for that is having dark skin. So if you have dark skin, you are black. This term isn't mean or anything, it's just easier to say than "African-American", or whatever race that person is from.
Someone who is an older fan of the show fits into the criteria of the term Brony, therefore, whether you like it or not, you're a brony. Again, there is nothing mean intended from this term, if you look at the criteria.
I have nothing against disliking being called a brony and saying not to refer to you as such...But really, you're all bronies :U