I've honestly asked myself the same thing. I have other activities and interests I pursue, but Internet-wise it's all pony. It's a tough call, but I say that if someone asks you, "Hey, what do you do in your spare time," you better have more to say that "hit F5 on Equestria Daily."
Really, that's you improving your vocabulary and expanding your range of thinking. Most non-bronies I know can barely hold their attention for a 2,000 word count story. Personally, having read long works like Fallout: Equestria and The Immortal Game, I've learned so many new words and ways to construct sentences. So, nah, I say you're not doing yourself much harm.