Well...hmm. I guess it's all subjective. I don't really think about the show in those terms--"going downhill." But I know it is changing. It's gained scope and maturity, but it's lost some of its consistency and charm. In the early seasons, we had character-driven plots and the letter-writing theme tying everything together. That's gone now, so most episodes are just a bunch of stuff happening. And Seasons 1 and 2 were written primarily with little girls in mind. Now they're trying to please little girls and us, which isn't necessarily a bad thing but it does change the tone somewhat. We started to see that in Season 3 with episodes like Magic Duel and it really threw me off.
I do know that since then, and especially since Twilicorn, I have been feeling increasingly disconnected. I do still enjoy the show and find plenty of positives, but the innocence and simplicity of episodes like Griffon the Brush-Off and The Best Night Ever--that's what really drew me to MLP, and I didn't feel much of that from Seasons 3 and 4. Everything is slowly losing focus, dissipating. EQG, the lost balance within the Mane 6 that the show doesn't want to acknowledge. I'm not against change in the show, but none of those are the changes I was looking forward to. I can't objectively say whether it's gone downhill or not.