I've already left the fandom once, temporarily at least. I never began thinking that the show was not good, although after being a brony for a few months (last March-July) and seeing every episode that had been aired so far, combined with it becoming increasingly harder to hide, (I'm not open about my love for the show), I stopped being involved.
I watched some S3 episodes around December-January and was back in by February, and luckily this site and a few others have allowed me to maintain my interest a little easier.
I don't think I could ever leave permanently or stop supporting the show and its fans.