@@onlyone, Listen sugarcube, as someone who used to have a lot of anxiety surrounding liking and watching this show, let me give you a few bits of advice on how to cope with your anxiety. First off, it's easy to say "just don't listen to the haters", so I'm going to change that tune, and recommend that you turn off their voices. It's okay to take a step back from the fandom if you need to, it's okay to not engage people who are being negative and harsh, it's even okay to block or unfollow people. While listening to the opinions of others is important, your mental health is ultimately more important.
Second, you must come to understand that nothing is perfect, that every piece of media that has ever been or will ever be created has flaws. Strange as it may seem, I found it helpful to deal with this by becoming more of an analyst. By seeing the good and the bad in everything and understanding not just that I enjoy imperfect things but why I enjoy them, I found a lot of peace with myself and others.
Third, know that it is okay to be passionate about something "silly" like a TV series. My Little Pony was never just a show for me, it was a coping mechanism for my Anxiety disorder that taught me how to love myself while I was still surrounded by people who told me to hate myself. There is nothing wrong with coping with your struggles by seeking comfort in media, no matter how "silly" that media may seem, as long as you use it to move forward and better your life.
And finally, if you can work on all of these other things, and make peace with going against the views of others and enjoying what you enjoy without needing validation, you may also start to work on not harboring animosity towards "the haters", so to speak. There are people in this fandom who aggressively hate the later seasons, and use that hatred to try to build themselves and their opinions up by shoving down those who disagree with them, calling them "naive" or "weak" for not accepting their One True Opinion that the show is bad. Please understand, deep down these people are no different from you. They too seek validation for their opinions because they struggle to hold them on their own, they too feel alone no matter how many people agree with them. There is no side of the argument that is inherently more at peace, peace is only found when you can feel the way you feel without needing the majority of people to feel the same.