About the recent outage and optimization issues

[Mercury 351]

First off, I'd like to apologize for having the site unavailable for so long. We encountered an issue with the database locking up last night during one of our nightly backups and were forced to restart one of our host nodes as the process had gone into a non-interruptible state. While I did make an attempt to do a graceful shutdown of all of our services before this (Aside from the process that couldn't be exited) unfortunately I forgot to sync the disks when I did so so we lost our cache configuration for about 10 hours.

In the process, the mail server also failed to start back up properly so notifications and other email services from Poniverse were also down for that period of time, and it went unnoticed as our monitoring software saw that the SMTP port was listening properly. And MLPF itself was up and running from a glance, so we left it like that while Tinker went to work and I slept.

Now that everything has been fixed, with the email issue not being discovered until about 14 hours after the outage occurred I figured I'd address this and let people know what happened and that we'll do our best to check everything more thoroughly in the future now that we know this is a potential problem that crops up.

@tinker @Twilight Sparkle ✨ (Wow Feld pls) and I ended up spending a couple of hours troubleshooting these issues and Feld came up with a few ways to re optimize the forums for our new setup since we hadn't really done that since we moved servers a few years back. Hopefully you will all notice performance improvements as we've moved some of the critical cache operations off of our network attached HDDs and into RAM as well as taken some of the load from our MySQL database and put that to redis and PHP's opcache. From my own testing it's gotten load times down from about 8-10s per page down to 1-2s which should be a lot more manageable. As usual though, I hope everyone keeps an eye out for issues and brings them to our attention!

[Treeglow Flicker 13,014]

Thanks for all the hard work that goes on in the background!