Jump to content
Banner by ~ SFyr

About the recent outage and optimization issues


Recommended Posts

First off, I'd like to apologize for having the site unavailable for so long. We encountered an issue with the database locking up last night during one of our nightly backups and were forced to restart one of our host nodes as the process had gone into a non-interruptible state. While I did make an attempt to do a graceful shutdown of all of our services before this (Aside from the process that couldn't be exited) unfortunately I forgot to sync the disks when I did so so we lost our cache configuration for about 10 hours.

In the process, the mail server also failed to start back up properly so notifications and other email services from Poniverse were also down for that period of time, and it went unnoticed as our monitoring software saw that the SMTP port was listening properly. And MLPF itself was up and running from a glance, so we left it like that while Tinker went to work and I slept.

Now that everything has been fixed, with the email issue not being discovered until about 14 hours after the outage occurred I figured I'd address this and let people know what happened and that we'll do our best to check everything more thoroughly in the future now that we know this is a potential problem that crops up.

@tinker @Twilight Sparkle (Wow Feld pls) and I ended up spending a couple of hours troubleshooting these issues and Feld came up with a few ways to re optimize the forums for our new setup since we hadn't really done that since we moved servers a few years back. Hopefully you will all notice performance improvements as we've moved some of the critical cache operations off of our network attached HDDs and into RAM as well as taken some of the load from our MySQL database and put that to redis and PHP's opcache. From my own testing it's gotten load times down from about 8-10s per page down to 1-2s which should be a lot more manageable. As usual though, I hope everyone keeps an eye out for issues and brings them to our attention!

  • Brohoof 10
  • Thanks 2
  • Smile 2
Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Unfortunately, your content contains terms that we do not allow. Please edit your content to remove the highlighted words below.
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


  • Recently Browsing   0 members

    • No registered users viewing this page.
  • Create New...