LadyMercury

So about that one time we lost both of our hard drives

Recommended Posts

So I wanna preface this with this: I'm thankful that everyone was caring and offered lots of help and showed a lot of concern for the site. 

We weren't hacked, DDoSed or anything like that really. We just experienced a symptom of old age it turns out! To get into a bit of technical stuff, we actually lost both of the main hard drives (SSDs) in the main web hosting server on the same day. To put this succinctly: This kills the server. After spending a day diagnosing the issue, the ops team and I were ready to submit the case to our hosting provider OVH and let them start replacing the parts. 

This is a great time for us to revisit our policies so that we're very sure that something like this wouldn't be a huge loss in the future. We already take daily backups of all of our services and put them offsite as well so that losing a host server like this isn't the end of the world. However due to this extended outage we're going to take our time and verify a few things to make sure that all of the processes we have in place are working as intended just in case something like this happens again in the near future (We just got new hardware, so fingers crossed it doesn't!). This has also given us a lot of insight in the lifetime of our servers and when we actually need to schedule this kind of maintenance with our hosting provider so that in the future we can plan for this and only end up with a few hours of downtime rather than two whole days.

Really though. I want to thank everyone here, new and old for being here and giving the site life. This was a difficult time but a sign that we've been going long enough to wear out the hardware the site was running on and that to me is something to celebrate! Thanks again everyone and if you have any concerns or questions feel free to inbox me or post them here!

  • Brohoof 12
  • Yay 3
  • Hug 2

Share this post


Link to post
Share on other sites

Both devices failing at the same time is always a possibility (which is why it is recommended not to use drives from the same batch in an array since they are more likely to fail together). With SSDs it's even more pronounced as both drives wear out at the same time. So, either the drives used have to be of different ages (so that when the older one goes, the newer one is still OK) or the wear has to be monitored and at least one drive replaced before it wears out.

I had both HDDs in an array develop lots of bad sectors (thankfully, they were in different places), so, yea, not fun.

I also know that hosting providers do not like to admit that the hardware is bad.

Thank you for keeping the site up without the loss of data, you did a great job. Unlike a lot of users, I know what it's like to be in a similar situation. Let's hope the servers work great.

Share this post


Link to post
Share on other sites
Badges

It was a long few days. Make sure Gale and everyone else gets some sleep and coffee.

Also, SSD's aren't technically hard drives :derp: They're just drives.

Share this post


Link to post
Share on other sites

Good job on the staff!

Must surely have been a stressy time and sacrifices have been made. MLPF's queen even had to say farewell to her beauty sleep :rarity:

Thank you guys for doing all of this! ❤️

I wonder though, in which way anypony of the community was able to help? :ooh:

 

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


  • Recently Browsing   0 members

    No registered users viewing this page.