So about that one time we lost both of our hard drives

[Mercury 351]

So I wanna preface this with this: I'm thankful that everyone was caring and offered lots of help and showed a lot of concern for the site. 

We weren't hacked, DDoSed or anything like that really. We just experienced a symptom of old age it turns out! To get into a bit of technical stuff, we actually lost both of the main hard drives (SSDs) in the main web hosting server on the same day. To put this succinctly: This kills the server. After spending a day diagnosing the issue, the ops team and I were ready to submit the case to our hosting provider OVH and let them start replacing the parts. 

This is a great time for us to revisit our policies so that we're very sure that something like this wouldn't be a huge loss in the future. We already take daily backups of all of our services and put them offsite as well so that losing a host server like this isn't the end of the world. However due to this extended outage we're going to take our time and verify a few things to make sure that all of the processes we have in place are working as intended just in case something like this happens again in the near future (We just got new hardware, so fingers crossed it doesn't!). This has also given us a lot of insight in the lifetime of our servers and when we actually need to schedule this kind of maintenance with our hosting provider so that in the future we can plan for this and only end up with a few hours of downtime rather than two whole days.

Really though. I want to thank everyone here, new and old for being here and giving the site life. This was a difficult time but a sign that we've been going long enough to wear out the hardware the site was running on and that to me is something to celebrate! Thanks again everyone and if you have any concerns or questions feel free to inbox me or post them here!

[ShadOBabe 18,984]

Good job admins! You did great! 

[cuteycindyhoney 13,283]

Good job. I didn't even notice a hiccup!

[Pentium100 2,056]

Both devices failing at the same time is always a possibility (which is why it is recommended not to use drives from the same batch in an array since they are more likely to fail together). With SSDs it's even more pronounced as both drives wear out at the same time. So, either the drives used have to be of different ages (so that when the older one goes, the newer one is still OK) or the wear has to be monitored and at least one drive replaced before it wears out.

I had both HDDs in an array develop lots of bad sectors (thankfully, they were in different places), so, yea, not fun.

I also know that hosting providers do not like to admit that the hardware is bad.

Thank you for keeping the site up without the loss of data, you did a great job. Unlike a lot of users, I know what it's like to be in a similar situation. Let's hope the servers work great.

[Comp 84]

Props to you, @tinker, Solar Bliss, and @Twilight Sparkle ✨ for working day and night to resolve the issue! We site devs are forever indebted to you.

[Early Sunsets 2,106]

It was a long few days. Make sure Gale and everyone else gets some sleep and coffee.

Also, SSD's aren't technically hard drives  They're just drives.

[Venomous 18,813]

Closing now that this is now in the past ~