Forum Instability (2018-03-31)

[DDR 215]

Hello everyone,

We apologise for the instability and downtime of the forums that occurred today. A spammer was, probably inadvertently, performing a denial of service attack on us – they were requesting so many forum pages and searches that no legitimate traffic could get through. This also maxed out our database, which led to instability and downtime in some other Poniverse services which shared the DB instance.

We've blocked the spammer's IP for now, and added basic rate limiting. Let us know if you encounter any 403 "service temporarily unavailable" errors, please.

A long-term solution would be to introduce better rate-limiting (so offenders can't take us down as easily) and fail2ban (automatically block offenders from accessing MLPF at all). We will be investigating this over the coming weeks.

[The Recherche 29,965]

Good to hear!

Though, of course, that could also mean that they will try again under a separate IP address... 

Regardless, it's good to hear that the problem has been resolved for the time being! 

[Nightfall Gloam 18,231]

That's good!

The downtime was really annoying for me, especially with the notifications never properly loading.

[King of Canterlot 9,588]

Damn spammers and hackers, they only like to be an annoyance to everyone else ....................

[SparklingSwirls 21,261]

@DDR I appreciate all your work on the forums! Thanks to you and the other admins for working to get MLPF fixed as soon as possible. 

[The Kaeya Simp 13,953]

Well, it could've been worse!

 

@DDR you and all of the other admins are the best, there is so much shit you need to deal with!

[shadowwarp940 1,302]

Ugh, spammers! Aren't useful for nothing, I'm telling you, But that aside, I'm glad that the issue is over now

[Will Guide 21,325]

So THAT was what was going on! So it wasn't MY connection particularly. 

[Nyactis Mewcis Catlum 670]

4 hours ago, DDR said:

fail2ban (automatically block offenders from accessing MLPF at all)

What will this exactly entail? Will it block people who are loading lots of pages? I tend to reload pages a lot while I'm waiting for replies in threads, and of course i check notifications the instant I notice them. I'm assuming it won't block what could be considered normal user activity?

[Prospekt 10,984]

Glad things are up and running again and that the issue has been rectified for now. It was indeed frustrating having pages quit loading on the same day as a new episode of the show.

@DDR Thank you for all you've done and all you continue to do. It's very much appreciated.

[Esom 56]

Well glad to hear things were taken care of in time.

[Ganondorf8 11,118]

At least you were on top of things before they could get out of hand.

[Sketchy Tune 964]

Thank goodness. At least it's being taken care of ^^

[SDP40F amtrak 2,441]

Who is the spammer ?? 

[DDR 215]

On 31/03/2018 at 9:14 PM, Scootaloved said:

What will this exactly entail? Will it block people who are loading lots of pages? I tend to reload pages a lot while I'm waiting for replies in threads, and of course i check notifications the instant I notice them. I'm assuming it won't block what could be considered normal user activity?

You are correct; it will probably only block you if you try to load load several hundred pages in a few seconds. It was looking like we were getting a few dozen a second from the spammer, which is fairly easy to filter out.

 

8 hours ago, Moondancer is best said:

Who is the spammer ?? 

We'll never know. Probably some automated bot that was set upon us and hundreds of other forums. Such is the nature of the internet; every networked computer is effectively constantly under attack. My own home machine sees hundreds of user/pass login attempts every day, for example, despite the fact you can't log in to it with a password.

 

On 31/03/2018 at 4:41 PM, Recherche said:

Though, of course, that could also mean that they will try again under a separate IP address... 

Yep. Happened earlier today while I was out. We should be more resilient to these sorts of attacks anyway, it's kind of shameful that we aren't.

[Hierok 11,829]

So that's the reason mlpforums would not load. I thought it was because mlpforums had updates\upgrades.

[DDR 215]

Upgrades are always announced at https://twitter.com/Poniverse , if you're ever curious. I wonder if I can make that Twitter feed show up on the actual error page, so it's more useful than just 'forums gone'.

I actually should have posted outage updates there too, but I forgot. I've added that step my incident response checklist for next time.

[BloodDrops 762]

On 01/04/2018 at 12:32 AM, DDR said:

Hello everyone,

We apologise for the instability and downtime of the forums that occurred today. A spammer was, probably inadvertently, performing a denial of service attack on us – they were requesting so many forum pages and searches that no legitimate traffic could get through. This also maxed out our database, which led to instability and downtime in some other Poniverse services which shared the DB instance.

We've blocked the spammer's IP for now, and added basic rate limiting. Let us know if you encounter any 403 "service temporarily unavailable" errors, please.

A long-term solution would be to introduce better rate-limiting (so offenders can't take us down as easily) and fail2ban (automatically block offenders from accessing MLPF at all). We will be investigating this over the coming weeks.

Whats there Ip

I shall throttle their port