DDR

OK, so that didn't go as smoothly as I'd hoped. Unfortunately, we hit an error while upgrading the forum, and the upgrade had to be rolled back. The error probably had something to do with a disk drive MLPF runs on freezing. Still. Third time's the charm, right?

As always, updates can be found at https://twitter.com/Poniverse.

Hello, everyone!

Earlier today, a forum upgrade was started and aborted until we could gather more information on it. (Two files we'd customised are going to be overwritten.) I want to at least have a copy of what they were, and the ability to put them back, and ideally a diff so I could see what we'd changed. I hope to have that done by tomorrow, at which point I will upgrade the forums. It should be fairly minor this time, no breaking changes.

If anyone has any questions, as always, post them here or ping me on Discord on the Poniverse server. 🙂

 Cheers,
–DDR

p.s.: I'll also be at EFNW in a few weeks if anyone would like to chat about the forums then.

I don't see anything suspicious on mine. But I think I may also be antisocial…

Would you be able to post a screenshot, @Deae Rising Shine~? (Or PM me one if you don't want to share publicly.)

I think emoji preview - ie, typing

:appleh

is broken. It doesn't show a picture in the preview.

12 minutes ago, tinker said:

Status updates are working again. Activity logs should rebuild over time as the backend reindexes all the content (and there's a lot); this may take a day or two.

@CHA0SXIII Should be covered by this. 👍

Oh, I thought it was the desktop popup. I think I don't get to see the in-page ones since I enabled that? Thank you for the report.

12 minutes ago, Mirage said:

I'm sure y'all are fine but if you ever need a hand, I own two IPS sites myself.

We could use a hand, to be quite honest. I'm just asking around how we let you do that now. O_o

9 minutes ago, TheTaZe said:

One more thing I noticed is that the notification popup for the forums isn't clickable anymore.

wat

Confirmed, Califorum.

TaZe: Profile Arrays are no more. (change: $value -> $value['value'] 🙄)

Thanks for the reports, TaZe.

The prefix one will be a bit hard to find, since we have no idea where the error actually is.

Whoops, forgot to hit "save" when I removed the upgrade warning on the main page. It's gone now.

Hello everyone! The latest round of updates is complete - it seems pretty minimal, and everything still seems to work about as well as it did before. However, it did mention something about the login page code getting reset -- I've tested it, and it works for me for now… 😒

There are still some known issues with the previous upgrade. 😬 We hope to get those fixed.

Bleah. Does anyone have a screenshot of how it should be? I don't remember how everything used to line up.

18 hours ago, PinkiePie97 said:

Not sure if this is related to the upgrade as I only just noticed it, but is the timestamp supposed to be displaying on the right side of the post now? If it is, that's fine with me, but I thought I should point it out in case it isn't.

Yeahhh, I didn't really know where to put it. It was overlapping some text on mobile view, and there wasn't quite room to squeeze things around there. I could probably stand have another whack at it.

Drat. I'll have to fix that tomorrow. Gotta sleep, gotta work soon.

Also, a bug I just discovered: Typing ":[" and trying to move the cursor left with the arrow keys doesn't work until you type another 4 or 5 characters.

I've fixed the login issue in just the sketchiest manner possible. Let me know if there's any trouble logging in with poniverse, part of the authentication process is commented out now.

ok, first issue, you can't log in now

Hello everyone! We're now running on the latest and greatest forum software, which probably means some things have stopped working. We'll try to fix anything obvious over the next few days, but please – let us know if you've noticed anything strange. Thanks!

1 hour ago, Jon the VGNerd said:

I noticed today that the usual background of Ponyville as been replaced with a blank and flat background. Any idea what happened to it? 

… No.

Flip. I'll have a look at it asap, after work.

Upgrades are always announced at https://twitter.com/Poniverse , if you're ever curious. I wonder if I can make that Twitter feed show up on the actual error page, so it's more useful than just 'forums gone'.

I actually should have posted outage updates there too, but I forgot. I've added that step my incident response checklist for next time.

Banned for circular motif in avatar.

On 31/03/2018 at 9:14 PM, Scootaloved said:

What will this exactly entail? Will it block people who are loading lots of pages? I tend to reload pages a lot while I'm waiting for replies in threads, and of course i check notifications the instant I notice them. I'm assuming it won't block what could be considered normal user activity?

You are correct; it will probably only block you if you try to load load several hundred pages in a few seconds. It was looking like we were getting a few dozen a second from the spammer, which is fairly easy to filter out.

8 hours ago, Moondancer is best said:

Who is the spammer ?? 

We'll never know. Probably some automated bot that was set upon us and hundreds of other forums. Such is the nature of the internet; every networked computer is effectively constantly under attack. My own home machine sees hundreds of user/pass login attempts every day, for example, despite the fact you can't log in to it with a password.

On 31/03/2018 at 4:41 PM, Recherche said:

Though, of course, that could also mean that they will try again under a separate IP address... 

Yep. Happened earlier today while I was out. We should be more resilient to these sorts of attacks anyway, it's kind of shameful that we aren't.

Hello everyone,

We apologise for the instability and downtime of the forums that occurred today. A spammer was, probably inadvertently, performing a denial of service attack on us – they were requesting so many forum pages and searches that no legitimate traffic could get through. This also maxed out our database, which led to instability and downtime in some other Poniverse services which shared the DB instance.

We've blocked the spammer's IP for now, and added basic rate limiting. Let us know if you encounter any 403 "service temporarily unavailable" errors, please.

A long-term solution would be to introduce better rate-limiting (so offenders can't take us down as easily) and fail2ban (automatically block offenders from accessing MLPF at all). We will be investigating this over the coming weeks.

Yeah; they're pretty hidden in there aren't they. I'd really like to get them out into the first view, it's stupid to have to click in to find them.

On 21/03/2018 at 5:52 PM, Ken said:

Why don't you switch over to another forum software already?

From my end, it would be a heck of a lot of work. There's a massive amount of meta-stuff we've got working right now that just wouldn't be able to be exported to a different piece of software - things like reactions, bans, e-commerce, email template customization, and so on.

I think there's been a few whacks at making something in-horse over the years, but nothing that's got anywhere. It's a surprisingly huge software engineering project.

5 hours ago, ChB said:

I think it would be nice to have "Smiley," "Yay," and "Thanks" reactions count towards your reputation count, since they show just as big (if not bigger) signs of approval as a Brohoof.

I'd be down with that. (I'm not really involved in that decision, though - that's more of a front-end thing. I just try to keep the software up to date and bug free.)

5 hours ago, Cwanky said:

I understand that many of the older emojis were removed and phased out.

Which older emojis? All the ones you posted still seem to be in the drop-down, just under the Emoticons category?

It could be that we just broke a permissions thing somewhere along the line, too. 🤔

I'll have a look after I fix changing/setting signatures being broken. I'm not sure how much we can do about it, though. :/