An (incomplete) post-mortem on recent federation problems there (7/?)
As we dug deeper, it turned out that it was not our well-behaved users who followed these bots. It was that "forfast" user who followed every single out of them.
It was highly implausible that the real person would follow dozens of these picture bots for a combined picture per second - no real person can catch up with this!
So, in mid-January 2020 we finally noticed forfast, along with its follow list.
An (incomplete) post-mortem on recent federation problems there (6/?)
* So in the middle of January 2020, we started to look into Mastodon storage usage. Turned out that most of the storage was consumed by various pic bots from other instances - bots that post a new pic every couple of minutes. And these instances apparently only existed to support these bots.
Our first instinct was to "reject media files" from these instances, but thankfully we did not stop after that and started to dug deeper..
An (incomplete) post-mortem on recent federation problems there (5/?)
* It took us two days to notice and somehow diagnose the problem. We just thought that it's normal for mastodon to store a lot of media, and so purchased 500GB of media storage.
* However, in January we noticed that 500GB storage is almost full, too. We cannot financially sustain Mastodon growing at 300GB/month, so it was time to investigate...
An (incomplete) post-mortem on recent federation problems there (4/?)
* At this point, we did not know anything is wrong. Federated timeline was flooded with a lot of bullshit (like several toots/second), so everybody thought it's just other users having large follow lists; as a result, nobody used it and nobody noticed there were any nazi instances in it.
* On approx. December 3rd, 2019, we had a system-wide crash caused by our 100GB media storage running out of space: https://embracing.space/@fil/103255642491111681
An (incomplete) post-mortem on recent federation problems there (3/?)
* "Most recent activity" of this user was on October 12, 2019. It is possible that all of these fake follows were made during that day.
* On October 25, 2019, beach.city suspended this user on their side, with a public announcement: https://beach.city/@moderation_alerts/103024097829866681
It seems that at no point beach.city admins attempted to contact this instance, or to issue any moderation report. We had no way of knowing about that announcement.
An (incomplete) post-mortem on recent federation problems there (2/?)
* It is possible that it was an actual person at that time.
* However, somewhere between March and October, someone gained a control on this user (or maybe it was an evildoer from the beginning), and started to follow users from other instances en masse. Including from nazi and "free speech" instances - one we did not yet know about, and so did not block. It followed about 250 users in total, mostly ones who post a lot.
An (incomplete) post-mortem on recent federation problems there (1/?)
This instance has faced federation problems for the past months (almost a year). Some other instances unfederated from it. Here is a brief outline of what happened:
* Originally (since its establishment in August 2018), this instance had open registrations. Anybody could register.
* On March 13, 2019, a new user @email@example.com registered there. It posted three toots during its life (I cannot recover the exact dates now)...
Sorry, had a small downtime because of https://github.com/tootsuite/mastodon/issues/14131
Everything should work now.
Hi! I've noticed on your "about" page (https://lgbt.io/about/more) that you have limited federation with this instance (embracing.space). I have no idea why, I've never received any moderation report from you, and I'd hate to be in the same list as gab. Could you please explain why did that happen?
Hi! While I've been cleaning my instance, I came upon your moderation alert: https://beach.city/@moderation_alerts/103024097829866681
(As a side note, it does indeed seem to be a follow bot, and with that follow list, this account alone consumed ~98% of my instance resources and turned our federated timeline into garbage)
But sadly I've never received any moderation report from you. Not sure if that's how it's supposed to be, or some problem with federation?
The social network of the future: No ads, no corporate surveillance, ethical design, and decentralization! Own your data with Mastodon!