An (incomplete) post-mortem on recent federation problems there (1/?)
This instance has faced federation problems for the past months (almost a year). Some other instances unfederated from it. Here is a brief outline of what happened:
* Originally (since its establishment in August 2018), this instance had open registrations. Anybody could register.
* On March 13, 2019, a new user @firstname.lastname@example.org registered there. It posted three toots during its life (I cannot recover the exact dates now)...
An (incomplete) post-mortem on recent federation problems there (2/?)
* It is possible that it was an actual person at that time.
* However, somewhere between March and October, someone gained a control on this user (or maybe it was an evildoer from the beginning), and started to follow users from other instances en masse. Including from nazi and "free speech" instances - one we did not yet know about, and so did not block. It followed about 250 users in total, mostly ones who post a lot.
An (incomplete) post-mortem on recent federation problems there (3/?)
* "Most recent activity" of this user was on October 12, 2019. It is possible that all of these fake follows were made during that day.
* On October 25, 2019, beach.city suspended this user on their side, with a public announcement: https://beach.city/@moderation_alerts/103024097829866681
It seems that at no point beach.city admins attempted to contact this instance, or to issue any moderation report. We had no way of knowing about that announcement.
An (incomplete) post-mortem on recent federation problems there (4/?)
* At this point, we did not know anything is wrong. Federated timeline was flooded with a lot of bullshit (like several toots/second), so everybody thought it's just other users having large follow lists; as a result, nobody used it and nobody noticed there were any nazi instances in it.
* On approx. December 3rd, 2019, we had a system-wide crash caused by our 100GB media storage running out of space: https://embracing.space/@fil/103255642491111681
An (incomplete) post-mortem on recent federation problems there (5/?)
* It took us two days to notice and somehow diagnose the problem. We just thought that it's normal for mastodon to store a lot of media, and so purchased 500GB of media storage.
* However, in January we noticed that 500GB storage is almost full, too. We cannot financially sustain Mastodon growing at 300GB/month, so it was time to investigate...
An (incomplete) post-mortem on recent federation problems there (6/?)
* So in the middle of January 2020, we started to look into Mastodon storage usage. Turned out that most of the storage was consumed by various pic bots from other instances - bots that post a new pic every couple of minutes. And these instances apparently only existed to support these bots.
Our first instinct was to "reject media files" from these instances, but thankfully we did not stop after that and started to dug deeper..
An (incomplete) post-mortem on recent federation problems there (7/?)
As we dug deeper, it turned out that it was not our well-behaved users who followed these bots. It was that "forfast" user who followed every single out of them.
It was highly implausible that the real person would follow dozens of these picture bots for a combined picture per second - no real person can catch up with this!
So, in mid-January 2020 we finally noticed forfast, along with its follow list.
An (incomplete) post-mortem on recent federation problems there (8/?)
As we started researching this "forfast" user and its follow list, it turned out that it also followed a lot of nazi accounts from nazi instances.
So, after having spent several days manually silencing picture bots and "rejecting media files" from their instances, we were able to realize "forfast" is a follow bot, to ban it, and to stop a flood of nazi and pic data from overflowing our instance.
An (incomplete) post-mortem on recent federation problems there (9/?)
This was not an end of it, though! We still had to go through every nazi instance this bot managed to follow, and unfederate from it manually. We still had to go through every pic bot this bot managed to follow, and suspend it manually (so as to free some storage space).
We also tried to search for "forfast" on google and found that beach.city public notice from two months ago: https://beach.city/@moderation_alerts/103024097829866681
An (incomplete) post-mortem on recent federation problems there (10/?)
So, by 20th January 2020, after two days of downtime in December, and several days of manual mitigation in both December and January, we finally got rid of the follow bot, somehow cleaned up our federated timeline, and made Mastodon resources consumption somewhat manageable.
Around 90% of all media storage over 17 months of this instance was generated by that single bot over two months of its rampage.
An (incomplete) post-mortem on recent federation problems there (11/?)
However, that was not all of it.
Turned out somewhere between October 2019 and January 2020, lgbt.io took beach.city public notice as a call to action, and unfederated from our instance, breaking several mutual relationships in the process.
lgbt.io has a public block list, meaning that everyone could see our instance on it, alongside gab and the like. And at least some other instances based *their* block lists off lgbt.io.
An (incomplete) post-mortem on recent federation problems there (12/?)
Unfortunately, at no point has anybody contacted the admins of that instance. lgbt.io just silently unfederated, and some other followed the lead.
We realized that by doing a google search for "embracing.space", which is not as easy as it sounds. It was unpleasant, to learn that some large and nice instances publicly unfederated from you without notifying you or stating any reason for unfederation.
An (incomplete) post-mortem on recent federation problems there (13/?)
Thankfully, we managed to negotiate with some of these instances to federate with them again. Most recent addition is lgbt.io, who restored back a federation with us on July 28th, 2020 (and it was during negotiation with them that we learned beach.city public notice was a reason for unfederation).
So we can hope that federation is mostly restored by now. However...
An (incomplete) post-mortem on recent federation problems there (14/14)
...However, we cannot be sure federation is completely back. Google Search for "embracing.space site:*/about/more" still returns two Mastodon instances with our instance on their public block list. And who knows how many non-Mastodon instances, or Mastodon instances with non-public block lists are there who do not federate with us?
It might well be that full links with fediverse will be never restored, not under this name...
Conclusion on our federation problems, lessons for other instances (1/3)
All the previous post-mortem boils down to this: we were unknowingly infected with a follow-bot who caused significant technical and financial problems on our instances. Other instances, while knowing, did not warn us about that, but just silently unfederated from us. And it was even more effort getting federation back than it was to fix the follow-bot problem.
This could happen to everyone! Any instance can be a victim!
Conclusion on our federation problems, lessons for other instances (2/3)
Note how, had beach.city not only published their notice, but also warned us (or even simply mentioned us in that notice), it could all be fixed in the beginning, without any trouble, without any downtime, without any additional hosting expenses for this instance.
Note how, had lgbt.io told us they were going to unfederate for that reason, the problem could be solved so much earlier and without other unfederations?
Conclusion on our federation problems, lessons for other instances (3/3)
Now think about how 4channers or other bad actors can create a similar bot on any instance with open registrations (or just having an user with a weak password), and in that way cause such an instance to be cut off from a significant chunk of fediverse, without its admin noticing? Do we want to help them dismantle the fediverse?
Warn other instances' admins about bad things going on, not just silently unfederate from them!
The social network of the future: No ads, no corporate surveillance, ethical design, and decentralization! Own your data with Mastodon!