Show newer

An (incomplete) post-mortem on recent federation problems there (7/?) 

As we dug deeper, it turned out that it was not our well-behaved users who followed these bots. It was that "forfast" user who followed every single out of them.
It was highly implausible that the real person would follow dozens of these picture bots for a combined picture per second - no real person can catch up with this!

So, in mid-January 2020 we finally noticed forfast, along with its follow list.

Show thread

An (incomplete) post-mortem on recent federation problems there (6/?) 

* So in the middle of January 2020, we started to look into Mastodon storage usage. Turned out that most of the storage was consumed by various pic bots from other instances - bots that post a new pic every couple of minutes. And these instances apparently only existed to support these bots.
Our first instinct was to "reject media files" from these instances, but thankfully we did not stop after that and started to dug deeper..

Show thread

An (incomplete) post-mortem on recent federation problems there (5/?) 

* It took us two days to notice and somehow diagnose the problem. We just thought that it's normal for mastodon to store a lot of media, and so purchased 500GB of media storage.

* However, in January we noticed that 500GB storage is almost full, too. We cannot financially sustain Mastodon growing at 300GB/month, so it was time to investigate...

Show thread

An (incomplete) post-mortem on recent federation problems there (4/?) 

* At this point, we did not know anything is wrong. Federated timeline was flooded with a lot of bullshit (like several toots/second), so everybody thought it's just other users having large follow lists; as a result, nobody used it and nobody noticed there were any nazi instances in it.

* On approx. December 3rd, 2019, we had a system-wide crash caused by our 100GB media storage running out of space:

Show thread

An (incomplete) post-mortem on recent federation problems there (3/?) 

* "Most recent activity" of this user was on October 12, 2019. It is possible that all of these fake follows were made during that day.

* On October 25, 2019, suspended this user on their side, with a public announcement:
It seems that at no point admins attempted to contact this instance, or to issue any moderation report. We had no way of knowing about that announcement.

Show thread

An (incomplete) post-mortem on recent federation problems there (2/?) 

* It is possible that it was an actual person at that time.

* However, somewhere between March and October, someone gained a control on this user (or maybe it was an evildoer from the beginning), and started to follow users from other instances en masse. Including from nazi and "free speech" instances - one we did not yet know about, and so did not block. It followed about 250 users in total, mostly ones who post a lot.

Show thread

An (incomplete) post-mortem on recent federation problems there (1/?) 

This instance has faced federation problems for the past months (almost a year). Some other instances unfederated from it. Here is a brief outline of what happened:

* Originally (since its establishment in August 2018), this instance had open registrations. Anybody could register.

* On March 13, 2019, a new user registered there. It posted three toots during its life (I cannot recover the exact dates now)...

Mastodon updated to v3.1.5. Sorry for the interruption. instance is blocked.
Reason: the same as for

List of currently blocked instances:

@admin @self
Hi! I've noticed on your "about" page ( that you have limited federation with this instance ( I have no idea why, I've never received any moderation report from you, and I'd hate to be in the same list as gab. Could you please explain why did that happen?

Blocked instances:
Reason: nazis and bigots.

Blocked instance:
Reason: bitcoin.

Hi! While I've been cleaning my instance, I came upon your moderation alert:
(As a side note, it does indeed seem to be a follow bot, and with that follow list, this account alone consumed ~98% of my instance resources and turned our federated timeline into garbage)

But sadly I've never received any moderation report from you. Not sure if that's how it's supposed to be, or some problem with federation? instance is blocked. Reason: nazi TERFs

Версия Mastodon на этом инстансе обновлена до 2.9.2, технические работы завершены

В ближайший час-два будут проводиться технические работы, инстанс может быть недоступен. instance is blocked. Reason: nazis instance is silenced. Reason: hardcore NSFW userpics as an instance policy.

Технические работы последних двух дней, для которых приходилось на некоторое время отключать сервер, завершены. В ближайшее время отключений не планируется.

Show older
Embracing space

The social network of the future: No ads, no corporate surveillance, ethical design, and decentralization! Own your data with Mastodon!