We’re aware of ongoing federation issues for activities being sent to us by lemmy.ml.
We’re currently working on the issue, but we don’t have an ETA right now.
Cloudflare is reporting 520 - Origin Error when lemmy.ml is trying to send us activities, but the requests don’t seem to properly arrive on our proxy server. This is working fine for federation with all other instances so far, but we have seen a few more requests not related to activity sending that seem to occasionally report the same error.
Right now we’re about 1.25 days behind lemmy.ml.
You can still manually resolve posts in lemmy.ml communities or comments by lemmy.ml users in our communities to make them show up here without waiting for federation, but this obviously is not something that will replace regular federation.
We’ll update this post when there is any new information available.
Update 2024-11-19 17:19 UTC:
Federation is resumed and we’re down to less than 5 hours lag, the remainder should be caught up soon.
The root cause is still not identified unfortunately.
Update 2024-11-23 00:24 UTC:
We’ve explored several different approaches to identify and/or mitigate the issue, which included replacing our primary load balancer with a new VM, updating HAproxy from the latest version packaged in Ubuntu 24.04 LTS to the latest upstream version, finding and removing a configuration option that may have prevented logging of certain errors, but we still haven’t really made any progress other than ruling out various potential issues.
We’re currently waiting for lemmy.ml admins to be available to reset federation failures at a time when we can start capturing some traffic to get more insights on the traffic that is hitting our load balancer, as the problem seems to be either between Cloudflare and our load balancer, or within the load balancer itself. Due to real life time constraints, we weren’t able to find a suitable time this evening, we expect to be able to continue with this tomorrow during the day.
As of this update we’re about 2.37 days behind lemmy.ml.
We are still not aware of similar issues on other instances.
Removed by mod
Unfortunately not entirely true. They have the largest Linux Community. To be precise, the big one.
ProgrammerHumor is there as well. The Open Source and Privacy communties are very bigs as well there. Completely dwaring their .world counterpart.
I’m not a fan of .ml either but you can’t deny that they have this 3 parts (which very much represent the ideals behind lemmy) covered very well.
I think things like this can be helpful at curbing network effect. Which can only be really curbed by force. Beehaw used to have quite strong network effect in some communities too in the earlier days, but now, they have much less of a vice, and are actually much quieter than they were before.
Might be a good idea to check out !de_ml@lemmy.blahaj.zone in these trying times. In case it’s a while before the issue is fixed.
I’m out of the loop. is there some post or link you can send me about . ml being malicious
I don’t have a link because it’s been a while, but others and I have been piled-on by .ml and Hexbear users. In my case, 40+ people replied with nasty comments to a single comment I had made. And then I got straight-up banned from .ml because they misinterpreted what I was saying about communism. A mod cited whatever ambiguous and unrelated bullshit sidebar rule and I got the hammer anyway calling me a troll. But all the nasty circle-jerk comments stayed up. So fuck’em.
My experiences mirror this, tho not to that degree. It’s similar to Mos Eisley over there.