59
Lemmy.World federation is on the fritz again (lemmy.dbzer0.com)
submitted 6 months ago* (last edited 6 months ago) by GuyFleegman@lemmy.dbzer0.com to c/div0@lemmy.dbzer0.com
24 comments fedilink hide all child comments

/c/news@lemmy.world is the second biggest community on Lemmy.World and yet on /0, there is nothing newer than two days.

/c/technology@lemmy.world has two posts from today but based on the vote count, I think it's only showing votes from this instance

top 24 comments
sorted by: hot top controversial new old
[-] empireOfLove2@lemmy.dbzer0.com 21 points 6 months ago

I think the real problem is world is getting too big. They are by far the largest instance by users, content and post volume. The federation bandwidth requirements (and front end serving requirements) have got to be insane with 7200 people actively posting, with thousands more federating in. It exposes all the cracks that lemmy inevitably has in it's underlying data handling.

[-] ada@lemmy.blahaj.zone 24 points 6 months ago* (last edited 6 months ago)

It's more to do with the lemmy itself as a platform handles than bandwith. Basically, there is only one channel between any two instances, and it's serial, and each step requires multiple handshakes to complete. Add in geographic distance making those handshakes take a significant part of a second to complete, and you end up with a single channel that gets flooded.

Blahaj.zone is 1.3 million activities behind on lemmy.world for example, but we're not behind on any other instance, because those channels don't hit capacity. Now, if we could use multiple channels at once to talk to lemmy.world, we wouldn't have a problem, but lemmy isn't built for that at the moment

[-] db0@lemmy.dbzer0.com 7 points 6 months ago

Are you in the matrix general chat?

[-] ada@lemmy.blahaj.zone 6 points 6 months ago

I'm in the lemmy.world admin back channel, but that's it as far as lemmy.world goes. @ada:chat.blahaj.zone

[-] db0@lemmy.dbzer0.com 6 points 6 months ago
[-] ada@lemmy.blahaj.zone 5 points 6 months ago

I left that channel when one of the lemmy devs decided to take a swipe at our instance for the way we handled transphobia on another instance

[-] db0@lemmy.dbzer0.com 3 points 6 months ago

Oh :(

[-] Blaze@dormi.zone 3 points 6 months ago* (last edited 6 months ago)

Lemmy.world is really too big. We just moved !casualconversation@lemmy.world to !casualconversation@lemm.ee to try help with those issues.

Hopefully other communities will do the same, but the issue doesn't seem to be very well known.

By the way, thank you for your post about centralization of communities a while back, I used it in the post to explain to the community why we were moving: https://lemmy.blahaj.zone/post/10810804?scrollToComments=true

[-] empireOfLove2@lemmy.dbzer0.com 2 points 6 months ago* (last edited 6 months ago)

and you end up with a single channel that gets flooded.

The idea behind federation was that you don't get single instances so large that the single sync channel gets flooded. You're meant to have more smaller instances spread out that then federate with one another like a mesh- only one copy of data needs to be sent to each instance to serve additional thousands of users without further load.

That said, realistically the fediverse needs single instances with 10-30k users to really thrive, and L.W hasn't even hit 10k actives... so there does need to be some batch improvements in the backend still.

Having the ability to "merge content" from multiple instance communities would help a lot too. People congregate on L.W both from inside and outside the instance because it's big and has the most content. But if you have multiple communities (e.g, search "memes" and see how many different memes communities there are) the others with less content get ignored. Being able to merge those from a user viewing perspective, so you would just have a "memes" group that sees federated content from all the memes communities, could reduce the need for L.W to be all encompassing.

[-] OnlyTakesLs@sh.itjust.works 0 points 6 months ago

Couldnt they batch them up? Im not a technical person but this seems solvable.

[-] ada@lemmy.blahaj.zone 7 points 6 months ago

Yep. Batching and/or multiple parallel channels per remote instance would solve it.

[-] db0@lemmy.dbzer0.com 11 points 6 months ago

Issue was on lemmy.world side. It should start getting resolved now.

[-] GuyFleegman@lemmy.dbzer0.com 3 points 6 months ago

Thank you for looking into it!

[-] ptz@dubvee.org 10 points 6 months ago* (last edited 6 months ago)

I think it's something where Kbin sends an insane amount of duplicate AP messages or something like that.

Every time my instance falls behind with them, there's usually some community ban in LW's modlog soon after about a kbin user sending suspicious traffic. My understanding is that's unintentional on that user's part.

I thought there was a post breaking down the problem but I can't seem to find it.

Edit: It was in admin chat, not a post (or if there was a post, I still can't find it).

Kbin (the software) is like the bad neighbor in the fedihood.

[-] Blaze@dormi.zone 3 points 6 months ago

Mbin is the nicer version of Kbin

[-] ptz@dubvee.org 2 points 6 months ago

Yeah, I've read good things about it.

[-] Zuberi@lemmy.dbzer0.com 9 points 6 months ago

I'm on team fuck lemmy.world because their bot problem is obscene

[-] originalucifer@moist.catsweat.com 8 points 6 months ago

whew. i thought i was goin crazy.. i see traffic in my logs but no new posts

[-] LinkOpensChest_wav@lemmy.blahaj.zone 8 points 6 months ago

Same here on blahaj. Barely any posts from .world for the past couple days coming through.

[-] ada@lemmy.blahaj.zone 9 points 6 months ago

We're 1.3 million posts behind l.w. at the moment.

[-] LinkOpensChest_wav@lemmy.blahaj.zone 4 points 6 months ago

Wow, so is it just that .world has become too large?

[-] ada@lemmy.blahaj.zone 11 points 6 months ago

.world is too large for the way lemmy is currently designed. It's a software limit, not a hardware limit.

[-] db0@lemmy.dbzer0.com 7 points 6 months ago

sigh

This seems to be way more impacting at this point. I'll check if there's anything I can do, but since it seems to be affecting other larger instances as, not sure how much that would be.

[-] FartsUnited@lemmy.dbzer0.com 2 points 6 months ago

I can't seem to reply to anything, and was surprised to see that this thread is full of replies.

this post was submitted on 11 Apr 2024
59 points (98.4% liked)

/0

1546 readers
1 users here now

Meta community. Discuss about this lemmy instance or lemmy in general.

Service Uptime view

founded 1 year ago
MODERATORS
db0@lemmy.dbzer0.com