this post was submitted on 27 Sep 2024
29 points (100.0% liked)

Programming

17391 readers
170 users here now

Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!

Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.

Hope you enjoy the instance!

Rules

Rules

  • Follow the programming.dev instance rules
  • Keep content related to programming in some way
  • If you're posting long videos try to add in some form of tldr for those who don't want to watch videos

Wormhole

Follow the wormhole through a path of communities !webdev@programming.dev

founded 1 year ago
MODERATORS
snowe@programming.dev
Ategon@programming.dev
MaungaHikoi@lemmy.nz
29
Lemmy federation protocol: How is the "posts from all communities" view sourced? (feddit.org)
submitted 1 month ago by 7EP6vuI@feddit.org to c/programming@programming.dev
8 comments fedilink hide all child comments
 

I'm a newbie to ActivityPub so please be patient with me.

All intros into ActivityPub speak about how a user of a server A subscribes to a specific community from server B, and then server A will be informed about changes in that community.

But on lemmy it's possible to look at the posts of all communities. For a single concrete community it would be relatively easy: server A gets the request to serve the top post of a community on server B, so A simple asks B for the posts.

But there is also the "posts from all communities" tab on the lemmy front page. This opens questions:

Does each lemmy instance has a full copy of all posts of all communities? If this is true: How are new Instances discovered? Is each Instance distributing all updates to all other Instances?

If each lemmy instance has only a partial dataset (this theory is backed by [1] "Only if a least one user on your instance subscribes to the remote community, will the community send updates to your instance.") then how is the "all posts" view composed? is it in reality not "all" but only "all posts that at least one user of this instance is subscribed to"?

If this is the case: what happens if a bad actor subscribes to all communities of all servers? Is there a maximum number of subscriptions per user?

The source of those questions is, that I'm looking for a way to subscribe to all events of all lemmy instances, to be able to build statistics about upvotes, new posts, comments etc. There seems to be a similar API endpoint for mastadon [2] but nothing for lemmy?!

you are viewing a single comment's thread
view the rest of the comments
[–] 7EP6vuI@feddit.org 1 points 1 month ago (1 children)

Thanks for the suggestion and the code snippets.

i want to see how votes/comments accumulate over time on a post, therefore i would have to poll the "all" posts endpoint in a regular interval. but I would either see new posts with small number of comments/upvoted, or already upvoted post, or i would have to download all posts in a regular time interval which seems impossible to me.

[–] rust_alt@discuss.online 4 points 1 month ago* (last edited 1 month ago)

Comments are also easy, the API allows pulling them by latest too. If I was writing a search engine, I would probably just track all known instances and just pull local content from them instead of deduplicating. I haven't really looked at how votes are federated though, so that might be more complicated to keep updated.

I expect just syncing posts and comments from all instances to be mostly easy. In the past I was able to pull all posts and comments from smaller instances in like less than 10 minutes. It's mostly just text so it doesn't take that long. After it's pulled, it can be kept mostly up to date by just pulling to the last date received, and should take much less time than the first sync.

I've noticed there's lots of stuff on Lemmy that fails to federate to other instances. I think there's also actually a 3000€ reward at the moment for improving federation, so if you spend very much time on it, it might be a good idea to see if it can be claimed. Though, I don't really know how the milestone system works, and it might only be available to inside contributors.