this post was submitted on 19 Jun 2025
25 points (87.9% liked)

Lemmy

13645 readers
9 users here now

Everything about Lemmy; bugs, gripes, praises, and advocacy.

For discussion about the lemmy.ml instance, go to !meta@lemmy.ml.

founded 5 years ago
MODERATORS
25
submitted 3 weeks ago* (last edited 1 week ago) by KaKi87@jlai.lu to c/lemmy@lemmy.ml
you are viewing a single comment's thread
view the rest of the comments
[–] KaKi87@jlai.lu 2 points 2 days ago (1 children)

What am I looking for in the logs though ? All I see is "InboxTimeout"...

[–] kernelle@0d.gs 1 points 2 days ago (1 children)

/inbox is the endpoint for federation with ActivityPub, so definitely on the right track. Try increasing verbosity to get a full stack traceback to where in the code it's bugging.

InboxTimeout means your server is taking too long to process API calls to /inbox, I've had them before too due to misconfiguration of the reverse proxy. Make sure you're passing on the right headers, especially the IP headers so they aren't rate limited and the Accept header so your server knows to send json when requested.

Troubleshooting federation gives many options for testing and checking your configuration, running through them might give your more insight.

[–] KaKi87@jlai.lu 2 points 23 hours ago (1 children)

So, here's the full error :

lemmy-1  | 2025-07-14T00:34:32.098630Z DEBUG HTTP request{http.method=POST http.scheme="https" http.host=blog.kaki87.net http.target=/inbox otel.kind="server" request_id=a9c359f6-eb9d-4888-9ac2-55c1f5bb8e43}: actix_web::middleware::logger:
Error in response: LemmyError { message: InboxTimeout, inner: InboxTimeout, context: SpanTrace [{ target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "http.method=POST http.scheme=\"https\" http.host=blog.kaki87.net http.target=/inbox otel.kind=\"server\" request_id=a9c359f6-eb9d-4888-9ac2-55c1f5bb8e43", file: "src/root_span_builder.rs", line: 16 }] }
lemmy-1  | 2025-07-14T00:34:32.098834Z  WARN Error encountered while processing
the incoming HTTP request: lemmy_server::root_span_builder: InboxTimeout: InboxTimeout

I've also been seeing this one :

lemmy-1  | 2025-07-14T00:41:00.407762Z DEBUG HTTP request{http.method=GET http.scheme="http" http.host=lemmy:8536 http.target=/api/v3/site otel.kind="server" request_id=9aab8319-3f3b-4f4e-8530-3b3633afe2f1}: actix_web::extract: Error for Option<T> extractor: IncorrectLogin: IncorrectLogin
lemmy-1  | 2025-07-14T00:41:00.427630Z DEBUG HTTP request{http.method=GET http.scheme="http" http.host=lemmy:8536 http.target=/api/v3/post/list otel.kind="server" request_id=ef746ac7-b9a3-4615-bb37-4c12f0ed0827}: actix_web::extract: Error for Option<T> extractor: IncorrectLogin: IncorrectLogin

As for the reverse proxy, I'm using Caddy with the configuration provided here, and it's supposed to pass those headers by default.

The suggested curl requests work.

I wish there was a tool that could just tell what's wrong, like https://federationtester.matrix.org/ for Matrix...

[–] kernelle@0d.gs 1 points 1 hour ago

Hmmm, some InboxTimeouts are to be expected. The second one could be the result of a scraper or bot. It's still strange your federation list is empty. I'd take a look at the database, clearing the federation queue (public.federation_queue_state) is a good idea. 'public.instance' should be populated with servers (mine has 2836). Peaking into the log files of the database could give some more insight as well.

A successfully federated incoming action will look like this: 2025-07-14T22:40:54.321151Z INFO HTTP request{http.method=POST http.scheme="http" http.host=0d.gs http.target=/inbox otel.kind="server" request_id=61bf0ea2-8df3-42ff-a433-3bec8b37047c}: actix_web::middleware::logger: 10.10.0.1 'POST /inbox HTTP/1.1' 200 0 '-' 'Lemmy/0.19.12; +https://lemmy.ml/' 0.158242

Your outgoing federation seems to be fine now on my end, this is how my server sees yours:

id 2794; ;domain "blog.kaki87.net"; published "2025-06-30T11:35:32.398830Z" ;updated "2025-07-14T00:42:32.758669Z" ;software "lemmy" ;version "0.19.11" ;federation_state: ;instance_id 2794 ;last_successful_id 46037 ;last_successful_published_time "2025-07-14T21:12:27.481507Z" ;fail_count 0

You could try updating or reinstalling your back-end as well, lemmy is at 0.19.12.