this post was submitted on 24 Jun 2023

5 points (72.7% liked)

Lemmy Administration

698 readers

1 users here now

Anything about running your own Lemmy instance. Including how to install it, maintain and customise it.

Be sure to check out the docs: https://join-lemmy.org/docs/en/administration/administration.html

If you have any problems, describe them here and we will try to help you fixing them.

founded 4 years ago

MODERATORS

nutomic@lemmy.ml

Lemmy as a project has suffered all month because Lemmy.ml has not been sharing critical logs from Nginx and Lemmy's code logging itself (lemmy.ml)

submitted 1 year ago* (last edited 1 year ago) by RoundSparrow@lemmy.ml to c/lemmy_admin@lemmy.ml

48 comments fedilink hide all child comments

Lemmy.ml front page has been full of nginx errors, 500, 502, etc. And 404 errors coming from Lemmy.

Every new Lemmy install begins with no votes, comments, postings, users to test against. So the problems related to performance, scaling, error handling, stability under user load can not easily be matched given that we can not download the established content of communities.

Either the developers have an attitude that the logs are of low quality and not useful for identifying problems in the code and design, or the importance of getting these logs in front of the technical community and trying to identify the underlying patterns of faults is being given too low of a priority.

It's also important to make each log of failures identifiable to where in the code this specific timeout, crash, exception, resource limit is encountered. Users and operations personnel reporting generic messages that are non-unique only slow down server operators, programmers, database experts, etc.

There are also a number of problems testing federation given the nature of multiple servers involved and trying not to bring down servers in front of end-users. It's absolutely critical that failures for servers to federate data be taken seriously and attempts to enhance logging activities and triangulate causes of why peer instances have missing data be track down to protocol design issues, code failures, network failures, etc. Major Lemmy sites doing large amounts of data replication are an extremely valuable source of data about errors and performance. Please, for the love of god, share these logs and let us look for the underlying causes in hard to reproduce crashes and failures!

I really hope internal logging and details of the inner workings of the biggest Lemmy instances is shared more openly with more eyes on how to keep scaling the applications as the number of posts, messages, likes and votes continue to grow each and every day. Thank you.

Three recently created communities: !lemmyperformance@lemmy.ml -- !lemmyfederation@lemmy.ml -- !lemmycode@lemmy.ml

top 23 comments

sorted by: hot top controversial new old

[–] monobot@lemmy.ml 6 points 1 year ago (2 children)

I agree with you, but I think you sound a bit too harsh for developers.

I think they are doing their best currently and have probably identified more immediate issues before addressing all that we see.

There are other big instances which could share the logs, let's ask lemmy.world and beehaw if they can share the logs and leave main developers to work.

Another bug thing I am thinking can benefit from information sharing is bot account detection.

I would like to take a look at that data and find ways to identify bots. I just don't know what data can be useful, but will try to make my own instance and work on it.

[–] RoundSparrow@lemmy.ml 2 points 1 year ago (1 children)

I think you sound a bit too harsh for developers.

Where is the caching in the Rust code for databases? Why isn't caching being discussed in a Lemmy community using Lemmy itself as a venue for discussing the major performance problems in the code?

The thread this comment is from: https://www.reddit.com/r/rust/comments/zvt1mu/tips_on_scaling_a_monolithic_rust_web_server/

[–] monobot@lemmy.ml 0 points 1 year ago (1 children)

https://github.com/LemmyNet/lemmy/issues/2975

Caching is being worked on in shape of cache control headers, not in a way you mention sql cache, but will get better.

If it shows it itms not enough I can imagine devs will change their opinion on it, like they did with websockets.

[–] RoundSparrow@lemmy.ml 0 points 1 year ago* (last edited 1 year ago)

not in a way you mention sql cache, but will get better.

Avoiding the SQL datababase and use caching is webapp programming 101, it is fundamental to all the crashes Lemmy is showing. We are talking the MOST BASIC thing in creating webapps. I really can't over-emphasize this point.

You don't go query the site table every single time a federation incoming comment comes in.

SELECT "local_site"."id", "local_site"."site_id", "local_site"."site_setup", "local_site"."enable_downvotes", "local_site"."enable_nsfw", "local_site"."community_creation_admin_only", "local_site"."require_email_verification", "local_site"."application_question", "local_site"."private_instance", "local_site"."default_theme", "local_site"."default_post_listing_type", "local_site"."legal_information", "local_site"."hide_modlog_mod_names", "local_site"."application_email_admins", "local_site"."slur_filter_regex", "local_site"."actor_name_max_length", "local_site"."federation_enabled", "local_site"."federation_worker_count", "local_site"."captcha_enabled", "local_site"."captcha_difficulty", "local_site"."published", "local_site"."updated", "local_site"."registration_mode", "local_site"."reports_email_admins" FROM "local_site" LIMIT $1

And back to the very subjhect line of this posting, you SHARE YOUR CRASH LOGS when your server is crashing, why is lemmy.ml not putting the crash logs up on Girhub issues when for 30 days I've seen 500 errors on the front page?

[–] RoundSparrow@lemmy.ml 0 points 1 year ago* (last edited 1 year ago) (2 children)

I agree with you, but I think you sound a bit too harsh for developers.

The failure to inform users by official announcement or mention in the 0.18 release notes that Lemmy is failing to replicate data reliably I think is a failure of the project management. "your data doesn't matter here on the Lemmy network". Why are end users not being told that their messages are in fact not reliably being shared to other instances? Why are the server install and release notes not warning the community that each additional instance being brought online is increasing the replication workload of establishes sites - that are already faltering?

The problem is being covered up, brushed under the rug. The issues of creating tools to adequately load test federation and track problems wasn't raised during project development as an important ToDo item, call for assistance, nor has it really been noticed by most of the server operators. I've personally been going around to dozens of Lemmy instances and hand observing the failures to replicate data. No thought was put into even the most primitive tools to operate a server and have a sense of 'how would you know' if federation was failing?

Yet, the leaders of Lemmy have created directories of "recommended sites" to go sign up with and given the impression that you can access active communities from peer instances to help offload the server reliability problem. Federation itself is unreliable on Lemmy to Lemmy!

Either they are covering up the problem, hiding it out of pride, or not opening bugs on GitHub or not calling for help in the 0.18 Release Notes. Which is it?

[–] ericjmorey@lemmy.world 5 points 1 year ago* (last edited 1 year ago) (1 children)

The problem is being covered up, brushed under the rug

Either they are covering up the problem, hiding it out of pride, or not opening bugs on GitHub or not calling for help in the 0.18 Release Notes.

You've raised many good points to come to a wildly accusatory conclusion.

Get off of this line of thinking if you want to raise support for fixing the valid issues you've raised.

Have you made any attempt to see if these issues have been raised on GitHub? Have you made any attempt to create issues on GitHub? Have you made any attempt to submit code enhancements via GitHub?

[–] RoundSparrow@lemmy.ml 2 points 1 year ago (1 children)

Have you made any attempt to see if these issues have been raised on GitHub? Have you made any attempt to create issues on GitHub? Have you made any attempt to submit code enhancements via GitHub?

Have the developers of Lemmy put a message to end-users that data is being lost? have the operators of servers opened issues on GitHub about 'pending subscribe' failures on federation? Have the developers of Lemmy put warnings in the Release Notes that scaling is an active concern and needs urgent attention? Have the operators of major Lemmy websites published their nginx server logs and application code logs so that the bugs and design problems fo the code are logged?

Ir is the logging worthless, so poor, that they don't share logs?

Most of all,e at your own dogfood, where are Lemmy developers using Lemmy to archive discussions about scalability and performance issues causing major crashes?

"I like Rust" but don't like using Lemmy to discuss the performance concerns of scaling and data integrity in a monolithic Rust application?

[–] ericjmorey@lemmy.world 1 points 1 year ago

It looks like you want to vent and issue demands of others rather than be productive. I hope that serves you well.

[–] monobot@lemmy.ml 1 points 1 year ago (1 children)

I like your posts and hunt for performance issues, I just think that developers decided (wether you and I agree or not) some other features are more important.

Until few weeks ago communication was clear since there were not many people here, so there was no need for some specific notes you are mentioning.

Now we do need them and reminding developers of it, or even better doing it would be much appreciated I expect.

I have seen developers on some threads here or on github issues commenting on repication problems and they are hard at work for those.

Even caching is discussed, as I understand, they first need to implement cache control headers so that admins can set up caching, as they see fit, outside lemmy.

There is a lo of good will around, please have understanding and be part of it. Give the time to grow up to this opportunity.

[–] RoundSparrow@lemmy.ml -1 points 1 year ago* (last edited 1 year ago) (1 children)

Even caching is discussed, as I understand,

Where? On Lemmy? Why do they think Lemmy isn't good enough to discuss Lemmy?

Sure a lot of people in denial that"eat your own dogfood" is a concept in development. You are creating a messaging system, why aren't you using it?

I just think that developers decided (wether you and I agree or not) some other features are more important.

End-user comments not making it out of a server to peer servers is unimportant. That is how I would describe the Lemmy Release Notes for 0.18 - that is how I would describe the response I got to the GitHub issue I opened..

There is a lo of good will around, please have understanding and be part of it. Give the time to grow up to this opportunity.

I see you want me to shut my mouth up and not be honest about how unreliable Lemmy is. I've been authoring social media messaging apps since 1984. But tell me to "grow up". The project management doesn't need to "grow up", when you can't even see them using Lemmy to discuss Lemmy programming and database design?

So far, this community is insular and "Feedback not welcome" has been the response. Lemmy branding mania with no reality about how unstable and unreliable Lemmy is right here, right now.

[–] iamhazel@beehaw.org 3 points 1 year ago (1 children)

I understand their GitHub comment is a little confusing, but interpreting it as telling you to grow up is... telling.

[–] RoundSparrow@lemmy.ml 0 points 1 year ago* (last edited 1 year ago) (1 children)

I understand their GitHub comment is a little confusing, but interpreting it as telling you to grow up is… telling.

Please explain it to me then, if I misinterperted?

The creators of Lemmy have misinterpreted how to read books on when to add caching layers to a webapp, and how to test with significant amounts of data.

Am I misinterpreting the project management's incompetence and areas that need improvement?

Help me out in interpretation please, i have autism, and I don't always interpret things the same as other people. In fact, i question if people take interpretaiton for granted and like mocking and insulting each other as a way to deflect truth and honesty in social matters.

interpreting it as telling you to grow up is… telling.

I interpreted the 500 errors on the front page of Lemmy.ml with 40 years of social media application development expertise under my belt. Maybe it is you who is interpreting the code development, server operations and how badly it iis being done wrongly? Do you know how to interpret the pattern of nginx 500 errors and missed federation comment replication?

Are you gaslighting me,m intimidating me as a human person, dehumanizing me?

[–] iamhazel@beehaw.org 4 points 1 year ago (1 children)

Are you gaslighting me,m intimidating me as a human person, dehumanizing me?

Yes I was trying to do exactly all those things, you got me.

And I'm autistic too. That doesn't preclude you from having a superiority complex or just being an asshole.

[–] RoundSparrow@lemmy.ml 0 points 1 year ago* (last edited 1 year ago)

Yes I was trying to do exactly all those things, you got me.

Yes, I did get you, as you obviously can not talk about the lack of sharing logs from the busy server that was crashing hourly / falling over itself under moderate load.

And I’m autistic too. That doesn’t preclude you from having a superiority complex or just being an asshole.

Yes, check yourself, you already admitted "yes" that you are gaslighitng me. That you are trying to intimidate me. I'm sorry you have been abused so much for being autistic and are afraid of facts and truth about an open source project that's being mismanaged. That your identity with a software application exeeds your love for human persons. Maybe read BIble page "1 John 4:20" for inspiration. You clearly seem like a damaged individual that society has harmed.

My name is Stephen Alfred Gutrknecht, what's your name since we are sharing personal details about our mental health?

[–] sunaurus@lemm.ee 6 points 1 year ago (1 children)

Hey buddy, I understand you're frustrated, but I just want to make a few points:

I have personally seen many instance admins and Lemmy contributors note many times over the past weeks that Lemmy is unoptimized and not ready for the current traffic
I have myself mentioned it several times in announcements to users of my own Lemmy instance
Lemmy maintainers have asked for help with optimization in several channels
Lemmy maintainers are clearly working hard at fixing Lemmy issues and improving performance - just look at the work that went into 0.18 - the fact that it's far from perfect is clear to everybody, but progress is constantly being made
Lemmy maintainers have mentioned multiple times that their inboxes are full of notifications and DMs - it's not that they're brushing anything under the rug, it's just that they're not physically able to keep up with the volume of communication that is being thrown at them

I really believe that you have some useful insights and can be very helpful for Lemmy, but I'm afraid that if you take this accusatory tone and blame people for not doing enough then that will overshadow anything helpful that you're actually saying.

Having said all that, if you would like to take a look at some stats about queries on lemm.ee (a Lemmy instance with 4k users - definitely much smaller than lemmy.ml), I have put together a spreadsheet here: https://docs.google.com/spreadsheets/d/e/2PACX-1vSPpqM6QCZYAAvnWe8p-xxN553ukRIquHw71j3nB763x7TNeqeUO-Oss51yPC7zVaT2x4jll39NCeMu/pubhtml#

[–] RoundSparrow@lemmy.ml 2 points 1 year ago* (last edited 1 year ago)

Lemmy maintainers have asked for help with optimization in several channels

I do not see them using Lemmy itself to actually discuss the problems of Lemmy. Specific to lemmy.ml and the developer relationship with this specific server, crashes (logs) are not being shared.

10 days ago: https://lemmy.ml/post/1271936

I can not emphasize the title of the posting you are reading enough. "Lemmy as a project has suffered all month because Lemmy.ml has not been sharing critical logs from Nginx and Lemmy's code logging itself"

Logs, logs, logs. Why were these crash logs not shared as part of the Lemmy project? When the most busy server on the whole project is not sharing their Rust code logs and crashes, what are us trying to work on the SQL and architecture problems supposed to do? I didn't even report 1 in 100 of the crashes I was experiencing.

It is a peer to peer network, server to server, and the central hub has encouraged everyone to run out and create new servers without any concern to report the crashes going on within the central hub. I just don't get why everyone here is defending such behavior and leadership.

What I see was sharing of CONCLUSIONS - that "increase the worker count" was the problem. No, the problem is fundamental to the whole Rust application's automatically generated SQL statements, lack of data caching, lack of proper MTA and queue for federation inbound and outbound data. Just saying that the federation worker count was the problem and making the value infinite was not in any way getting to the problems that sharing the server crash logs would have exposed.

June 14, the GitHub issue on "Scaling Federation" was CLOSED by project leadership! Meanwhile, lemmy.ml was crashing for me every hour! Failing to federate with any reliability too. June 15 is when https://lemmy.ml/post/1271936 was opened, the day after this CLOSE of a GitHub issue:

The DDOS is coming from WITHIN THE HOUSE. Lemmy's performance problems are causing federation to bring down peer servers, and the LOGS of Rust code exceptions that are being KEPT SECRET will reveal this! The sharing of logs and making this a federation-wide announcement that the hub is failing on data exchange is critical, not optional

It's sad to me that the leadership of this project can't just come out and openly admit it is "experimental" project and "unstable", and is ignoring https://lemmy.ml/post/1271936 and bragging on GitHub that it is "high performance Rust". It might have seemed high performance when you sent 8 whole test messages to 4 servers a day, but that isn't the meaning of "high performance". depressing to see such denial and the people who believe in the "reality distortion field" around the project.

[–] RoundSparrow@lemmy.ml 2 points 1 year ago* (last edited 1 year ago) (1 children)

So far, I've gotten nothing but replies that do not talk about the failure to show logs and the importance of logging in server applications.

Logging matters
Sharing logs matter
Server apps don't have a nice GUI, you use logs
Logging matters

Replies are DEFLECTING the problem

Do I need to keep repeating how much lemmy.ml's UNAVAILABLE logs of actual failures has been holding back the entire platform and community? this should be blindly obvious to anyone who has built and supported big client/server apps - what do the error logs say is crashing when you get a 500 error? Issue a BOLO to other server operators on Github or on LEMMY social media platform!

Data integrity, failure to replicate comments to peer instances, is also being ignored. WHY WAS THIS NOT IN THE 0.18 RELEASE NOTES when the application is being pushed as 'High Performance' is on the front of the Github page?

Lemmy isn't being used to even discuss the technical problems of Lemmy. "Eat your own dogfood" isn't cared about here. The people running servers aren't reporting major problems and sharing logs to the community.

[–] ericjmorey@lemmy.world 1 points 1 year ago

The replies are a reflection of your abrasive and antisocial approach to this. Start working with people instead of yelling at them and maybe you can help improve the situation.

[–] RoundSparrow@lemmy.ml 1 points 1 year ago* (last edited 1 year ago)

GO ahead, keep trying to gaslight me that Lemmy is "high performance" server application

This is on a low-traffic community with a targeted audience, not on Reddit, not on /c/Lemmy

I have the receipts, I know where the bugs are, I know how much this problem is being IGNORED and I am personally being gaslight that the problem isn't real and true. DEFLECTION is the first response in this unhealthy community to newcomers who know their shit.

It's a disgrace to Rust, Linux, and PostgreSQL that this false statement is on the home page of GitHub for Lemmy.

“In the land of the blind, the one-eyed man is a hallucinating idiot...for he sees what no one else does: things that, to everyone else, are not there.” ― Marshall McLuhan

EDIT: I see your downvotes, and you replies intimidating me. Praise "the powers that be" of the project, eh?

[–] RoundSparrow@lemmy.ml 1 points 1 year ago* (last edited 1 year ago)

Lemmy community is an echo-chamber of people ignoring the fact that the developers bold claim "HIGH PERFORMANCE" on the GitHub project page without any validation.

Not one person was willing to stand up and say "this is LOW performance, amature mistakes in not having caching of" SELECT "local_site"."id", "local_site"."site_id", "local_site"."site_setup", "local_site"."enable_downvotes", "local_site"."enable_nsfw", "local_site"."community_creation_admin_only", "local_site"."require_email_verification", "local_site"."application_question", "local_site"."private_instance", "local_site"."default_theme", "local_site"."default_post_listing_type", "local_site"."legal_information", "local_site"."hide_modlog_mod_names", "local_site"."application_email_admins", "local_site"."slur_filter_regex", "local_site"."actor_name_max_length", "local_site"."federation_enabled", "local_site"."federation_worker_count", "local_site"."captcha_enabled", "local_site"."captcha_difficulty", "local_site"."published", "local_site"."updated", "local_site"."registration_mode", "local_site"."reports_email_admins" FROM "local_site" LIMIT $1

It's outright disillusion that people labeled it "HIGH PERFORMANCE".

The Lemmy community sure have the bullying part and shit-talking peer applications down on Social media, but the project internally shows that there is no street credibility when it comes to "HIGH PERFORMANCE" bold claims against even the basic email systems.

Lemmy admins who won't use Lemmy to talk about and try to solve their hourly server crashes .

Gaslighting me about it was way more serious than me speaking up bravely about the problem. Intimidation tactics you are showing to me say a ton about why you have these problems in the first place. The lack of concern for the end-user data being lost also speaks volumes as to priorities.

[–] RoundSparrow@lemmy.ml 1 points 1 year ago

[–] RoundSparrow@lemmy.ml 1 points 1 year ago* (last edited 1 year ago)

There was a time, far in the past before Reddit mocking became the central focus. that sharing server logs was actually recommended:

But then nginx 500 errors came along, and nobody running big servers bothered to share their crash logs.

[–] RoundSparrow@lemmy.ml 1 points 1 year ago

People report the problem on Github, server operators ignore it, and they give up. The problem is being systemically ignored, and those who persistently raise that there are serious problem in the design, execution, and operation of the code - are ignored, silenced and deflected.

This isn't me, this is another person ignored on Girhub:

RELEASE 0.18 went out without any warning of the ongoing federation failures.

load more comments