this post was submitted on 22 Jul 2023
2315 points (98.8% liked)

Lemmy.World Announcements

29079 readers
245 users here now

This Community is intended for posts about the Lemmy.world server by the admins.

Follow us for server news 🐘

Outages πŸ”₯

https://status.lemmy.world/

For support with issues at Lemmy.world, go to the Lemmy.world Support community.

Support e-mail

Any support requests are best sent to info@lemmy.world e-mail.

Report contact

Donations πŸ’—

If you would like to make a donation to support the cost of running this platform, please do so at the following donation URLs.

If you can, please use / switch to Ko-Fi, it has the lowest fees for us

Ko-Fi (Donate)

Bunq (Donate)

Open Collective backers and sponsors

Patreon

Join the team

founded 2 years ago
MODERATORS
 

Today, like the past few days, we have had some downtime. Apparently some script kids are enjoying themselves by targeting our server (and others). Sorry for the inconvenience.

Most of these 'attacks' are targeted at the database, but some are more ddos-like and can be mitigated by using a CDN. Some other Lemmy servers are using Cloudflare, so we know that works. Therefore we have chosen Cloudflare as CDN / DDOS protection platform for now. We will look into other options, but we needed something to be implemented asap.

For the other attacks, we are using them to investigate and implement measures like rate limiting etc.

(page 2) 50 comments
sorted by: hot top controversial new old
[–] mordred@lemmy.world 20 points 1 year ago (1 children)

Thank you for your efforts, work and results. Those "attackers" only deserve disgust.

load more comments (1 replies)
[–] Anaralah_Belore223@lemmy.world 20 points 1 year ago* (last edited 1 year ago) (1 children)

Benefit of using Cloudflare CDN:

Commenting and editing took about 0.5 seconds!

Also, ping is now from 200-300 miliseconds to just between 50 and 60 (depending on your ISP):

64 bytes from 172.67.218.212: icmp_seq=1 ttl=64 time=56.2 ms
64 bytes from 172.67.218.212: icmp_seq=2 ttl=64 time=60.2 ms
64 bytes from 172.67.218.212: icmp_seq=3 ttl=64 time=55.8 ms
64 bytes from 172.67.218.212: icmp_seq=4 ttl=64 time=58.9 ms
64 bytes from 172.67.218.212: icmp_seq=5 ttl=64 time=60.6 ms
64 bytes from 172.67.218.212: icmp_seq=6 ttl=64 time=60.5 ms
64 bytes from 172.67.218.212: icmp_seq=7 ttl=64 time=60.1 ms
64 bytes from 172.67.218.212: icmp_seq=8 ttl=64 time=55.0 ms
64 bytes from 172.67.218.212: icmp_seq=9 ttl=64 time=60.0 ms
64 bytes from 172.67.218.212: icmp_seq=10 ttl=64 time=61.4 ms
64 bytes from 172.67.218.212: icmp_seq=11 ttl=64 time=59.3 ms
64 bytes from 172.67.218.212: icmp_seq=12 ttl=64 time=58.5 ms
64 bytes from 172.67.218.212: icmp_seq=13 ttl=64 time=56.0 ms
64 bytes from 172.67.218.212: icmp_seq=14 ttl=64 time=60.6 ms
64 bytes from 172.67.218.212: icmp_seq=15 ttl=64 time=58.7 ms
[–] BitOneZero@lemmy.world 17 points 1 year ago* (last edited 1 year ago)

The bugs in Lemmy are such that you don't even need to touch a server for it to be vulnerable. Cloudflare does not defend against such mistakes. Other servers can trigger deep PostgreSQL logic problems within Lemmy. Growing pains, a lot of the federation code was never tested, and today's crash is due to a logic issue with lemmy_server mistakenly updating 1700 servers it knows of through federation for a delete instead of the 1 local server.

[–] itadakimasu@lemmy.world 19 points 1 year ago

I'm learning a lot by following lemmy.worlds actions. Appreciate the transparency!

[–] fence_prude@lemmy.world 18 points 1 year ago (4 children)

Where can we donate toward server costs?

load more comments (4 replies)
[–] Tag365@lemmy.world 18 points 1 year ago (1 children)
load more comments (1 replies)
[–] drmoose@lemmy.world 17 points 1 year ago (3 children)

I hope lemmy.world can avoid using Cloudflare which goes against the spirit of Fediverse as it's just an objectively evil company.

[–] ruud@lemmy.world 22 points 1 year ago

Agreed. This is an emergency fix. Will look for final solution later.

[–] phar@lemmy.ml 16 points 1 year ago (11 children)

Can you give some insight to this?

load more comments (11 replies)
load more comments (1 replies)
[–] Resol@lemmy.world 17 points 1 year ago (3 children)

What doesn't kill you makes you stronger

load more comments (3 replies)
[–] Anaralah_Belore223@lemmy.world 16 points 1 year ago* (last edited 1 year ago) (1 children)

The more you attack a Lemmy instance, the more stronger it gets.

It's like tempering iron or steel.

load more comments (1 replies)
[–] free@lemmy.world 16 points 1 year ago

ty for all ur hard work β™₯️

[–] AphoticDev@lemmy.dbzer0.com 16 points 1 year ago (1 children)

It's not ideal, but there's not a whole lot of options out there for DDoS mitigation.

load more comments (1 replies)
[–] boulderly@lemmyadmin.site 14 points 1 year ago (2 children)

I put this site behind cloudflare in response to this post. Other than having to change SSL/TLS encryption mode to Full, it seemed easy. I turned on bot fight mode and I'm using the managed WAF ruleset that comes with the free tier. Any configuration recommendations anywhere in the panel?

load more comments (2 replies)
[–] stevedidWHAT@lemmy.world 14 points 1 year ago

Welcome to the internet, glad there’s a working solution

[–] zazaserty@discuss.tchncs.de 14 points 1 year ago (3 children)

Damn these script kiddies.. I don't like Cloudflare at all but it does its job well. It may just be my paranoia, but putting a single entity in control of so many websites seems dangerous. I think we have all learned about the intentions of big corporations. But hey, it's better than being taken down tbf.

load more comments (3 replies)
[–] michael@lemmy.perthchat.org 13 points 1 year ago (5 children)

Hmmm, we're getting a fuckload of web requests on our Lemmy too... I think I'll enable CloudFlare too! :)

load more comments (5 replies)
[–] db0@lemmy.dbzer0.com 13 points 1 year ago (2 children)

I wonder now with the semi-adversarial/semi-cooperative nature between lemmy instances, if wer'e not going to see more DDOS and other types of raids happening because a different instance has an ax to grind against yours. Say between you defederated them, or they consider your instance too big etc.

load more comments (2 replies)
[–] chipamogli@lemmy.world 13 points 1 year ago (1 children)

Maybe is that Reddit dude, jealous of Lemmy's increasing popularity.

[–] TerryMathews@lemmy.world 12 points 1 year ago (1 children)

It seems like you made this comment in jest, but I wouldn't say it's outside the realm of possibility. We can't fly off the handle and lob accusations absent any sort of proof, but it would hardly be the first example of a corporation targeting an up-and-coming disruptive service run by amateurs.

load more comments (1 replies)
load more comments
view more: β€Ή prev next β€Ί