Our instance is currently down and I can't get remote access to the servers. It appears that there might have been a hardware failure of the main firewall, which is the one thing I can't work around remotely.

I am still trying a few things, but I am not very optimistic that I can get access.

The really unfortunate part is that just now I am on one of my rare work deployments abroad, so I also can't access it physically during the next few weeks and my usual back up that could restart it is not available either.

As something like that never happened in 3 years operating the servers, I thought I can risk it, but murphy's law seems inescapable 😓

I will try to keep you posted here on any updates, but probably there will not be much I can do for a while. Really bad timing 😥

Edit: we might use this "opportunity" to migrate the instance to Piefed, which has been an idea for quite some time now. I will keep you posted on that.

you are viewing a single comment's thread
view the rest of the comments

[–] Five@lemmy.cafe 8 points 1 day ago (1 children)

You're very welcome! Kris@feddit.org / @poVoq@slrpnk.net is an incredible admin, and up until now has declined donations, funding the entire instance several years now from his own pocket.

This current downtime is the exception that proves the rule. I hope he's not stressing out too much. We'll all be ready to jump right back into it when he gets back.

[–] Kris@feddit.org 6 points 1 day ago* (last edited 22 hours ago) (4 children)

Already thinking about how to make the best of it 🤷 Maybe we can use this opportunity to try and migrate to Piefed? I had this in the back of my mind for some time already and Rimu seems optimistic that it is possible.

[–] Draconic_NEO@lemmy.world 1 points 19 hours ago* (last edited 19 hours ago) (1 children)

Wouldn't that mean the instance would completely restart? I mean I don't know much about piefed but logically it seems like the DB structure would be incompatible. Thus "Migrating" to piefed means a complete reboot, all posts, all users, all content just bye bye.

[–] Kris@feddit.org 4 points 19 hours ago (1 children)

Not necessarily, no. We aim to preserve users, communities and posts/comments. Image uploads might get lost though.

Such an in-place migration will need extensive database operations and likely some support by the Piefed developer (to add support for bcrypt hashed passwords), but we are hopeful to make it happen and maybe this will result in a database migration script other Lemmy instances could also use.

If this turns out to be infeasible, we will stay with Lemmy rather than reset everything.

[–] Draconic_NEO@lemmy.world 3 points 19 hours ago

Oh awesome, I mean I've heard of other platforms like sublinks offering database migration but I didn't think it would be feasible with piefed since unlike sublinks it's very different than lemmy.

[–] AccountMaker@lemmy.dbzer0.com 2 points 1 day ago

That would be so nice if we could move to Piefed.

[–] tfm@europe.pub 2 points 1 day ago (1 children)

What do you like more about Piefed compared to Lemmy?

[–] Kris@feddit.org 3 points 22 hours ago

Lemmy has a lot of individual parts that don't interact very well with each other, especially the image host part. Futhermore the main UI is quite a mess and we were thinking of switching to an alternative already, but this would further increase the "too many moving parts" issue. Piefed on the otherhand has an integrated and very lightweight UI, which also has some nice additional filtering and moderation features Lemmy currently lacks.

And I personally feel more at home with the Python codebase, as it allows better troubleshooting and more standartized (Flask) tooling. The Rust codebase of Lemmy has a lot of obscure custom stuff and the error messages are extremely obstruse from a sysadmin perspective.

And looking at the performance metrics of Lemmy, the main limiting factor seems to be the Postgres database anyways, so the theoretically slower Python codebase of Piefed should not have much impact.

[–] PirateFrog@lemmy.dbzer0.com 6 points 1 day ago

That'd actually be pretty rad if it's doable, especially now that the API for piefed was released and mobile apps are already beginning to support it.