Our instance is currently down and I can't get remote access to the servers. It appears that there might have been a hardware failure of the main firewall, which is the one thing I can't work around remotely.

I am still trying a few things, but I am not very optimistic that I can get access.

The really unfortunate part is that just now I am on one of my rare work deployments abroad, so I also can't access it physically during the next few weeks and my usual back up that could restart it is not available either.

As something like that never happened in 3 years operating the servers, I thought I can risk it, but murphy's law seems inescapable 😓

I will try to keep you posted here on any updates, but probably there will not be much I can do for a while. Really bad timing 😥

Edit: we might use this "opportunity" to migrate the instance to Piefed, which has been an idea for quite some time now. I will keep you posted on that.

you are viewing a single comment's thread
view the rest of the comments

[–] tenchiken@lemmy.dbzer0.com 6 points 2 days ago (1 children)

Not sure about your area, but a wireguard accessible OOB connection is a great piece of kit to keep handy. I use a cheap 768kbps SIM in an Ethernet connected switch into my personal systems. It's saved my skin numerous times.

I'm sure this is obvious, especially in hindsight, but just mentioning because the existence of IoT LTE data plans for a minimal fee ( $100/year for me in Midwest US) was NOT obvious to me until 2 years ago.

[–] Kris@feddit.org 6 points 2 days ago* (last edited 2 days ago) (2 children)

Yeah I had plans to set up something like that, but always other priorities and in this specific case I could maybe access other internal servers but i would need KVM access to reboot the firewall or some other way to cut physical power. And exfiltrating hundreds of GBs of lemmy database wouldn't work over such a small pipe either.

[–] MelodiousFunk@startrek.website 5 points 2 days ago

This is something that even larger corps struggle with. My old company would buy some other company, lay off a bunch of people, others would quit, and then it was shocked Pikachu faces all around when That One Thing stopped working in an office they turned into a ghost town and, well, no, nobody was going to be there until Monday morning to power cycle etc. True lights out/OOB reachability is WORK. And there's always going to be a SPOF somewhere that requires hands on-site.

Do what you can, when you can. And thank you for all that y'all have done so far.

[–] curbstickle@lemmy.dbzer0.com 2 points 2 days ago

For that, I'd highly recommend a power conditioner with outlet controls (and preferably sequencing).

Most are pretty expensive, but you can get something like a surgex squid or a gude expert power control for a few hundred here and there, which will give you great data on power usage as well. You can also grab an older model with serial and/or relay controls, and use a pi or whatever is handy to trigger it. Actually have a setup like that in a few racks for clients (mostly orange pis and a couple of Asus tinker boards).

Hope your trip goes well!