Sysadmin

When I was working in that old job we had one particular fiasco that legit stresses me to remember. We have this account, no one knows what it does, but the password has never been rotated, it's not vaulted, etc. There's 5 apps that share the DB. I contact all the app owners, no response.

I wait a week and escalate the their bosses. No response. I send emails every single day to everyone including all the dev teams. Not one "lemme check on that" or anything. Our policy was to wait 90 freaking days for a non single user account. I'm getting yelled at to get this ticket closed when the day comes.

I go in, lock the account, change the password, and kill all DB sessions. Within 15 minutes I'm paged for a priority one incident because a trading app is down, causing the whole floor to be out and they're losing millions every minute.

I tell them what I did and forwarded emails to everyone. The executive director is screaming at me, telling me I'm gonna be fired soon and I better fix it right now.

Sure, I can unlock the account and even force the password back to the old version. What's that? No one knows what the old password was? Nothing I can do. Fortunately my executive director was awesome and stepped in to take the call. Overall they were down for an hour and a half. I looked at the incident later and they claimed $100 million in losses. The app owners wanted me fired. They got the uno reverse though and lost their jobs over it.

Fuck that job lol.