Nice to haves:

Boot from an NVMe without a hat/cape/shield

Both wifi and ethernet

I really want to run ceph because it fits a number of criteria I have: gradually adding storage, mismatched disks, fault tolerance, erasure encoding, encryption, support out-of-the-box from other software (like Incus).

But then I look at the hardware suggestions, and they seem like an up-front investment and ongoing cost to keep at least three machines evenly matched on RAM and physical storage. I also want more of a single-box NAS.

Would it be idiotic to put a ceph setup all on one machine? I could run three mons on it with separate physical device backing each so I don't lose everything from a disk failure with those. I'm not too concerned about speed or network partitioning, this would be lukewarm storage for me.

I'm used to using Linux from the terminal. I have a new machine which I plan to use mostly headless but would occasionally like to run a desktop environment and play games with GPU acceleration. I know I don't have to launch the desktop environment on startup, but I was wondering if it's possible to have that entire portion containerized, like an instance in LXD.

I am trying Bazzite right now, I really like the idea of layering on top an immutable base. That's close to what I want. If I understand correctly, I could have a different layer for the headless part to keep them totally separate, but I'd have to do restarts to switch from one to the other.

I also think NixOS could also be what I want, just with a steeper learning curve.

I'm wondering if anyone has already set something like this up? It would be helpful to read about what software people have for this and their experiences using that.

I have seen several cards that will do up to 4 NVMe from a single x16 slot (with MB and CPU that support bifurcation), but I have only found cards that are 1 PCIe slot to 1 M.2 A+E.

I think one way to do this would be to have a regular bifurcation x16 to 4 x4s and then use the 1x cards. But are there other options?

The reason I am asking is because I am procrastinating on other things I am supposed to be doing. I have no actual need for this and putting 4 wifi cards so close probably creates horrible interference anyway.

What storage software could I run to have an archive of my personal files (a couple TB of photos) that doesn't require I keep a full local copy of all the data? I like the idea of a simple and focused tool like Syncthing, but they seem to be angling towards replication.

Is the simple choice to run some S3-like backend and use CLI or other client to append and browse files? I'd love something with fault tolerance that someone can gradually add disks to. If ceph were either less complicated or used less resources I'd want to do that.

I'm planning to set up LUKS on an SSD. Many guides are suggesting using a simple key to set things up and then revoke it when everything is in place.

Given the wear leveling behavior on SSDs I am assuming a simple key might be able to unlock even beyond the revocation if a determined attacker has the disk. I don't want someone to be able to put the disk in factory access mode and be able to brute force attempt their way to browser cookies and email accounts.

I'm going to ignore the suggestion about using a weak key to set up, but am I being overly paranoid? Am I being not paranoid enough and I should also not rely on revocation for a spinning rust disk?