I have been lurking on this community for a while now and have really enjoyed the informational and instructional posts but a topic I don't see come up very often is scaling and hoarding. Currently, I have a 20TB server which I am rapidly filling and most posts talking about expanding recommend simply buying larger drives and slotting them in to a single machine. This definitely is the easiest way to expand, but seems like it would get you to about 100TB before you cant reasonably do that anymore. So how do you set up 100TB+ networks with multiple servers?
My main concern is that currently all my services are dockerized on a single machine running Ubuntu, which works extremely well. It is space efficient with hardlinking and I can still seed back everything. From different posts I've read, it seems like as people scale they either give up on hardlinks and then eat up a lot of their storage with copying files or they eventually delete their seeds and just keep the content. Does the Arr suite and Qbit allow dynamically selecting servers based on available space? Or are there other ways to solve these issues with additional tools? How do you guys set up large systems and what recommendations would you make? Any advice is appreciated from hardware to software!
Also, huge shout out to Saik0 from this thread: https://lemmy.dbzer0.com/post/24219297
I learned a ton from his post, but it seemed like the tip of the iceberg!
Thanks for this fantastic write up, and your other response! I definitely learned a lot just looking up all the terms. Just a couple of questions if you have time.
For your 16x bay boxes, are you running like old Optiplex or PowerEdge hardware or something else? I ask because these seem to be available in large supplies from surplus sites and Im curious if one is strictly better or easier to work with. Also, I've read that you should loosely match TB of storage to GB of RAM. The PowerEdge hardware has tons of DIMMs but old PCs don't so curious if you've had to deal with that since zfs seems so well optimized.
For the split categories, ie. 2x for TV you mentioned, do you need to run two instances of Sonarr? Or do you just manually change the path when a single box gets full? Otherwise, how do you keep the two instances in sync?
Lastly, I've done quite a bit of reading on OMV and Proxmox but I don't actually use them yet. Do you recommend Proxmox with an OMV vm or just OMV baremetal?
Thanks for taking the time!