Personally, I do believe that rootless Docker/Podman have a strong enough security boundary for personal/individual self-hosting where you have decent trust in the software you're running. Linux privilege escalation and container escape exploits fetch decent amounts of money on the exploit market, and nobody's gonna waste them on some people running software ending in *arr when Zerodium will pay five figures for a local privilege escalation or container escape. If you're running a business or you might be targeted for whatever reason (journalist or whatever) then that doesn't apply.

If you want more security, there are container runtimes that do cooler security stuff under the hood, like Firecracker/Kata Containers implementing a managed VM, or Google's gVisor which very strongly intercepts kernel syscalls and essentially reimplements Linux in userspace. Those are used by AWS and Google Cloud respectively. You can integrate those into Docker, though not all networking/etc options are supported.