Selfhosted

I'll be honest man, just don't do it. I tried, I really did, to make this exact scenario to work. You can get it to work - but it will be extremely brittle. You're essentially hacking around LXC to do things it wasn't built to do, and most of it is disabling security that's there for a reason. At the end of the day you are essentially running docker directly on the host anyway, the passthrough lxc becomes less and less "there" vs passing stuff through. Then, every update to proxmox became anxiety riddled because every update would change or break something on my setup.

If you want to continue, more power to you, but I hope you heed my warnings. This is a path you will spend a lot of time on and experience a lot of frustration. Spin up a tiny debian VM and run the containers there, the overhead of the VM has been negligible, and any speedup I might have had has been made up 10x by cutting the amount of time I've had to hack proxmox to make it work.

The issue is that the docker container will still be running as the LXC's root user even if you specify another user to run as in the docker compose file or run command, and if root doesn't have access to the dir the container will always fail.

The solution to this is to remap the unprivileged LXC's root user to a user on the Proxmox host that has access to the dir using the LXC's config file, mount the container's filesystem using pct mount, and then chown everything in the container owned by the default root mapped user (100000).

These are the commands I use for this:

find /var/lib/lxc/xxx/rootfs -user 100000 -type f -exec chown username {} +;
find /var/lib/lxc/xxx/rootfs -user 100000 -type d -exec chown username {} +;
find /var/lib/lxc/xxx/rootfs -user 100000 -type l -exec chown -h username {} +;
find /var/lib/lxc/xxx/rootfs -group 100000 -type f -exec chown :username {} +;
find /var/lib/lxc/xxx/rootfs -group 100000 -type d -exec chown :username {} +;
find /var/lib/lxc/xxx/rootfs -group 100000 -type l -exec chown -h :username {} +

(Replace xxx with the LXC number and username with the host user/UID)

If group permissions are involved you'll also have to map those groups in the LXC config, create them in the LXC with the corresponding GIDs, add them as supplementary groups to the root user in the LXC, and then add them to the docker compose yaml using group_add.

It's super confusing and annoying but this is the workflow I'm using now to avoid having to have any resources tied up in VMs unnecessarily.

I'm not a docker expert- i tend to just run everything in an LXC. But, doesn't docker typically run as root? It might be that you gave your lxc user UID proper permissions, but not the lxc root UID.

Alternatively, you are aware that LXC UID 1000 != Host UID 1000, yes?

FWIW, permissions in proxmox/LXC are really clear and predictable... once you understand the way the map in the config files.

So I don’t use docker. But. Could you set up an lxc container running samba and then bind mount the shares to that container. Then mount those shares inside docker running on the other lxc container?

In incus, I had the same setup of an LCX container with a Docker container inside of it. I passed 1000/1000 to the LXC container but the LXC container’s default root user has a an ID set of 0/0. So I had to pass 0/0 to the Docker container, not 1000/1000 to get the read/write permissions working.

That may fix your issue as it’s basically the same tech, just different automated things implementing the LXC container!

It is not 1000 it 10000.

You can try with setting g+w on pve.

I run dozens of proxmox LXCs, most with docker, but can confirm that while proxmox is fantastic to host NFS shares, it is near impossible to mount external ones to LXCs without doing some weirdness on the host. best practice would be to turn the NFS into samba or something and configure the docker vms and LXCs to use that (45drives has awesome repos on GitHub that work really well; that's how I worked around that issue) the only downside is you need to usually offload everything as a backup first before the switch.

Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I've seen in this thread:

[Thread #882 for this sub, first seen 24th Jul 2024, 09:45] [FAQ] [Full list] [Contact] [Source code]