For anyone that may stumbled upon this, we worked around this issue by setting a more fine-grained dirs on the SYSTEMD_SYSEXT_HIERARCHIES env var. So it would just match those instead of overlaying the full /usr and getting our mounts under it lost :D
root@localhost:~# SYSTEMD_SYSEXT_HIERARCHIES=/usr/local/bin:/usr/local/sbin:/usr/local/include:/usr/local/lib:/usr/local/share:/usr/local/src:/usr/bin:/usr/share:/usr/lib:/usr/include:/usr/src:/usr/sbin systemd-sysext refresh Using extensions 'work'. Merged extensions into '/usr/local/bin'. Merged extensions into '/usr/lib'. That made our sysext work AND respected our mount under /usr/local with our own writable mounts and such :D On Mon, Jun 10, 2024 at 6:43 PM Itxaka Serrano Garcia < itxaka.gar...@spectrocloud.com> wrote: > Ah, I guess this is because overlayfs does not respect any underlying > mounts in the /usr so they get lost > > So this has nothing to do with sysext. > > In any case, is this a valid use case that we can see happening? Or now > that we are here already, if anyone got a suggestion it would be welcome, > otherwise feel free to ignore this email :) > > El lun, 10 jun 2024, 17:19, Itxaka Serrano Garcia < > itxaka.gar...@spectrocloud.com> escribió: > >> Hey again, >> >> I'm back with more annoying questions about sysext :D >> >> According to the docs, sysext only "extends" the existing usr/opt/etc >> with the sysext contents but we are seeing a different thing here: >> >> root@cos-recovery:~# stat /usr/local/file2 >> File: /usr/local/file2 >> Size: 0 Blocks: 0 IO Block: 4096 regular empty file >> Device: 0,47 Inode: 171 Links: 1 >> Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) >> Access: 2024-06-10 14:56:50.725904625 +0000 >> Modify: 2024-06-10 14:56:50.725904625 +0000 >> Change: 2024-06-10 14:56:50.725904625 +0000 >> Birth: 2024-06-10 14:56:50.725904625 +0000 >> root@cos-recovery:~# systemd-sysext refresh >> Using extensions 'work'. >> Merged extensions into '/usr'. >> root@cos-recovery:~# stat /usr/local/file2 >> stat: cannot statx '/usr/local/file2': No such file or directory >> root@cos-recovery:~# systemd-sysext unmerge >> Unmerged '/usr'. >> root@cos-recovery:~# stat /usr/local/file2 >> File: /usr/local/file2 >> Size: 0 Blocks: 0 IO Block: 4096 regular empty file >> Device: 0,47 Inode: 171 Links: 1 >> Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) >> Access: 2024-06-10 14:56:50.725904625 +0000 >> Modify: 2024-06-10 14:56:50.725904625 +0000 >> Change: 2024-06-10 14:56:50.725904625 +0000 >> Birth: 2024-06-10 14:56:50.725904625 +0000 >> >> Weirdly enough, any files under /usr/ that we create are NOT overridden >> when we load any sysext, it only seems to happen in the dirs inside /usr >> >> It's true that we are using USI here and on a "normal" non-USI/UKI system >> this behaviour doesn't seem to happen. >> >> The main difference I can see is that the root fs in the failure case is >> a tmpfs while in the working case it's an ext4 fs, but for example our >> /usr/local is mounted to a ext4 disk: >> >> /dev/mapper/vda3 on /usr/local type ext4 (rw,relatime) >> >> >> If I do create an overlay on /usr then it seems to work as expected but >> its too late for us, we lose anything mounted there and hidden dirs. >> >> Is this a bug or its expected behaviour? Is there anything we could be >> missing that triggers this behaviour? Any pointers on why this would react >> like this? >> >> >> Cheers, >> Itxaka >> >