[Bug 2056442] Re: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported
This bug was fixed in the package crun - 0.17+dfsg-1.1ubuntu0.1 --- crun (0.17+dfsg-1.1ubuntu0.1) jammy; urgency=medium * lp2056442-ignore-enotsup-when-chmod-symlink.patch: Ignore ENOTSUP when chmod a symlink (LP: #2056442) -- Julian Andres Klode Thu, 07 Mar 2024 14:17:06 +0100 ** Changed in: crun (Ubuntu Jammy) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056442 Title: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2056442/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056442] Re: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported
There was no clear indication which running kernel was used during the verifications, so I did it: Using the Dockerfile from the test plan: FROM ubuntu:noble RUN apt install -U systemd -y CMD ["/lib/systemd/systemd"] And building with "podman build -t systemd ." a) 5.15.0-105 kernel (jammy GA kernel) crun 0.17+dfsg-1.1 from jammy fails (odd, I thought it was unaffected): $ podman run --systemd always systemd Error: OCI runtime error: chmod `run/shm`: Operation not supported crun 0.17+dfsg-1.1ubuntu0.1 from jammy-proposed works: $ podman run --systemd always systemd (nothing here) In a separate terminal we can see it's running: $ podman ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES b2decb3d2820 localhost/systemd:latest /lib/systemd/syst... 34 seconds ago Up 34 seconds ago flamboyant_hertz b) Jammy HWE kernel $ uname -r 6.5.0-34-generic crun 0.17+dfsg-1.1 from jammy fails: $ podman run --systemd always systemd Error: OCI runtime error: chmod `run/shm`: Operation not supported crun 0.17+dfsg-1.1ubuntu0.1 from jammy-proposed works: $ podman run --systemd always systemd (nothing here) In another terminal: $ podman ps CONTAINER ID IMAGE COMMAND CREATED STATUSPORTS NAMES 867d113be7a3 localhost/systemd:latest /lib/systemd/syst... 2 seconds ago Up 2 seconds ago confident_mirzakhani -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056442 Title: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2056442/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056442] Re: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported
Marking mantic as "won't fix" according to comment #33 above. ** Changed in: crun (Ubuntu Mantic) Status: Confirmed => Won't Fix -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056442 Title: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2056442/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056442] Re: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported
** Merge proposal linked: https://code.launchpad.net/~fun2program8/ubuntu/+source/crun/+git/crun/+merge/464233 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056442 Title: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2056442/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056442] Re: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: crun (Ubuntu Mantic) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056442 Title: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2056442/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056442] Re: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported
jammy to mantic is not a particular interesting upgrade path, it's opt- in only, so if nobody is interested in a mantic fix we shouldn't block the jammy LTS fix for it. jammy users will be upgrading to noble and not be affected by a mantic regression. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056442 Title: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2056442/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056442] Re: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported
Yeah, but the default is crun. I could have modified the config file of course. But I have not: ...snip # podman run --systemd always -d cruntest 231947c35e1d58cfb3d1ca37e52b6324288af515e3ce6c25a0873a721a213f1b # podman run --systemd always --runtime crun -d cruntest e655a24b5c11d7bd572942533db4b95fe53cbd0338c35b76c5f0129e78f806e9 # podman inspect 231947c35e1d | jq .[0].OCIRuntime "crun" # podman inspect e655a24b5c11 | jq .[0].OCIRuntime "crun" snap... -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056442 Title: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2056442/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056442] Re: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported
> # dpkg -l crun Markus, thanks for the verification, but crun can be installed together with runc, and it's not clear which one will end up being used. That's why it's best to also make sure runc is *not* installed. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056442 Title: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2056442/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056442] Re: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported
This is blocked on a mantic upload fixing the same issue. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056442 Title: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2056442/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056442] Re: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported
I also tried the package in some CI runs on github. Looks Good. I changed verification-needed-jammy to verification-done-jammy, however, I would appreciate if another one affected can confirm that it works for him too. Don't know about tag verification-needed, so I did not touch it. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056442 Title: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2056442/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056442] Re: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported
** Tags removed: verification-needed-jammy ** Tags added: verification-done-jammy -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056442 Title: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2056442/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056442] Re: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported
I installed the package ...snip # dpkg -l crun Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name VersionArchitecture Description +++-==-==--== ii crun 0.17+dfsg-1.1ubuntu0.1 amd64lightweight OCI runtime for running containers snap... and run the the test as in #8: ...snip # cat - >Containerfile
[Bug 2056442] Re: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported
I see the same code in mantic's crun package, and the problem can also be reproduced there: $ uname -a Linux nsnx2 6.5.0-25-generic #25-Ubuntu SMP PREEMPT_DYNAMIC Wed Feb 7 14:58:39 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux $ grep VERSION_CODENAME /etc/os-release VERSION_CODENAME=mantic $ podman run --systemd always systemd Error: OCI runtime error: crun: chmod `run/shm`: Operation not supported I'm going to add a mantic task to this bug, and either expect a mantic upload to fix it, or clarification as to why mantic will not be fixed. ** Also affects: crun (Ubuntu Mantic) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056442 Title: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2056442/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056442] Re: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported
I'm also updating the test case to make sure crun is installed, because runc can also satisfy podman's dependency on an OCI runtime. ** Description changed: [Impact] podman containers using the default crun backend do not work anymore with the 6.5 HWE kernel in 22.04 [Test plan] 1. Boot jammy with 6.5 HWE kernel (e.g. install linux-virtual-hwe-22.04 in a VM) - 2. `podman build -t systemd .` with the Dockerfile: + 2. Make sure you have crun installed, and not runc, because podman can also use runc and this bug is about crun. + 3. `podman build -t systemd .` with the Dockerfile: FROM ubuntu:noble RUN apt install -U systemd -y CMD ["/lib/systemd/systemd"] - 3. Run it `podman run --systemd always systemd` you should not get + 4. Run it `podman run --systemd always systemd` you should not get Error: OCI runtime error: chmod `run/shm`: Operation not supported Optimally submitter can do end-to-end-verification on their side. [Where problems could occur] The patch ignores ENOTSUP for fchmodat() in one function, so at most we could silently hide some other issues in fchmodat() inside that function, e.g. AppArmor denials. But generally that is what you would want as a behavior for ENOTSUP... [Original bug report] The problem is very well described in https://github.com/actions/runner- images/issues/9425. ## COPY FROM LINK I think there might be a regression in this release of the ubuntu-22.04 image which breaks podman.[1] The image updated the kernel from 6.2.y to 6.5.y, but podman/crun don't seem to be updated. Our build fails with this error link to run:[2] STEP 1/1: FROM ghcr.io/gardenlinux/builder:3ab2bb52bc46bb200c761369c087e9413d1ce0ac Trying to pull ghcr.io/gardenlinux/builder:3ab2bb52bc46bb200c761369c087e9413d1ce0ac... Getting image source signatures Copying blob sha256:041b542221cfde2f9fa4ac13f8b5804e25b23ab48ba47db2822c382a134256e1 Copying blob sha256:041b542221cfde2f9fa4ac13f8b5804e25b23ab48ba47db2822c382a134256e1 Copying config sha256:1eba10d0345cc6df78b7c3a6ced45da9db675d05eb20d5d286996e4f7ffb24d5 Writing manifest to image destination Storing signatures COMMIT localhost/builder --> 1eba10d0345 Successfully tagged localhost/builder:latest Successfully tagged ghcr.io/gardenlinux/builder:3ab2bb52bc46bb200c761369c087e9413d1ce0ac 1eba10d0345cc6df78b7c3a6ced45da9db675d05eb20d5d286996e4f7ffb24d5 Error: OCI runtime error: chmod `run/shm`: Operation not supported Error: Process completed with exit code 126. This is with this image version: Current runner version: '2.313.0' Operating System Ubuntu 22.04.4 LTS Runner Image Image: ubuntu-22.04 Version: 20240225.1.0 Included Software: https://github.com/actions/runner-images/blob/ubuntu22/20240225.1/images/ubuntu/Ubuntu2204-Readme.md Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu22%2F20240225.1 Trying to reproduce, it seems like I am only able to get this image version 20240218.1.0 where the issue does not appear. Tried to reproduce in this repo[3], but I'm not able to get this with image version 20240225.1.0. Is this a known issue and version 20240225.1.0 is not in use anymore? This blog post seems to suggest that the crun version is too old.[4] [1]https://github.com/actions/runner-images/releases/tag/ubuntu22%2F20240225.1 [2]https://github.com/gardenlinux/gardenlinux/actions/runs/8061893233/job/22020522535 [3]https://github.com/fwilhe/turbo-lamp/blob/main/.github/workflows/podman.yml [4]https://noobient.com/2023/11/15/fixing-ubuntu-containers-failing-to-start-with-systemd/ ** Description changed: [Impact] podman containers using the default crun backend do not work anymore with the 6.5 HWE kernel in 22.04 [Test plan] 1. Boot jammy with 6.5 HWE kernel (e.g. install linux-virtual-hwe-22.04 in a VM) 2. Make sure you have crun installed, and not runc, because podman can also use runc and this bug is about crun. 3. `podman build -t systemd .` with the Dockerfile: FROM ubuntu:noble RUN apt install -U systemd -y CMD ["/lib/systemd/systemd"] 4. Run it `podman run --systemd always systemd` you should not get Error: OCI runtime error: chmod `run/shm`: Operation not supported + 5. Repeat the above steps on the jammy GA kernel, to make sure we did not regress that use case which is unaffected by this bug. Optimally submitter can do end-to-end-verification on their side. [Where problems could occur] The patch ignores ENOTSUP for fchmodat() in one function, so at most we could silently hide some other issues in fchmodat() inside that function, e.g. AppArmor denials. But generally that is what you would want as a behavior for ENOTSUP... [Original bug report] The problem is very well described in https://github.com/actions/runner- images/issues/9425. ## COPY FROM LINK I think there might be
[Bug 2056442] Re: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported
I updated the test case one more time to request a run with the jammy GA kernel, to make sure we haven't regressed that (unaffected) use case. I'm accepting the jammy upload to get things going, but the mantic question from comment #24 will need to be addressed before the jammy fix can hit updates. If mantic will get an update as well (preferable), then please amend the test case to also include mantic. Note that the jammy upload is deleting debian/.gitlab-ci.yml. This change is not needed, nor mentioned in d/changelog. Since it doesn't affect the build, I'm not blocking on it, but it would be cool to watch out for such changes in the future :) ** Changed in: crun (Ubuntu Jammy) Status: In Progress => Fix Committed ** Tags added: verification-needed verification-needed-jammy -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056442 Title: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2056442/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056442] Re: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported
Sorry, I just did not fix the first line when editing the test plan ** Description changed: [Impact] podman containers using the default crun backend do not work anymore with the 6.5 HWE kernel in 22.04 [Test plan] - Needs to be finalized, but roughly: - 1. Boot jammy with 6.5 HWE kernel (e.g. install linux-virtual-hwe-22.04 in a VM) 2. `podman build -t systemd .` with the Dockerfile: FROM ubuntu:noble RUN apt install -U systemd -y CMD ["/lib/systemd/systemd"] 3. Run it `podman run --systemd always systemd` you should not get Error: OCI runtime error: chmod `run/shm`: Operation not supported Optimally submitter can do end-to-end-verification on their side. [Where problems could occur] The patch ignores ENOTSUP for fchmodat() in one function, so at most we could silently hide some other issues in fchmodat() inside that function, e.g. AppArmor denials. But generally that is what you would want as a behavior for ENOTSUP... [Original bug report] The problem is very well described in https://github.com/actions/runner- images/issues/9425. ## COPY FROM LINK I think there might be a regression in this release of the ubuntu-22.04 image which breaks podman.[1] The image updated the kernel from 6.2.y to 6.5.y, but podman/crun don't seem to be updated. Our build fails with this error link to run:[2] STEP 1/1: FROM ghcr.io/gardenlinux/builder:3ab2bb52bc46bb200c761369c087e9413d1ce0ac Trying to pull ghcr.io/gardenlinux/builder:3ab2bb52bc46bb200c761369c087e9413d1ce0ac... Getting image source signatures Copying blob sha256:041b542221cfde2f9fa4ac13f8b5804e25b23ab48ba47db2822c382a134256e1 Copying blob sha256:041b542221cfde2f9fa4ac13f8b5804e25b23ab48ba47db2822c382a134256e1 Copying config sha256:1eba10d0345cc6df78b7c3a6ced45da9db675d05eb20d5d286996e4f7ffb24d5 Writing manifest to image destination Storing signatures COMMIT localhost/builder --> 1eba10d0345 Successfully tagged localhost/builder:latest Successfully tagged ghcr.io/gardenlinux/builder:3ab2bb52bc46bb200c761369c087e9413d1ce0ac 1eba10d0345cc6df78b7c3a6ced45da9db675d05eb20d5d286996e4f7ffb24d5 Error: OCI runtime error: chmod `run/shm`: Operation not supported Error: Process completed with exit code 126. This is with this image version: Current runner version: '2.313.0' Operating System Ubuntu 22.04.4 LTS Runner Image Image: ubuntu-22.04 Version: 20240225.1.0 Included Software: https://github.com/actions/runner-images/blob/ubuntu22/20240225.1/images/ubuntu/Ubuntu2204-Readme.md Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu22%2F20240225.1 Trying to reproduce, it seems like I am only able to get this image version 20240218.1.0 where the issue does not appear. Tried to reproduce in this repo[3], but I'm not able to get this with image version 20240225.1.0. Is this a known issue and version 20240225.1.0 is not in use anymore? This blog post seems to suggest that the crun version is too old.[4] [1]https://github.com/actions/runner-images/releases/tag/ubuntu22%2F20240225.1 [2]https://github.com/gardenlinux/gardenlinux/actions/runs/8061893233/job/22020522535 [3]https://github.com/fwilhe/turbo-lamp/blob/main/.github/workflows/podman.yml [4]https://noobient.com/2023/11/15/fixing-ubuntu-containers-failing-to-start-with-systemd/ ** Changed in: crun (Ubuntu Jammy) Status: Incomplete => In Progress -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056442 Title: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2056442/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056442] Re: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported
Echoing Robie's question, and marking task as incomplete pending a test plan update. ** Changed in: crun (Ubuntu Jammy) Status: In Progress => Incomplete -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056442 Title: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2056442/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056442] Re: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported
> [Test plan] > Needs to be finalized... Is this finalized yet please? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056442 Title: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2056442/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056442] Re: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported
Ok, in #6 I tested with debian 11 which is not impacted, re-testing... Ok all good, I can reproduce it too: buildbot@bb-monty-bbw1-x64-ubuntu-2204-amd64:~$ uname -r 6.5.0-1015-azure buildbot@bb-monty-bbw1-x64-ubuntu-2204-amd64:~$ podman run --rm -d ghcr.io/fauust/docker-systemd:debian-12 Error: OCI runtime error: chmod `run/shm`: Operation not supported -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056442 Title: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2056442/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056442] Re: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported
Hi Markus! Indeed not all images are impacted, here is a non exhaustive list (alpine is particular since it use OpenRC): https://github.com/fauust/docker-systemd/actions/runs/8139697623 My problem was that I was not able to reproduce it, see #6. Anyway, I am a bit lost here but apparently some of you could reproduce the bug and more importantly a fix is in the pipe. Hopefully it will hit next GH runner images soon so that I will not have to modify all my CI depending on it. The current workaround in my case (forcing arguments) is a bit more complex since there is another layer above podman (molecule) and I don't want to spend time fixing it if the fix works, see: https://github.com/fauust/ansible-role-mariadb/actions. In any case, thank you all for your reactivity on this! -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056442 Title: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2056442/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056442] Re: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported
Hi Faustin, #12 This bug is a little bit about which image to use, because not every image is triggering this bug. First, systemd must be present in the image. Second, some images are known to not trigger it, e.g. Debian older than 12, Fedora, Rocky, the whole RedHat world does not show the behaviour. #16 I used the --systemd always switch in #8 because it makes my life easier not because it is strictly necessary. If I did not use it I had to specify --tmpfs and volume for /sys/fs/cgroup manually (it does a bunch foo magic in the background), see podman-run(1). But, of course, the complexity grows with rootful vs. rootless or cgroups v1 vs. cgroups v2 and stuff. It's easy to mix issues. FWIW, here are my references for this bug. https://noobient.com/2023/11/15/fixing-ubuntu-containers-failing-to-start-with-systemd/ https://github.com/containers/crun/issues/1308 https://github.com/containers/crun/pull/1309 ** Bug watch added: github.com/containers/crun/issues #1308 https://github.com/containers/crun/issues/1308 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056442 Title: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2056442/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056442] Re: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported
Indeed, the issue was that I tried to use ENTRYPOINT instead of CMD to start systemd. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056442 Title: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2056442/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056442] Re: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported
Hi Julian! Strange, the GH CI was failing for me without this argument, see https://github.com/fauust/docker- systemd/blob/main/.github/workflows/build.yml#L129-L140 and https://github.com/fauust/docker- systemd/actions/runs/8139697623/job/22386833452. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056442 Title: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2056442/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056442] Re: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported
Test plan adjusted; fix uploaded. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056442 Title: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2056442/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056442] Re: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported
The success case surprisingly ends up being: ubuntu@test-crun-20240307:~$ podman run --systemd always systemd Explicit --user argument required to run as user manager. But using CMD makes it work. weird stuff ** Description changed: [Impact] podman containers using the default crun backend do not work anymore with the 6.5 HWE kernel in 22.04 [Test plan] Needs to be finalized, but roughly: 1. Boot jammy with 6.5 HWE kernel (e.g. install linux-virtual-hwe-22.04 in a VM) 2. `podman build -t systemd .` with the Dockerfile: FROM ubuntu:noble RUN apt install -U systemd -y - ENTRYPOINT /usr/bin/systemd - 3. Run it `podman run --systemd always systemd` + CMD ["/lib/systemd/systemd"] + 3. Run it `podman run --systemd always systemd` you should not get Error: OCI runtime error: chmod `run/shm`: Operation not supported - Optimally submitter can do end-to-end-verification on their side. [Where problems could occur] The patch ignores ENOTSUP for fchmodat() in one function, so at most we could silently hide some other issues in fchmodat() inside that function, e.g. AppArmor denials. But generally that is what you would want as a behavior for ENOTSUP... [Original bug report] The problem is very well described in https://github.com/actions/runner- images/issues/9425. ## COPY FROM LINK I think there might be a regression in this release of the ubuntu-22.04 image which breaks podman.[1] The image updated the kernel from 6.2.y to 6.5.y, but podman/crun don't seem to be updated. Our build fails with this error link to run:[2] STEP 1/1: FROM ghcr.io/gardenlinux/builder:3ab2bb52bc46bb200c761369c087e9413d1ce0ac Trying to pull ghcr.io/gardenlinux/builder:3ab2bb52bc46bb200c761369c087e9413d1ce0ac... Getting image source signatures Copying blob sha256:041b542221cfde2f9fa4ac13f8b5804e25b23ab48ba47db2822c382a134256e1 Copying blob sha256:041b542221cfde2f9fa4ac13f8b5804e25b23ab48ba47db2822c382a134256e1 Copying config sha256:1eba10d0345cc6df78b7c3a6ced45da9db675d05eb20d5d286996e4f7ffb24d5 Writing manifest to image destination Storing signatures COMMIT localhost/builder --> 1eba10d0345 Successfully tagged localhost/builder:latest Successfully tagged ghcr.io/gardenlinux/builder:3ab2bb52bc46bb200c761369c087e9413d1ce0ac 1eba10d0345cc6df78b7c3a6ced45da9db675d05eb20d5d286996e4f7ffb24d5 Error: OCI runtime error: chmod `run/shm`: Operation not supported Error: Process completed with exit code 126. This is with this image version: Current runner version: '2.313.0' Operating System Ubuntu 22.04.4 LTS Runner Image Image: ubuntu-22.04 Version: 20240225.1.0 Included Software: https://github.com/actions/runner-images/blob/ubuntu22/20240225.1/images/ubuntu/Ubuntu2204-Readme.md Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu22%2F20240225.1 Trying to reproduce, it seems like I am only able to get this image version 20240218.1.0 where the issue does not appear. Tried to reproduce in this repo[3], but I'm not able to get this with image version 20240225.1.0. Is this a known issue and version 20240225.1.0 is not in use anymore? This blog post seems to suggest that the crun version is too old.[4] [1]https://github.com/actions/runner-images/releases/tag/ubuntu22%2F20240225.1 [2]https://github.com/gardenlinux/gardenlinux/actions/runs/8061893233/job/22020522535 [3]https://github.com/fwilhe/turbo-lamp/blob/main/.github/workflows/podman.yml [4]https://noobient.com/2023/11/15/fixing-ubuntu-containers-failing-to-start-with-systemd/ ** Changed in: crun (Ubuntu Jammy) Status: Incomplete => In Progress ** No longer affects: libpod (Ubuntu) ** No longer affects: libpod (Ubuntu Jammy) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056442 Title: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2056442/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056442] Re: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported
I can reproduce the issue now, I missed a --systemd always argument. I used ubuntu@test-crun-20240307:~$ cat Dockerfile FROM ubuntu:noble RUN apt install -U systemd -y ENTRYPOINT /usr/bin/systemd ubuntu@test-crun-20240307:~$ podman run --systemd always systemd Error: OCI runtime error: chmod `run/shm`: Operation not supported ** Description changed: [Impact] podman containers using the default crun backend do not work anymore with the 6.5 HWE kernel in 22.04 [Test plan] Needs to be finalized, but roughly: 1. Boot jammy with 6.5 HWE kernel (e.g. install linux-virtual-hwe-22.04 in a VM) - 2. podman run --rm some image + 2. `podman build -t systemd .` with the Dockerfile: + FROM ubuntu:noble + RUN apt install -U systemd -y + ENTRYPOINT /usr/bin/systemd + 3. Run it `podman run --systemd always systemd` + Error: OCI runtime error: chmod `run/shm`: Operation not supported + Optimally submitter can do end-to-end-verification on their side. [Where problems could occur] The patch ignores ENOTSUP for fchmodat() in one function, so at most we could silently hide some other issues in fchmodat() inside that function, e.g. AppArmor denials. But generally that is what you would want as a behavior for ENOTSUP... [Original bug report] The problem is very well described in https://github.com/actions/runner- images/issues/9425. ## COPY FROM LINK I think there might be a regression in this release of the ubuntu-22.04 image which breaks podman.[1] The image updated the kernel from 6.2.y to 6.5.y, but podman/crun don't seem to be updated. Our build fails with this error link to run:[2] STEP 1/1: FROM ghcr.io/gardenlinux/builder:3ab2bb52bc46bb200c761369c087e9413d1ce0ac Trying to pull ghcr.io/gardenlinux/builder:3ab2bb52bc46bb200c761369c087e9413d1ce0ac... Getting image source signatures Copying blob sha256:041b542221cfde2f9fa4ac13f8b5804e25b23ab48ba47db2822c382a134256e1 Copying blob sha256:041b542221cfde2f9fa4ac13f8b5804e25b23ab48ba47db2822c382a134256e1 Copying config sha256:1eba10d0345cc6df78b7c3a6ced45da9db675d05eb20d5d286996e4f7ffb24d5 Writing manifest to image destination Storing signatures COMMIT localhost/builder --> 1eba10d0345 Successfully tagged localhost/builder:latest Successfully tagged ghcr.io/gardenlinux/builder:3ab2bb52bc46bb200c761369c087e9413d1ce0ac 1eba10d0345cc6df78b7c3a6ced45da9db675d05eb20d5d286996e4f7ffb24d5 Error: OCI runtime error: chmod `run/shm`: Operation not supported Error: Process completed with exit code 126. This is with this image version: Current runner version: '2.313.0' Operating System Ubuntu 22.04.4 LTS Runner Image Image: ubuntu-22.04 Version: 20240225.1.0 Included Software: https://github.com/actions/runner-images/blob/ubuntu22/20240225.1/images/ubuntu/Ubuntu2204-Readme.md Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu22%2F20240225.1 Trying to reproduce, it seems like I am only able to get this image version 20240218.1.0 where the issue does not appear. Tried to reproduce in this repo[3], but I'm not able to get this with image version 20240225.1.0. Is this a known issue and version 20240225.1.0 is not in use anymore? This blog post seems to suggest that the crun version is too old.[4] [1]https://github.com/actions/runner-images/releases/tag/ubuntu22%2F20240225.1 [2]https://github.com/gardenlinux/gardenlinux/actions/runs/8061893233/job/22020522535 [3]https://github.com/fwilhe/turbo-lamp/blob/main/.github/workflows/podman.yml [4]https://noobient.com/2023/11/15/fixing-ubuntu-containers-failing-to-start-with-systemd/ -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056442 Title: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2056442/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056442] Re: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported
Hi Markus! The problem is not which image to use (you can use one of mine, see https://github.com/fauust/docker-systemd/ and #6). Or as explained in #5 by Julian, you can also try to build gardenlinux. But if your are able to reproduce the bug on your setup that's good, can you please describe your setup. Thanks! -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056442 Title: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2056442/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056442] Re: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported
for the record, the crun from https://launchpad.net/~juliank/+archive/ubuntu/podman/+packages passes my reproducer Setting up crun (0.17+dfsg-1.1ubuntu0.1) ... ...snip $ podman run --systemd always -d cruntest 792cc9c24c9c1e936686e63c26befdb67b7e45dd8f9e70c608d164fb49fff7a4 snap... no error message, compare with my reproducer in comment #8 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056442 Title: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2056442/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056442] Re: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: libpod (Ubuntu Jammy) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056442 Title: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2056442/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056442] Re: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: libpod (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056442 Title: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2056442/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056442] Re: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported
to reproduce create a systemd image ...snip $ cat - >Containerfile
[Bug 2056442] Re: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported
** Changed in: crun (Ubuntu Jammy) Assignee: (unassigned) => Julian Andres Klode (juliank) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056442 Title: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2056442/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056442] Re: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported
The problem we have is that we need to be able to reproduce it so we can verify the change actually fixes the bug, otherwise we can't even upload it. ** Also affects: crun (Ubuntu Jammy) Importance: Undecided Status: New ** Also affects: libpod (Ubuntu Jammy) Importance: Undecided Status: New ** Changed in: crun (Ubuntu Jammy) Status: New => Incomplete ** Changed in: crun (Ubuntu) Status: Incomplete => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056442 Title: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2056442/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056442] Re: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported
I am not sure that this is something easy to find (at least I don't know how to proceed). And above all, my understanding is that the problem seems to be related to the "old" crun version and quite new kernels (6.5). Also, 22.04 (and Bullseye) comes with crun 0.17 and there is a huge gab with the next release 23.04 that seems to come with crun 1.8-1 (same version in Bookworm). Anyway, after testing, I can't reproduce the problem on a generic cloud- init image, so that can maybe help? Here is my test setup: - Ubuntu 22.04 KVM/Qemu started from generic cloud-init images (amd64); - kernel upgraded to 6.5.0-1015-azure; - podman/crun installed from ubuntu repos. Following command works: $ podman run --rm -d ghcr.io/fauust/docker-systemd:debian-11 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056442 Title: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2056442/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056442] Re: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported
I installed podman on a jammy host in Azure and I tried running a default noble image and building an image from whater gardenlinux is (same as in the bug report), but both work fine. ubuntu@test-crun-20240307:~$ uname -r 6.5.0-1015-azure ubuntu@test-crun-20240307:~$ podman run --rm -it ubuntu:jammy bash #Resolved "ubuntu" as an alias (/etc/containers/registries.conf.d/shortnames.conf) Trying to pull docker.io/library/ubuntu:jammy... Getting image source signatures Copying blob bccd10f490ab done Copying config ca2b0f2696 done Writing manifest to image destination Storing signatures root@e7e468635c7e:/# exit ubuntu@test-crun-20240307:~$ podman build --squash . STEP 1/2: FROM ghcr.io/gardenlinux/builder:3ab2bb52bc46bb200c761369c087e9413d1ce0ac STEP 2/2: RUN echo hello hello COMMIT Getting image source signatures Copying blob 4a906f38f63e skipped: already exists Copying blob 60432b4390ff done Copying config 3f586b4fd7 done Writing manifest to image destination Storing signatures --> 3f586b4fd7e 3f586b4fd7e43d2f13e42a1de376489907c53151f679af1bebf9e2795f013935 ** Changed in: crun (Ubuntu) Status: New => Incomplete -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056442 Title: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2056442/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056442] Re: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported
If you have some time please let me know if that works for you. I tried spinning up a jammy lxd VM but sadly my VM agent crashed or something and I can't access the console due to other bugs so it takes a while to test this on my end - but this is basically the upstream patch applied. ** Description changed: + [Impact] + podman containers using the default crun backend do not work anymore with the 6.5 HWE kernel in 22.04 + + [Test plan] + Needs to be finalized, but roughly: + + 1. Boot jammy with 6.5 HWE kernel (e.g. install linux-virtual-hwe-22.04 in a VM) + 2. podman run --rm some image + + Optimally submitter can do end-to-end-verification on their side. + + [Where problems could occur] + The patch ignores ENOTSUP for fchmodat() in one function, so at most we could silently hide some other issues in fchmodat() inside that function, e.g. AppArmor denials. But generally that is what you would want as a behavior for ENOTSUP... + + [Original bug report] + The problem is very well described in https://github.com/actions/runner- images/issues/9425. ## COPY FROM LINK I think there might be a regression in this release of the ubuntu-22.04 image which breaks podman.[1] The image updated the kernel from 6.2.y to 6.5.y, but podman/crun don't seem to be updated. Our build fails with this error link to run:[2] STEP 1/1: FROM ghcr.io/gardenlinux/builder:3ab2bb52bc46bb200c761369c087e9413d1ce0ac Trying to pull ghcr.io/gardenlinux/builder:3ab2bb52bc46bb200c761369c087e9413d1ce0ac... Getting image source signatures Copying blob sha256:041b542221cfde2f9fa4ac13f8b5804e25b23ab48ba47db2822c382a134256e1 Copying blob sha256:041b542221cfde2f9fa4ac13f8b5804e25b23ab48ba47db2822c382a134256e1 Copying config sha256:1eba10d0345cc6df78b7c3a6ced45da9db675d05eb20d5d286996e4f7ffb24d5 Writing manifest to image destination Storing signatures COMMIT localhost/builder --> 1eba10d0345 Successfully tagged localhost/builder:latest Successfully tagged ghcr.io/gardenlinux/builder:3ab2bb52bc46bb200c761369c087e9413d1ce0ac 1eba10d0345cc6df78b7c3a6ced45da9db675d05eb20d5d286996e4f7ffb24d5 Error: OCI runtime error: chmod `run/shm`: Operation not supported Error: Process completed with exit code 126. This is with this image version: Current runner version: '2.313.0' Operating System - Ubuntu - 22.04.4 - LTS + Ubuntu + 22.04.4 + LTS Runner Image - Image: ubuntu-22.04 - Version: 20240225.1.0 - Included Software: https://github.com/actions/runner-images/blob/ubuntu22/20240225.1/images/ubuntu/Ubuntu2204-Readme.md - Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu22%2F20240225.1 + Image: ubuntu-22.04 + Version: 20240225.1.0 + Included Software: https://github.com/actions/runner-images/blob/ubuntu22/20240225.1/images/ubuntu/Ubuntu2204-Readme.md + Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu22%2F20240225.1 Trying to reproduce, it seems like I am only able to get this image version 20240218.1.0 where the issue does not appear. Tried to reproduce in this repo[3], but I'm not able to get this with image version 20240225.1.0. Is this a known issue and version 20240225.1.0 is not in use anymore? This blog post seems to suggest that the crun version is too old.[4] [1]https://github.com/actions/runner-images/releases/tag/ubuntu22%2F20240225.1 [2]https://github.com/gardenlinux/gardenlinux/actions/runs/8061893233/job/22020522535 [3]https://github.com/fwilhe/turbo-lamp/blob/main/.github/workflows/podman.yml [4]https://noobient.com/2023/11/15/fixing-ubuntu-containers-failing-to-start-with-systemd/ -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056442 Title: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2056442/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056442] Re: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported
Fix and follow up fix cherry-picked in https://launchpad.net/~juliank/+archive/ubuntu/podman/+packages -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056442 Title: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2056442/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056442] Re: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported
upstream commit in `crun` : https://github.com/containers/crun/pull/1309/commits/57262a2710c83fa08767f0ce3ba7a80993515bb2 I think a trivial reproducer, outside GH runners, is * start a system with a 6.5 kernel (for a VM downloaded from cloud-images, need to install the HWE kernel. or start on your favourite $BIG_CLOUD as they should have all rolled. Azure is an environment) * install podman from Ubuntu 22.04 archive * try to run a container That _should_ be it, if I'm reading everything correctly. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056442 Title: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2056442/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056442] Re: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported
If you find out which commit(s) in crun need to be cherry-picked and how to reproducibly verify that this fixes the issue, I'm happy to upload them for you. ** Summary changed: - Podman (crun) regression in Ubuntu 22.04 + Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056442 Title: Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/2056442/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
