So some thoughts: 1. The root failure of snapd failing to finish seeding is that it is trying to run `udevadm trigger --subsystem-nomatch=input` which ends up dying on every write like so:
openat(AT_FDCWD, "/sys/devices/virtual/workqueue/writeback/uevent", O_WRONLY|O_NOCTTY|O_NOFOLLOW|O_CLOEXEC) = -1 EACCES (Permission denied) writeback: Failed to write 'change' to '/sys/devices/virtual/workqueue/writeback/uevent': Permission denied which is peculiar to me because that command fails on both the 03 and the 04 image, but only causes snapd to fail to seed on the 04 image for some reason. This is triply peculiar to me because that same command run in a 21.04 container (amd64 and arm64) works fine without issues. Quadruply peculiar is that the same command in a 21.10 daily container on amd64 actually also fails, but again does not cause snapd to fail seeding on the amd64 image. My guess is that maybe snapd doesn't think it needs to run this command on the other 21.10 images and so it doesn't die on this command, but for some reason on the new arm64 images it thinks it has to run this command. Quintuply peculiar is the failure around EACCES. 2. If you diff the manifest between the image you see this change: -libc-bin 2.33-0ubuntu9 -libc6:arm64 2.33-0ubuntu9 +libc-bin 2.34-0ubuntu2 +libc6:arm64 2.34-0ubuntu2 I have no reason to believe that libc6 is the root of this problem, other than the fact that it has been at the root of many other problems that present themselves in weird ways especially when containers are involved. 3. The specific task that snapd is running when the failure happens seems to be either setup-profiles for the snapd snap OR running the install hook for lxd, although I don't think it is related at all to the specific install hook for the lxd snap since that doesn't seem to do anything real and it fails in different ways, sometimes it's a segmentation fault and sometimes it just says the hook exited 1. 4. If I modify the seed.yaml to not seed the lxd snap and then obliterate state.json and restart snapd, it now proceeds to be able to seed properly, so I think there is something about the lxd snap + the state of the system which implies to snapd that it needs to setup the udev backend which runs the failing command, most likely the fact that the lxd snap uses interfaces (even though it's interfaces are rather pointless in that they don't really have any policy). Indeed I can also successfully seed a snap like jq in the seed.yaml which has no interfaces used and it doesn't trigger snapd to run udev things. So part of the story is definitely that snapd thinks it needs to generate udev backend things for the lxd snap while seeding and that causes it to run that udevadm command which fails. Why this doesn't affect amd64 though is unclear to me. 5. The two rootfs' provided above are both preseeded with the same set of snaps with the same assertions and same seed.yaml file, so it doesn't appear to be a snapd regression or some trigger from the snaps themselves. Finally, I also note that this is also reproducible on a Raspberry Pi too if anyone wants to debug on their own, just run on a Pi: lxc launch ubuntu-daily:21.10 impish-testing and then you can see the issue. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1944004 Title: snapd.seeded.service never finishes on non-amd64 To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-images/+bug/1944004/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
