[Bug 1878225] Re: Please remove lxd.snap from lxd images, as it fails to seed thus failing the first boot - snapd.seeded.service waits forever (?) to have snaps seeded in LXD on s390x and arm64
Ubuntu 20.10 (Groovy Gorilla) has reached end of life, so this bug will not be fixed for that specific release. ** Changed in: autopkgtest (Ubuntu Groovy) Status: Triaged => Won't Fix -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1878225 Title: Please remove lxd.snap from lxd images, as it fails to seed thus failing the first boot - snapd.seeded.service waits forever (?) to have snaps seeded in LXD on s390x and arm64 To manage notifications about this bug go to: https://bugs.launchpad.net/auto-package-testing/+bug/1878225/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1878225] Re: Please remove lxd.snap from lxd images, as it fails to seed thus failing the first boot - snapd.seeded.service waits forever (?) to have snaps seeded in LXD on s390x and arm64
I'm seeing snapd stall on boot of generic ubuntu:22.04 images via lxd. root@linuxtest2:~# systemctl list-jobs JOB UNIT TYPE STATE 151 cloud-config.service start waiting 133 snapd.seeded.service start running 148 cloud-init.targetstart waiting 128 systemd-update-utmp-runlevel.service start waiting 117 snapd.autoimport.service start waiting 1 graphical.target start waiting 150 cloud-final.service start waiting 2 multi-user.targetstart waiting I'm trying to get user-data scripts to run, but snapd never completes its run. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1878225 Title: Please remove lxd.snap from lxd images, as it fails to seed thus failing the first boot - snapd.seeded.service waits forever (?) to have snaps seeded in LXD on s390x and arm64 To manage notifications about this bug go to: https://bugs.launchpad.net/auto-package-testing/+bug/1878225/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1878225] Re: Please remove lxd.snap from lxd images, as it fails to seed thus failing the first boot - snapd.seeded.service waits forever (?) to have snaps seeded in LXD on s390x and arm64
** Changed in: snapd Assignee: Samuele Pedroni (pedronis) => (unassigned) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1878225 Title: Please remove lxd.snap from lxd images, as it fails to seed thus failing the first boot - snapd.seeded.service waits forever (?) to have snaps seeded in LXD on s390x and arm64 To manage notifications about this bug go to: https://bugs.launchpad.net/auto-package-testing/+bug/1878225/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1878225] Re: Please remove lxd.snap from lxd images, as it fails to seed thus failing the first boot - snapd.seeded.service waits forever (?) to have snaps seeded in LXD on s390x and arm64
I'm marking this as triaged. We should look at the seeding code to ensure it can eventually actually fail and allow systems to continue booting. ** Changed in: snapd Status: New => Triaged ** Changed in: snapd Importance: Undecided => Medium -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1878225 Title: Please remove lxd.snap from lxd images, as it fails to seed thus failing the first boot - snapd.seeded.service waits forever (?) to have snaps seeded in LXD on s390x and arm64 To manage notifications about this bug go to: https://bugs.launchpad.net/auto-package-testing/+bug/1878225/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1878225] Re: Please remove lxd.snap from lxd images, as it fails to seed thus failing the first boot - snapd.seeded.service waits forever (?) to have snaps seeded in LXD on s390x and arm64
** Changed in: snapd Assignee: (unassigned) => Samuele Pedroni (pedronis) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1878225 Title: Please remove lxd.snap from lxd images, as it fails to seed thus failing the first boot - snapd.seeded.service waits forever (?) to have snaps seeded in LXD on s390x and arm64 To manage notifications about this bug go to: https://bugs.launchpad.net/auto-package-testing/+bug/1878225/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1878225] Re: Please remove lxd.snap from lxd images, as it fails to seed thus failing the first boot - snapd.seeded.service waits forever (?) to have snaps seeded in LXD on s390x and arm64
** Tags removed: rls-gg-incoming ** Also affects: autopkgtest (Ubuntu Groovy) Importance: Undecided Status: Triaged ** Also affects: lxd (Ubuntu Groovy) Importance: Undecided Status: Invalid ** Also affects: snapd (Ubuntu Groovy) Importance: Critical Status: Invalid -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1878225 Title: Please remove lxd.snap from lxd images, as it fails to seed thus failing the first boot - snapd.seeded.service waits forever (?) to have snaps seeded in LXD on s390x and arm64 To manage notifications about this bug go to: https://bugs.launchpad.net/auto-package-testing/+bug/1878225/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1878225] Re: Please remove lxd.snap from lxd images, as it fails to seed thus failing the first boot - snapd.seeded.service waits forever (?) to have snaps seeded in LXD on s390x and arm64
@stgraber I'd fully agree if snapd.seeded timed out after a few minutes, but I have a container where snapd.seeded.service is running for 49 days keeping the system in a starting state. While this problem may not occur only with snapd, but the snapd package plays an important enough role in the software stack around Ubuntu to be fixed. I agree that autopkgtest network should reject rather than drop packets because the assumption in autopkgtests is having Internet access. From https://people.debian.org/~mpitt/autopkgtest/README.package-tests.html: ... Network access autopkgtest needs access to the network at least for downloading test dependencies and possibly dist-upgrading testbeds. In environments with restricted internet access you need to set up an apt proxy and configure the testbed to use it. (Note that the standard tools like autopkgtest- build-lxc or mk-sbuild automatically use the apt proxy from the host system.) In general, tests are also allowed to access the internet. As this usually makes tests less reliable, this should be kept to a minimum; but for many packages their main purpose is to interact with remote web services and thus their testing should actually cover those too, to ensure that the distribution package keeps working with their corresponding web service. Debian's production CI infrastructure allows unrestricted network access, in Ubuntu's infrastructure access to sites other than *.ubuntu.com and *.launchpad.net happens via a proxy (limited to DNS and http/https). By having a stricter network setup in Ubuntu we are set up for facing failures time to time and rejecting packets at least speed up those failures. Autopkgtest can also be fixed to set up proxy for snaps. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1878225 Title: Please remove lxd.snap from lxd images, as it fails to seed thus failing the first boot - snapd.seeded.service waits forever (?) to have snaps seeded in LXD on s390x and arm64 To manage notifications about this bug go to: https://bugs.launchpad.net/auto-package-testing/+bug/1878225/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1878225] Re: Please remove lxd.snap from lxd images, as it fails to seed thus failing the first boot - snapd.seeded.service waits forever (?) to have snaps seeded in LXD on s390x and arm64
Go's http stack will eventually notice, but a TCP timeout can take up to 5 minutes, so the timeout in autopkgtest is nowhere near long enough to detect that. It's also by far not a snapd-specific thing. apt will experience the exact same issue and would also hold up boot if used through cloud-init. The only reason why apt doesn't hold up boot here is because autopkgtest has specific logic to configure its proxy. That same logic is missing for snapd causing this issue. So I don't think that messing with http/tcp timeouts in snapd is the right solution here. Instead the main actionable items would be: - Have the autopkgtest network reject packets rather than drop them (that would have avoided this issue) - Have autopkgtest's proxy config logic also configure the proxy for snapd (either through unit override or through /etc/environment) Either of those is sufficient to avoid this problem. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1878225 Title: Please remove lxd.snap from lxd images, as it fails to seed thus failing the first boot - snapd.seeded.service waits forever (?) to have snaps seeded in LXD on s390x and arm64 To manage notifications about this bug go to: https://bugs.launchpad.net/auto-package-testing/+bug/1878225/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1878225] Re: Please remove lxd.snap from lxd images, as it fails to seed thus failing the first boot - snapd.seeded.service waits forever (?) to have snaps seeded in LXD on s390x and arm64
@stgraber Thanks for triaging this @juliank @laney @johan-ehnberg I think snapd should not block cloud-init forever so I think autopkgtest nor autopkgtest infra is not at fault here and should not be fixed. I place a workaround to systemd's autopkgtest and I hope snapd(.seeded) will be able to detect when packets are dropped and fail unblocking cloud-init. ** Changed in: snapd Status: Invalid => New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1878225 Title: Please remove lxd.snap from lxd images, as it fails to seed thus failing the first boot - snapd.seeded.service waits forever (?) to have snaps seeded in LXD on s390x and arm64 To manage notifications about this bug go to: https://bugs.launchpad.net/auto-package-testing/+bug/1878225/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1878225] Re: Please remove lxd.snap from lxd images, as it fails to seed thus failing the first boot - snapd.seeded.service waits forever (?) to have snaps seeded in LXD on s390x and arm64
Can someone provide a merge request for lp:autopkgtest-cloud please? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1878225 Title: Please remove lxd.snap from lxd images, as it fails to seed thus failing the first boot - snapd.seeded.service waits forever (?) to have snaps seeded in LXD on s390x and arm64 To manage notifications about this bug go to: https://bugs.launchpad.net/auto-package-testing/+bug/1878225/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1878225] Re: Please remove lxd.snap from lxd images, as it fails to seed thus failing the first boot - snapd.seeded.service waits forever (?) to have snaps seeded in LXD on s390x and arm64
** Changed in: auto-package-testing Status: Invalid => Triaged -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1878225 Title: Please remove lxd.snap from lxd images, as it fails to seed thus failing the first boot - snapd.seeded.service waits forever (?) to have snaps seeded in LXD on s390x and arm64 To manage notifications about this bug go to: https://bugs.launchpad.net/auto-package-testing/+bug/1878225/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1878225] Re: Please remove lxd.snap from lxd images, as it fails to seed thus failing the first boot - snapd.seeded.service waits forever (?) to have snaps seeded in LXD on s390x and arm64
I've managed to reproduce this issue by reproducing the network setup of autopkgtest. All you have to do is setup your network to allow access only through an http proxy and drop all other traffic. Do that and you'll get: stgraber@castiana:~$ autopkgtest-build-lxd ubuntu-daily:groovy Creating autopkgtest-prepare-yzE Starting autopkgtest-prepare-yzE Created symlink /etc/systemd/system/serial-getty@getty.service → /dev/null. Timed out waiting for container to boot stgraber@castiana:~$ Note that the autopkgtest network drops traffic, it doesn't reject it, so you're not hitting normal connection failures that can be nicely handled. As a result snapd gets stuck in seeding. Manually fixing the autopkgtest-build-lxd script to configure snapd's proxy in the container, gets you. stgraber@castiana:~$ autopkgtest-build-lxd ubuntu-daily:groovy Creating autopkgtest-prepare-ybH Starting autopkgtest-prepare-ybH Created symlink /etc/systemd/system/serial-getty@getty.service → /dev/null. Container finished booting. Distribution Ubuntu, release groovy, architecture amd64 Running setup script /usr/share/autopkgtest/setup-commands/setup-testbed... sh: Attempting to set up Debian/Ubuntu apt sources automatically sh: Distribution appears to be Ubuntu Get:1 http://us.archive.ubuntu.com/ubuntu groovy InRelease [267 kB] Hit:2 http://us.archive.ubuntu.com/ubuntu groovy-updates InRelease Hit:3 http://us.archive.ubuntu.com/ubuntu groovy-security InRelease Get:4 http://us.archive.ubuntu.com/ubuntu groovy/main Sources [841 kB] Get:5 http://us.archive.ubuntu.com/ubuntu groovy/multiverse Sources [177 kB] Get:6 http://us.archive.ubuntu.com/ubuntu groovy/universe Sources [9906 kB] Get:7 http://us.archive.ubuntu.com/ubuntu groovy/restricted Sources [6476 B] Get:8 http://us.archive.ubuntu.com/ubuntu groovy/main amd64 Packages [975 kB] Get:9 http://us.archive.ubuntu.com/ubuntu groovy/main amd64 c-n-f Metadata [29.5 kB] Get:10 http://us.archive.ubuntu.com/ubuntu groovy/universe amd64 Packages [8734 kB] Get:11 http://us.archive.ubuntu.com/ubuntu groovy/universe amd64 c-n-f Metadata [267 kB] Get:12 http://us.archive.ubuntu.com/ubuntu groovy/multiverse amd64 Packages [154 kB] Get:13 http://us.archive.ubuntu.com/ubuntu groovy/multiverse amd64 c-n-f Metadata [9320 B] Get:14 http://us.archive.ubuntu.com/ubuntu groovy-updates/universe amd64 c-n-f Metadata [112 B] Get:15 http://us.archive.ubuntu.com/ubuntu groovy-updates/multiverse amd64 c-n-f Metadata [116 B] Get:16 http://us.archive.ubuntu.com/ubuntu groovy-security/universe amd64 c-n-f Metadata [116 B] Get:17 http://us.archive.ubuntu.com/ubuntu groovy-security/multiverse amd64 c-n-f Metadata [116 B] Fetched 21.4 MB in 4s (4957 kB/s) Marking all tasks invalid and re-opening autopkgtest task as that's what's broken here. ** Changed in: lxd (Ubuntu) Status: Incomplete => Invalid ** Changed in: snapd (Ubuntu) Status: Confirmed => Invalid ** Changed in: snapd Status: New => Invalid ** Changed in: cloud-images Status: New => Invalid ** Changed in: autopkgtest (Ubuntu) Status: Invalid => Triaged -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1878225 Title: Please remove lxd.snap from lxd images, as it fails to seed thus failing the first boot - snapd.seeded.service waits forever (?) to have snaps seeded in LXD on s390x and arm64 To manage notifications about this bug go to: https://bugs.launchpad.net/auto-package-testing/+bug/1878225/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1878225] Re: Please remove lxd.snap from lxd images, as it fails to seed thus failing the first boot - snapd.seeded.service waits forever (?) to have snaps seeded in LXD on s390x and arm64
@stgraber The "weird adt environment" is the autopkgtest infra we use for gating packages to prevent regressions sneaking in to the release pocket. LXD is not a .deb package but a snap so we can't gate regressions as part of the standard Ubuntu classic process, but at the moment it is somehow part of the snaps installed on images by default. The reason for adding the tests-in-lxd test to systemd is to ensure that new systemd versions work well running LXD and being installed in LXD container at the same time. Upstream changes frequently introduce LXC specific minor regressions thus I believe this test helps making the user experience better. If you see something in the aforementioned new test that is not expected to work, please state that. Otherwise please triage the failure that we are observing consistently on the autopkgtest infra. The easiest way of reproducing the problem is clicking on the "♻" links on this page: http://autopkgtest.ubuntu.com/packages/systemd/groovy/amd64 I agree with @xnox that snapd.seeding must not block boot indefinitely and I also believe that snap upstreams (LXD here) most ensure fixing problems affecting Ubuntu classic to have the privilege of being seeded on images. Shipping LXD as a .deb would also resolve the problem it is causing here by being a snap to be seeded inside the LXC container and the autopkgtest infra could gate LXD regressions again. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1878225 Title: Please remove lxd.snap from lxd images, as it fails to seed thus failing the first boot - snapd.seeded.service waits forever (?) to have snaps seeded in LXD on s390x and arm64 To manage notifications about this bug go to: https://bugs.launchpad.net/auto-package-testing/+bug/1878225/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1878225] Re: Please remove lxd.snap from lxd images, as it fails to seed thus failing the first boot - snapd.seeded.service waits forever (?) to have snaps seeded in LXD on s390x and arm64
Ok, I'm getting really tired of this one. Can you please get me a reproducer for this on stock (not some weird adt environment) groovy or focal please? As in, deploy the host using a clean image, install LXD normally from the snap store and then deploy an official Ubuntu image on top of that. So far none of the instructions showed in this bug have reproduced this issue here, trying on 18.04, 20.04 or groovy and on both amd64 and s390x. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1878225 Title: Please remove lxd.snap from lxd images, as it fails to seed thus failing the first boot - snapd.seeded.service waits forever (?) to have snaps seeded in LXD on s390x and arm64 To manage notifications about this bug go to: https://bugs.launchpad.net/auto-package-testing/+bug/1878225/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1878225] Re: Please remove lxd.snap from lxd images, as it fails to seed thus failing the first boot - snapd.seeded.service waits forever (?) to have snaps seeded in LXD on s390x and arm64
@stgraber Do you tell that the default LXD in groovy in autopkgtest CI does not qualify as current? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1878225 Title: Please remove lxd.snap from lxd images, as it fails to seed thus failing the first boot - snapd.seeded.service waits forever (?) to have snaps seeded in LXD on s390x and arm64 To manage notifications about this bug go to: https://bugs.launchpad.net/auto-package-testing/+bug/1878225/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1878225] Re: Please remove lxd.snap from lxd images, as it fails to seed thus failing the first boot - snapd.seeded.service waits forever (?) to have snaps seeded in LXD on s390x and arm64
Hmm, also, so far we've not seen any reproducer of this being an issue with current LXD. All my tests show the seeding working properly and we definitely have a bunch of users using the pre-seeded LXD in Ubuntu 20.04 images run inside containers. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1878225 Title: Please remove lxd.snap from lxd images, as it fails to seed thus failing the first boot - snapd.seeded.service waits forever (?) to have snaps seeded in LXD on s390x and arm64 To manage notifications about this bug go to: https://bugs.launchpad.net/auto-package-testing/+bug/1878225/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1878225] Re: Please remove lxd.snap from lxd images, as it fails to seed thus failing the first boot - snapd.seeded.service waits forever (?) to have snaps seeded in LXD on s390x and arm64
On classic, snapd.seeding must not block boot indefinately, as it prevents normal operation of the classic system and potentially access to it. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1878225 Title: Please remove lxd.snap from lxd images, as it fails to seed thus failing the first boot - snapd.seeded.service waits forever (?) to have snaps seeded in LXD on s390x and arm64 To manage notifications about this bug go to: https://bugs.launchpad.net/auto-package-testing/+bug/1878225/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1878225] Re: Please remove lxd.snap from lxd images, as it fails to seed thus failing the first boot - snapd.seeded.service waits forever (?) to have snaps seeded in LXD on s390x and arm64
@xnox wrote: > It sounds like we must remove seeded snaps from our LXD images, and not > install > any seeded snaps inside our container. And like only install the lxd stub deb. > Cause it looks like seeding snaps is not supported inside classic lxd > containers. Let's remember that these aren't just LXD images, they are generic squashfs images from cloud-images.ubuntu.com. This raises the issue that if the generic cloud squashfs is run in any environment where snap seeding can not complete then boot is stuck and the instance is broken without cloud-init completing. This might mean no ssh keys and difficulties debugging. So I'll reiterate that this is not an acceptable solution and note that the issue is broader than even just unprivileged LXD containers. Removing all snaps from the squashfs feels like a significant lost opportunity. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1878225 Title: Please remove lxd.snap from lxd images, as it fails to seed thus failing the first boot - snapd.seeded.service waits forever (?) to have snaps seeded in LXD on s390x and arm64 To manage notifications about this bug go to: https://bugs.launchpad.net/auto-package-testing/+bug/1878225/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
