[Kernel-packages] [Bug 2045586] Re: livecd-rootfs uses losetup -P for theoretically reliable/synchronous partition setup but it's not reliable

2024-04-29 Thread Launchpad Bug Tracker
** Merge proposal linked: https://code.launchpad.net/~simpoir/livecd-rootfs/+git/livecd-rootfs/+merge/465193 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2045586 Title: livecd-root

[Kernel-packages] [Bug 2045586] Re: livecd-rootfs uses losetup -P for theoretically reliable/synchronous partition setup but it's not reliable

2024-02-21 Thread Catherine Redfield
I tested the flock-based solution with some of the CPC pipelines in jammy and saw consistently clean builds (30 successful images built yesterday). Thank you very much for everyone's hard work debugging and fixing this race condition! -- You received this bug notification because you are a membe

[Kernel-packages] [Bug 2045586] Re: livecd-rootfs uses losetup -P for theoretically reliable/synchronous partition setup but it's not reliable

2024-02-20 Thread Launchpad Bug Tracker
** Merge proposal linked: https://code.launchpad.net/~ubuntu-core-dev/livecd-rootfs/+git/livecd-rootfs/+merge/460810 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2045586 Title: liv

[Kernel-packages] [Bug 2045586] Re: livecd-rootfs uses losetup -P for theoretically reliable/synchronous partition setup but it's not reliable

2024-02-19 Thread Launchpad Bug Tracker
This bug was fixed in the package livecd-rootfs - 2.765.39 --- livecd-rootfs (2.765.39) jammy; urgency=medium [ dann frazier ] * Use flock to avoid races with systemd-udevd that cause loop device partitions to briefly disappear. (LP: #2045586) -- Michael Hudson-Doyle Mon,

[Kernel-packages] [Bug 2045586] Re: livecd-rootfs uses losetup -P for theoretically reliable/synchronous partition setup but it's not reliable

2024-02-19 Thread Łukasz Zemczak
Releasing early as we're in the middle of a point-release. Autopkgtests look good, the actual testing will happen on the RCs. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2045586 Title:

[Kernel-packages] [Bug 2045586] Re: livecd-rootfs uses losetup -P for theoretically reliable/synchronous partition setup but it's not reliable

2024-02-19 Thread Łukasz Zemczak
Hello Steve, or anyone else affected, Accepted livecd-rootfs into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd- rootfs/2.765.39 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See http

[Kernel-packages] [Bug 2045586] Re: livecd-rootfs uses losetup -P for theoretically reliable/synchronous partition setup but it's not reliable

2024-02-18 Thread Michael Hudson-Doyle
** Description changed: - In mantic, we migrated livecd-rootfs to use losetup -P instead of - kpartx, with the expectation that this would give us a reliable, race- - free way of loop-mounting partitions from a disk image during image - build. + [impact] + In mantic, we migrated livecd-rootfs to u

[Kernel-packages] [Bug 2045586] Re: livecd-rootfs uses losetup -P for theoretically reliable/synchronous partition setup but it's not reliable

2024-02-18 Thread Launchpad Bug Tracker
** Merge proposal linked: https://code.launchpad.net/~mwhudson/livecd-rootfs/+git/livecd-rootfs/+merge/460732 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2045586 Title: livecd-roo

[Kernel-packages] [Bug 2045586] Re: livecd-rootfs uses losetup -P for theoretically reliable/synchronous partition setup but it's not reliable

2024-02-18 Thread Michael Hudson-Doyle
Here is a backport of the partial fix to jammy https://code.launchpad.net/~mwhudson/livecd-rootfs/+git/livecd- rootfs/+merge/460729 I'm not sure I am best placed to update this bug description to match the SRU template though -- I'm not sure which builds are being affected in practice and so shoul

[Kernel-packages] [Bug 2045586] Re: livecd-rootfs uses losetup -P for theoretically reliable/synchronous partition setup but it's not reliable

2024-02-18 Thread Launchpad Bug Tracker
** Merge proposal linked: https://code.launchpad.net/~mwhudson/livecd-rootfs/+git/livecd-rootfs/+merge/460729 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2045586 Title: livecd-roo

[Kernel-packages] [Bug 2045586] Re: livecd-rootfs uses losetup -P for theoretically reliable/synchronous partition setup but it's not reliable

2024-02-09 Thread dann frazier
I'm glad to hear you are seeing an improvement! The current implementation is still racy as mentioned in Comment #22. I did a search of the logs I downloaded, and I believe this one shows that the mount_partition() race isn't just theoretical: https://autopkgtest.ubuntu.com/results/autopkgtest- ma

[Kernel-packages] [Bug 2045586] Re: livecd-rootfs uses losetup -P for theoretically reliable/synchronous partition setup but it's not reliable

2024-02-09 Thread Catherine Redfield
I've been seeing this bug as far back as losetup is used (jammy) and the previously merged fix has a significant improvement on successful builds (0/3 succeeded without the patch applied vs 5/5 succeeded with the patch applied this week) in jammy. Is someone already looking to SRU that patch back

[Kernel-packages] [Bug 2045586] Re: livecd-rootfs uses losetup -P for theoretically reliable/synchronous partition setup but it's not reliable

2024-02-05 Thread dann frazier
While the above MP is now merged, I still see additional potential races in the code. For example, anything calling mount_partition() for the first partition, and also maybe bug 2030771. I don't see a good way to solve that w/o `udevadm lock` because these functions don't currently know what the fu

[Kernel-packages] [Bug 2045586] Re: livecd-rootfs uses losetup -P for theoretically reliable/synchronous partition setup but it's not reliable

2024-02-03 Thread dann frazier
** Also affects: util-linux (Ubuntu Mantic) Importance: Undecided Status: New ** Also affects: livecd-rootfs (Ubuntu Mantic) Importance: Undecided Status: New ** Also affects: linux (Ubuntu Mantic) Importance: Undecided Status: New ** Also affects: util-linux (Ubunt

[Kernel-packages] [Bug 2045586] Re: livecd-rootfs uses losetup -P for theoretically reliable/synchronous partition setup but it's not reliable

2024-01-31 Thread dann frazier
The test passed w/ flock as well: https://autopkgtest.ubuntu.com/results/autopkgtest-noble-dannf-loop/noble/amd64/l/livecd-rootfs/20240131_104633_df869@/log.gz 274 successful iterations before the timeout killed it. I've updated the MP: https://code.launchpad.net/~dannf/livecd-rootfs/+git/l

[Kernel-packages] [Bug 2045586] Re: livecd-rootfs uses losetup -P for theoretically reliable/synchronous partition setup but it's not reliable

2024-01-31 Thread Launchpad Bug Tracker
** Merge proposal linked: https://code.launchpad.net/~dannf/livecd-rootfs/+git/livecd-rootfs/+merge/459549 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2045586 Title: livecd-rootfs

Re: [Kernel-packages] [Bug 2045586] Re: livecd-rootfs uses losetup -P for theoretically reliable/synchronous partition setup but it's not reliable

2024-01-30 Thread dann frazier
On Tue, Jan 30, 2024 at 6:21 PM Michael Hudson-Doyle <2045...@bugs.launchpad.net> wrote: > > Oh wait, we've been through something very like this before > https://bugs.launchpad.net/cloud-initramfs-tools/+bug/1834875. I suspect > a judicious application of flock may be the most correct solution > a

[Kernel-packages] [Bug 2045586] Re: livecd-rootfs uses losetup -P for theoretically reliable/synchronous partition setup but it's not reliable

2024-01-30 Thread Michael Hudson-Doyle
Oh wait, we've been through something very like this before https://bugs.launchpad.net/cloud-initramfs-tools/+bug/1834875. I suspect a judicious application of flock may be the most correct solution available. -- You received this bug notification because you are a member of Kernel Packages, whic

[Kernel-packages] [Bug 2045586] Re: livecd-rootfs uses losetup -P for theoretically reliable/synchronous partition setup but it's not reliable

2024-01-29 Thread dann frazier
I updated my livecd-rootfs PPA test package that runs this section of code in a loop to use this pattern, and it survived until the autopkgtest timeout - 304 iterations: https://autopkgtest.ubuntu.com/results/autopkgtest-noble-dannf- loop/noble/amd64/l/livecd-rootfs/20240128_061640_5c909@/log.gz

[Kernel-packages] [Bug 2045586] Re: livecd-rootfs uses losetup -P for theoretically reliable/synchronous partition setup but it's not reliable

2024-01-27 Thread dann frazier
I was bothered by the fact that we only sometimes see the double partition rescan in dmesg. If udev always rescans the partitions of a full block devices, shouldn't we always see those messages twice? It turns out that udev doesn't rescan partitions just because a new block device appears. When it

[Kernel-packages] [Bug 2045586] Re: livecd-rootfs uses losetup -P for theoretically reliable/synchronous partition setup but it's not reliable

2024-01-25 Thread dann frazier
That kernel change doesn't fix the issue: https://autopkgtest.ubuntu.com/results/autopkgtest-noble-dannf- loop/noble/amd64/l/livecd-rootfs/20240125_203808_8b5c9@/log.gz Which actually didn't surprise me after thinking about it. systemd-udevd is going to ask for a partition reread when it gets the

[Kernel-packages] [Bug 2045586] Re: livecd-rootfs uses losetup -P for theoretically reliable/synchronous partition setup but it's not reliable

2024-01-24 Thread Launchpad Bug Tracker
This bug was fixed in the package livecd-rootfs - 24.04.21 --- livecd-rootfs (24.04.21) noble; urgency=medium * live-build/functions: avoid losetup -P as it appears to race with udev and do it a bit more by-hand instead. (LP: #2045586) -- Michael Hudson-Doyle Thu, 25 Jan 202

[Kernel-packages] [Bug 2045586] Re: livecd-rootfs uses losetup -P for theoretically reliable/synchronous partition setup but it's not reliable

2024-01-24 Thread Michael Hudson-Doyle
Here's my workaround then https://code.launchpad.net/~mwhudson/livecd- rootfs/+git/livecd-rootfs/+merge/459380 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2045586 Title: livecd-rootfs

[Kernel-packages] [Bug 2045586] Re: livecd-rootfs uses losetup -P for theoretically reliable/synchronous partition setup but it's not reliable

2024-01-24 Thread Launchpad Bug Tracker
** Merge proposal linked: https://code.launchpad.net/~mwhudson/livecd-rootfs/+git/livecd-rootfs/+merge/459380 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2045586 Title: livecd-roo

Re: [Kernel-packages] [Bug 2045586] Re: livecd-rootfs uses losetup -P for theoretically reliable/synchronous partition setup but it's not reliable

2024-01-23 Thread dann frazier
On Tue, Jan 23, 2024 at 8:55 PM Michael Hudson-Doyle <2045...@bugs.launchpad.net> wrote: > > Amazing debugging Dann. Until we can get a kernel fix, what's the way > forward here? Run losetup without -P, run udevadm settle, run partprobe > on the device (then maybe run udevadm settle again??) That

[Kernel-packages] [Bug 2045586] Re: livecd-rootfs uses losetup -P for theoretically reliable/synchronous partition setup but it's not reliable

2024-01-23 Thread Michael Hudson-Doyle
Amazing debugging Dann. Until we can get a kernel fix, what's the way forward here? Run losetup without -P, run udevadm settle, run partprobe on the device (then maybe run udevadm settle again??) -- You received this bug notification because you are a member of Kernel Packages, which is subscribe

[Kernel-packages] [Bug 2045586] Re: livecd-rootfs uses losetup -P for theoretically reliable/synchronous partition setup but it's not reliable

2024-01-23 Thread dann frazier
** Summary changed: - livecd-rootfs uses losetup -P for theoretically reliable/synchronous partition setup but it's not reliable in noble + livecd-rootfs uses losetup -P for theoretically reliable/synchronous partition setup but it's not reliable -- You received this bug notification because y

[Kernel-packages] [Bug 2045586] Re: livecd-rootfs uses losetup -P for theoretically reliable/synchronous partition setup but it's not reliable in noble

2024-01-22 Thread dann frazier
I ran the above test: https://autopkgtest.ubuntu.com/results/autopkgtest-jammy-dannf-test/jammy/amd64/l/livecd-rootfs/20240123_035147_6470b@/log.gz It does appear that systemd-udevd is trying to scan partitions at the same time as losetup: 1599s ++ losetup --show -f -P -v binary/boot/disk-uefi

[Kernel-packages] [Bug 2045586] Re: livecd-rootfs uses losetup -P for theoretically reliable/synchronous partition setup but it's not reliable in noble

2024-01-22 Thread dann frazier
I ran into this on jammy/amd64: https://autopkgtest.ubuntu.com/results/autopkgtest- jammy/jammy/amd64/l/livecd-rootfs/20240121_173406_e4f9a@/log.gz I downloaded all of the amd64 failures and searched for this failure pattern. These were the kernels that were running at the time: "Linux 5.15.0-91-

Re: [Kernel-packages] [Bug 2045586] Re: livecd-rootfs uses losetup -P for theoretically reliable/synchronous partition setup but it's not reliable in noble

2023-12-11 Thread Michael Hudson-Doyle
> Is only asking kernel to scan the device; to then generate "kernel udev" > events; for then udev to wakeup and process/emit "udev udev" events; and > create the required device nodes. > It's not udev that creates nodes like /dev/loop1p1 though is it? That's devtmpfs surely. -- You received thi

[Kernel-packages] [Bug 2045586] Re: livecd-rootfs uses losetup -P for theoretically reliable/synchronous partition setup but it's not reliable in noble

2023-12-11 Thread Emil Renner Berthing
I don't have a good explanation, but in the past I've "fixed" such races by adding a `sync "$loop_device"` before using any of the newly created partitons. Maybe it's worth trying. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in U

[Kernel-packages] [Bug 2045586] Re: livecd-rootfs uses losetup -P for theoretically reliable/synchronous partition setup but it's not reliable in noble

2023-12-11 Thread Dimitri John Ledkov
my expectation is that udev should be running (somewhere, not sure if it needs to be both the host and the lxd guest) and that it should process the device using locks https://systemd.io/BLOCK_DEVICE_LOCKING/. After that is done, the device should be safe to operate on, in a consistent manner. Af

[Kernel-packages] [Bug 2045586] Re: livecd-rootfs uses losetup -P for theoretically reliable/synchronous partition setup but it's not reliable in noble

2023-12-09 Thread Steve Langasek
Oh. To the question of whether there was a systemd change in this window: yes absolutely, because this is the point at which the riscv64 builders moved from lgw manually-operated qemu with a 20.04 guest image, to bos03 openstack-operated qemu with a 22.04 guest image. Which is also why we've move

Re: [Kernel-packages] [Bug 2045586] Re: livecd-rootfs uses losetup -P for theoretically reliable/synchronous partition setup but it's not reliable in noble

2023-12-09 Thread Steve Langasek
On Sat, Dec 09, 2023 at 05:13:28PM -, Andy Whitcroft wrote: > Was there any systemd/udev change in this timeframe? As the device > files are very much connected to those. My understanding is that these devices are supposed to be created directly by the kernel on devtmpfs and NOT via udev, whi

[Kernel-packages] [Bug 2045586] Re: livecd-rootfs uses losetup -P for theoretically reliable/synchronous partition setup but it's not reliable in noble

2023-12-09 Thread Andy Whitcroft
Was there any systemd/udev change in this timeframe? As the device files are very much connected to those. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2045586 Title: livecd-rootfs us

[Kernel-packages] [Bug 2045586] Re: livecd-rootfs uses losetup -P for theoretically reliable/synchronous partition setup but it's not reliable in noble

2023-12-08 Thread Steve Langasek
https://launchpad.net/~ubuntu- cdimage/+livefs/ubuntu/noble/cpc/+build/544490 is a log from a build with a new livecd-rootfs that spits out more debugging info on failure. + sgdisk binary/boot/disk-uefi.ext4 --print Disk binary/boot/disk-uefi.ext4: 9437184 sectors, 4.5 GiB Sector size (logical): 5

[Kernel-packages] [Bug 2045586] Re: livecd-rootfs uses losetup -P for theoretically reliable/synchronous partition setup but it's not reliable in noble

2023-12-04 Thread Steve Langasek
Failing build had kernel Kernel version: Linux bos03-riscv64-014 5.19.0-1021-generic #23~22.04.1-Ubuntu SMP Thu Jun 22 12:49:35 UTC 2023 riscv64 The build immediately before the first failure had kernel Kernel version: Linux riscv64-qemu-lgw01-069 5.13.0-1019-generic #21~20.04.1-Ubuntu SMP Thu M

[Kernel-packages] [Bug 2045586] Re: livecd-rootfs uses losetup -P for theoretically reliable/synchronous partition setup but it's not reliable in noble

2023-12-04 Thread Steve Langasek
November 16 was 2 days after livecd-rootfs 24.04.4 landed in the noble release pocket, superseding 24.04.2. The code delta between 24.04.2 and 24.04.4 includes removal of support for "legacy" images (SUBPROJECT=legacy) which doesn't apply here; and some reorganization of code related to "preinstal