[Bug 2056461] Re: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown
I can confirm the "kip1" kernel from the PPA does _not_ fix the issue for me. Here is a log excerpt, with the kernel version visible: autopkgtest [17:32:41]: testbed running kernel: Linux 6.8.0-35-generic #35ubuntu1+kip1-Ubuntu SMP PREEMPT_DYNAMIC Wed Jun 19 02:36:12 U autopkgtest: DBG: testbed command ['sh', '-c', 'nproc; cat /proc/cpuinfo 2>/dev/null || true'], kind short, sout pipe, serr pipe, env [] autopkgtest: DBG: testbed command exited with code 0 autopkgtest: DBG: Binaries: initialising autopkgtest [17:32:41]: unbuilt-tree /home/paride/debian/git/autopkgtest/ autopkgtest: DBG: blame += /home/paride/debian/git/autopkgtest/ autopkgtest: DBG: testbed reset: modified=False, deps_installed=[], deps_new=[] autopkgtest: DBG: testbed command ['mkdir', '-p', '/tmp/autopkgtest.HkRABQ'], kind short, sout raw, serr pipe, env [] autopkgtest: DBG: testbed command exited with code 0 autopkgtest: DBG: sending command to testbed: copydown /home/paride/debian/git/autopkgtest// /tmp/autopkgtest.HkRABQ/ubtree-/ autopkgtest: DBG: got reply from testbed: timeout -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056461 Title: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/2056461/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056461] Re: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown
I didn't have time to investigate further, but I encountered this hang again this morning with an Oracular image, with the PPA kernel (autopkgtest shows this version: Linux 6.8.0-35-generic #35ubuntu1+kip1-Ubuntu), and was still able to unlock the situation with the `stat` trick mentioned above. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056461 Title: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/2056461/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056461] Re: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown
Launchpad has imported 21 comments from the remote bug at https://bugzilla.kernel.org/show_bug.cgi?id=218916. If you reply to an imported comment from within Launchpad, your comment will be sent to the remote bug automatically. Read more about Launchpad's inter-bugtracker facilities at https://help.launchpad.net/InterBugTracking. On 2024-05-31T08:56:20+00:00 linux wrote: Reporting here to track a regression apparently introduced when 9p switched to netfslib in commit 80105ed2fd2715fb09a8fdb0655a8bdc86c120db "9p: Use netfslib read/write_iter" that is being reported on mailing list and in distros: "Re: [PATCH v5 40/40] 9p: Use netfslib read/write_iter" https://lore.kernel.org/all/Zj0ErxVBE3DYT2Ea@gpd/ "linux: regression in the 9p protocol in 6.8 breaks autopkgtest qemu jobs (affecting debci)" https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1072004 "autopkgtest-virt-qemu on noble images sometimes hangs doing copydown" https://bugs.launchpad.net/ubuntu/+source/autopkgtest/+bug/2056461 I've been digging into the bug and collecting data 9p developers may better understand. Host is Debian 12 Bookworm amd64 building the example Debian package "hello" and running autopkgtest in a QEMU virtual machine with Debian unstable (sid) and local build of latest mainline kernel (currently 4a4be1ad3a6). The bug was first reported against v6.8. I was hoping to create a 'simple' reproducer without the complexity of autopkgtest but first need to understand what is going wrong. After 31 iterations I believe I may finally have found a clue. It appears (in the guest OS of course) v9fs_file_read_iter() doesn't consume the complete content of the host file being shared via 9p. In the reproducer the host autopkgtest Python scripts control qemu- system-x86 and send commands to the guest OS for it to execute. These include dynamically writing and then executing a python script named "eofcat" [0] that is used as a "funnel" for stdin [1]. "eofcat" spins on a non-blocking read of its fd 0 until a specific file exists or the number of bytes read from fd 0 is >= to a limit value read from the file name passed in as argv[1]. By modifying the guest's mount -t 9p command [2] options to include "debug=0x08" and using "| ts %.T |" in logging pipelines to capture microsecond accurate timestamps I was able to identify the guest kernel log entries when autopkgtest 'hangs' (spins). What I see is the host file "stdin" being redirected to "eofcat" fd 0 and guest kernel shows a series of v9fs_file_read_iter(count=100, ...) calls with "offset" increasing each time: 0, 100, 200, 300, 400, 500, 5416960. At that point it starts spinning - repeatedly retrying a v9fs_file_read_iter(count=100, offset=5416960) and a single read of "stdin_eof" (to get the number of bytes to expect - the "limit" variable) - and that has 8253440 which is also the size of "stdin" file on the host. In the guest a stat or ls or tail of the shared "stdin" file will break the deadlock and operations continue. No 9p cache options are in use. In following comments I'll attach run logs of the host autopkgtest and the guest kernel dmesg with microsecond precision timestamps with mount option debug=0x0f ( P9_DEBUG_VFS | P9_DEBUG_9P | DEBUG_CURRENT | P9_DEBUG_ERROR ) with the exact timestamp where the host hangs, which can then be found in the guest kernel log. [0] https://sources.debian.org/src/autopkgtest/5.35/virt/autopkgtest- virt-qemu/?hl=320#L314 [1] https://sources.debian.org/src/autopkgtest/5.35/virt/autopkgtest- virt-qemu/?hl=492#L487 [2] https://sources.debian.org/src/autopkgtest/5.35/virt/autopkgtest- virt-qemu/#L297 Reply at: https://bugs.launchpad.net/ubuntu/+source/autopkgtest/+bug/2056461/comments/16 On 2024-05-31T09:24:18+00:00 linux wrote: Interesting finding: with debug=0x0f the bug is not triggered - I have to reduce the debug flags to trigger it. Currently working up from P9_DEBUG_VFS: With run #34 and debug=0x08 (P9_DEBUG_VFS) I captured logs (attached). Hang occurs in autopkgtest at timestamp 10:08:37.390685 (UTC+1). In the guest kernel log that maps to timestamp 09:08:37.408129. The first read on fd 0 is: 09:08:37.481590 v9fs_file_read_iter (1049): fid 6 count 100 offset 0 followed by subsequent reads: 09:08:37.517808 v9fs_file_read_iter (1049): fid 6 count 100 offset 100 ... 09:08:37.565864 v9fs_file_read_iter (1049): fid 6 count 100 offset 200 ... 09:08:37.609522 v9fs_file_read_iter (1049): fid 6 count 100 offset 300 ... 09:08:37.653503 v9fs_file_read_iter (1049): fid 6 count 100 offset 400 ... 09:08:37.693647 v9fs_file_read_iter (1049): fid 6 count 100 offset 500 ... 09:08:37.716325 v9fs_file_read_iter (1049): fid 6 count 100 offset 5275648 At which point it starts 'spinning' waiting for more data. On the host we
[Bug 2056461] Re: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown
** Changed in: autopkgtest (Debian) Status: Unknown => New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056461 Title: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/2056461/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056461] Re: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown
** Bug watch added: Debian Bug tracker #1071456 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1071456 ** Changed in: autopkgtest (Debian) Status: Fix Committed => Unknown ** Changed in: autopkgtest (Debian) Remote watch: Debian Bug tracker #1068363 => Debian Bug tracker #1071456 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056461 Title: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/2056461/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056461] Re: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown
** Changed in: autopkgtest (Debian) Status: Unknown => Fix Committed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056461 Title: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/2056461/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056461] Re: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown
** Also affects: linux via https://bugzilla.kernel.org/show_bug.cgi?id=218916 Importance: Unknown Status: Unknown ** Also affects: autopkgtest (Debian) via https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1068363 Importance: Unknown Status: Unknown -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056461 Title: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown To manage notifications about this bug go to: https://bugs.launchpad.net/linux/+bug/2056461/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 2056461] Re: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown
-- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056461 Title: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/autopkgtest/+bug/2056461/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056461] Re: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown
I added the correct fix to the upstream bug report last weekend (8 June): https://bugzilla.kernel.org/show_bug.cgi?id=218916 Waiting for a response from David Howells as to whether there is another, lower-level, fix in netfs rather than 9p itself. ** Bug watch added: Linux Kernel Bug Tracker #218916 https://bugzilla.kernel.org/show_bug.cgi?id=218916 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056461 Title: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/autopkgtest/+bug/2056461/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056461] Re: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown
The Debian ROCm Team has also seen this in their CI instance. One data point that might be relevant is that in our case, all our failures were limited to autopkgtests where there was a reboot between tests. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056461 Title: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/autopkgtest/+bug/2056461/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056461] Re: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown
** Description changed:
+ [Impact]
+
+ It seems that kernel 6.8 introduced a regression in the 9pfs related to
+ caching and netfslib, that can cause some user-space apps to read
+ content from files that is not up-to-date (when they are used in a
+ producer/consumer fashion).
+
+ It seems that the offending commit is this one:
+
+ 80105ed2fd27 ("9p: Use netfslib read/write_iter")
+
+ Reverting the commit seems to fix the problem. However the actual bug
+ might be in netfslib or how netfslib is used in the 9p context.
+
+ The regression has been reported upstream and we are still investigating
+ (https://lore.kernel.org/lkml/Zj0ErxVBE3DYT2Ea@gpd/).
+
+ In the meantime it probably makes sense to temporarily revert the commit
+ as a SAUCE patch. Then we will drop the SAUCE patch once we'll have a
+ proper fix upstream.
+
+ [Test case]
+
+ The following test should complete correctly without any timeout:
+
+ pull-lp-source -d hello
+ autopkgtest-buildvm-ubuntu-cloud -r noble
+ autopkgtest -U hello*.dsc -- qemu ./autopkgtest-noble-amd64.img
+
+
+ [Fix]
+
+ Revert the following commit (until we have a proper fix upstream):
+
+ 80105ed2fd27 ("9p: Use netfslib read/write_iter")
+
+ [Regression potential]
+
+ We may experience other regressions related to 9pfs with this change,
+ however it's quite unlikely to happen since we are reverting a commit,
+ restoring the previous behavior.
+
+ [Original bug report]
+
autopkgtest-virt-qemu sometimes hangs when running tests on noble
images. Originally reported by schopin, who also provided a reproducer:
- pull-lp-source -d hello
- autopkgtest-buildvm-ubuntu-cloud -r noble
- autopkgtest -U hello*.dsc -- qemu ./autopkgtest-noble-amd64.img
+ pull-lp-source -d hello
+ autopkgtest-buildvm-ubuntu-cloud -r noble
+ autopkgtest -U hello*.dsc -- qemu ./autopkgtest-noble-amd64.img
I've been able to reproduce it with debugging enabled:
- autopkgtest -ddd -U hello_2.10-3.dsc -- qemu --debug --show-boot
+ autopkgtest -ddd -U hello_2.10-3.dsc -- qemu --debug --show-boot
/path/to/image
It can get stuck during different stages, but AFAICT always during
"copydown" operations, log excerpts follow. It may be a coincidence, but
this started happening around the time linux-image-6.8.0-11-generic
(6.8.0-11.11) migrated to noble. The testbeds I used booted 6.6 but then
rebooted into that 6.8 kernel after being upgraded by autopkgtest.
-- logs --
Removing autopkgtest-satdep (0) ...
[...]
autopkgtest-virt-qemu: DBG: executing copydown
/tmp/autopkgtest.output.g8v75e8g/tests-tree/ /t/
autopkgtest-virt-qemu: DBG: ['cmdls', "(['tar', '--directory',
'/tmp/autopkgtest.output.g8v75e]
autopkgtest-virt-qemu: DBG: ['srcstdin', "<_io.BufferedReader
name='/dev/null'>", 'deststdout']
autopkgtest-virt-qemu: DBG: +< tar --directory
/tmp/autopkgtest.output.g8v75e8g/tests-tree/ --
autopkgtest-virt-qemu: DBG: +> /tmp/autopkgtest-qemu.ztmr6f5k/runcmd sh -ec
if ! test -d /tmp-
autopkgtest-virt-qemu: DBG: +>?
-- or --
autopkgtest: DBG: sending command to testbed: copydown
/tmp/autopkgtest.output.c9utq3bx/tests-tree/
/tmp/autopkgtest.H8NDfW/build.DLR/src/
autopkgtest-virt-qemu: DBG: executing copydown
/tmp/autopkgtest.output.c9utq3bx/tests-tree/
/tmp/autopkgtest.H8NDfW/build.DLR/src/
autopkgtest-virt-qemu: DBG: ['cmdls', "(['tar', '--directory',
'/tmp/autopkgtest.output.c9utq3bx/tests-tree/', '--warning=none', '-c', '.',
'-f', '-'], ['/tmp/autopkgtest-qemu.qtkcgg5l/runcm]
autopkgtest-virt-qemu: DBG: ['srcstdin', "<_io.BufferedReader
name='/dev/null'>", 'deststdout', "<_io.BufferedReader name='/dev/null'>",
'devnull_read', <_io.BufferedReader name='/dev/null'>]
autopkgtest-virt-qemu: DBG: +< tar --directory
/tmp/autopkgtest.output.c9utq3bx/tests-tree/ --warning=none -c . -f -
autopkgtest-virt-qemu: DBG: +> /tmp/autopkgtest-qemu.qtkcgg5l/runcmd sh -ec
if ! test -d /tmp/autopkgtest.H8NDfW/build.DLR/src/; then mkdir --
/tmp/autopkgtest.H8NDfW/build.DLR/src/; fi; cd-
autopkgtest-virt-qemu: DBG: +>?
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2056461
Title:
autopkgtest-virt-qemu on noble images sometimes hangs doing copydown
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/autopkgtest/+bug/2056461/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056461] Re: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown
I tested the the 6.8.0-32.32+lp2056461 kernel and it appears to reliably fix the problem. I tested it several times making autopkgtest 'copydown' a large source tree, and also verified that using an image with the standard kernel I did hit the problem. Thanks Andrea for doing the bisect! -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056461 Title: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/autopkgtest/+bug/2056461/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056461] Re: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown
** Changed in: linux (Ubuntu) Status: New => In Progress ** Changed in: linux (Ubuntu) Assignee: (unassigned) => Andrea Righi (arighi) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056461 Title: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/autopkgtest/+bug/2056461/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056461] Re: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown
I've uploaded a test kernel here:
https://people.canonical.com/~arighi/lp2056461/amd64/
With this one I can't reproduce the problem. It'd be great if someone
else could test it as well, to confirm that it's actually fixing the
problem.
Note: this kernel is a 6.8.0-32.32 with this commit reverted:
commit 80105ed2fd27 ("9p: Use netfslib read/write_iter")
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2056461
Title:
autopkgtest-virt-qemu on noble images sometimes hangs doing copydown
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/autopkgtest/+bug/2056461/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056461] Re: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown
I've done some bisecting and it looks like the culprit is this commit: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=80105ed2fd2715fb09a8fdb0655a8bdc86c120db I'll prepare a test kernel with this one reverted, then if it's confirmed, we can figure out a proper fix. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056461 Title: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/autopkgtest/+bug/2056461/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056461] Re: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown
Latest findings: * The deepest loop that actually never exits is actually the `while not os.path.exists(fexit):` in the `eofcat` script. This is because the `block = os.read(0, 100)` always reads nothing. * From the guest: * `head`ing the `stdin` file doesn't help. * `tail`ing the `stdin` file does help, it unblocks the loop, just the same way as `stat`ing it. * From the host: * `stat`ing `stdin` doesn't change anything. * Appending more data to it doesn't change anything either. * This adds up with the fact that it's a kernel change in the guest that triggered the bug. * That `stdin` file is stored on a 9p shared FS. This might be a related change, but nothing confirmed yet: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=c9b93cafb69cbbbe375de29c1ebf410dbc33ebfc -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056461 Title: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/autopkgtest/+bug/2056461/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056461] Re: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown
** Also affects: linux (Ubuntu) Importance: Undecided Status: New ** Tags added: rls-nn-incoming -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056461 Title: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/autopkgtest/+bug/2056461/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056461] Re: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown
AIUI that's an issue with a test which expects a timeout, and fails when things happen too fast. At first glance it doesn't seem to be related. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056461 Title: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/autopkgtest/+bug/2056461/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056461] Re: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown
Could this be related? https://bugs.debian.org/cgi- bin/bugreport.cgi?bug=1068363 ** Bug watch added: Debian Bug tracker #1068363 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1068363 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056461 Title: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/autopkgtest/+bug/2056461/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056461] Re: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown
Confirmed working: the `stat` immediately unblocks the copydown. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056461 Title: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/autopkgtest/+bug/2056461/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056461] Re: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown
Can confirm the stat workaround works for me as well -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056461 Title: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/autopkgtest/+bug/2056461/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056461] Re: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown
Okay, I've spend the last 24h digging into that issue, and here are my findings: * I confirm I can only reproduce the issue with 6.8 kernels, not with 6.5. Just to be sure, I've also tested 6.8.0-32.32 in kernel team's PPA, and it has the issue too. * Changing `--cpus` or `--ram-size` in `autopkgtest-virt-qemu` doesn't change anything. * The code responsible for this in autopkgtest is a wonderful piece of black magic: most lives in `virt/autopkgtest-virt-qemu`, and the main part is the `runcmd` script being crafted in the `make_auxverb` function. The hanging is the `# wait until command has exited` loop, that never exits. The most funny part is the new workaround that I found. Run the following from your host that runs `autopkgtest`. Obvious change the port number depending on which free port was available for `autopkgtest`. ``` ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -p 10022 ubuntu@localhost stat /run/autopkgtest/shared/job\*/stdin ``` For some reason, `stat`ing that file in the testbed makes the loop exit and everything continues as normal, meaning you can use the test results normally. I've seen this work at least 20 times on my machine, but I'd like someone else to confirm that this workaround indeed works. This bug is becoming more and more interesting, stay tuned for more adventures! -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056461 Title: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/autopkgtest/+bug/2056461/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056461] Re: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown
I'm hitting this as well, running autopkgtest (from latest main) against an amd64 noble image built with autopkgtest-buildvm-ubuntu-cloud on Debian stable with qemu 7.2. I can confirm that switching the kernel in the noble image to the 6.5 shipped in Mantic fixes the issue, with no other changes, so it does seem like a problem introduced by the recent noble kernel update. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056461 Title: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/autopkgtest/+bug/2056461/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056461] Re: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown
For anyone encountering this you can use a different virt server other than qemu to workaround the issue -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056461 Title: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/autopkgtest/+bug/2056461/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056461] Re: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown
Looks like I'm facing the same issue: /testbed-packages autopkgtest-virt-qemu: DBG: +> cat autopkgtest-virt-qemu: DBG: +>? autopkgtest-virt-qemu: DBG: +", 'deststdout', "<_io.BufferedReader name='/dev/null'>", 'devnull_read', <_io.BufferedReader name='/dev/null'>] autopkgtest-virt-qemu: DBG: +< tar --directory lxc-5.0.3// --warning=none -c . -f - autopkgtest-virt-qemu: DBG: +> /tmp/autopkgtest-qemu.1i2y4v3q/runcmd sh -ec if ! test -d /tmp/autopkgtest.5ldgRV/ubtree-/; then mkdir -- /tmp/autopkgtest.5ldgRV/ubtree-/; fi; cd /tmp/autopkgtest.5ldgRV/ubtree-/; tar --warning=none --preserve-permissions --extract --no-same-owner -f - autopkgtest-virt-qemu: DBG: +>? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056461 Title: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/autopkgtest/+bug/2056461/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056461] Re: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown
I've discovered this as well. For me, copydown works perfectly fine for individual files, but fails for directories. My reproducer is as follows: ./autopkgtest/runner/autopkgtest -d -o /tmp/autopkgtest-astra-toolbox8/ --apt-pocket=proposed=src:nvidia-cuda-toolkit astra-toolbox/ --env=ADT_TEST_TRIGGERS=nvidia-cuda-toolkit/12.0.1-4build4 -- qemu /home/andersson123/canonical/code/autopkgtest-noble-amd64.img with failures as follows: autopkgtest: DBG: sending command to testbed: copydown astra-toolbox// /tmp/autopkgtest.F/ autopkgtest-virt-qemu: DBG: executing copydown astra-toolbox// /tmp/autopkgtest.FixwAf/tr/ autopkgtest-virt-qemu: DBG: ['cmdls', "(['tar', '--directory', 'astra-toolbox//', '--warn] autopkgtest-virt-qemu: DBG: ['srcstdin', "<_io.BufferedReader name='/dev/null'>", 'destst] autopkgtest-virt-qemu: DBG: +< tar --directory astra-toolbox// --warning=none -c . -f - autopkgtest-virt-qemu: DBG: +> /tmp/autopkgtest-qemu.d2mlr22y/runcmd sh -ec if ! test -d- autopkgtest-virt-qemu: DBG: +>? ** Changed in: autopkgtest (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056461 Title: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/autopkgtest/+bug/2056461/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2056461] Re: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown
Communication (including copydown()) with the qemu virt server happens via the serial terminal. That isn't the most solid approach, and I suspect it's racy for some reason. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2056461 Title: autopkgtest-virt-qemu on noble images sometimes hangs doing copydown To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/autopkgtest/+bug/2056461/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
