** Description changed:
+ [ Impact ]
+
+ * libvirtd can hang on shutdown due to a deadlock in the node device
+ driver: nodeStateShutdownWait() holds the driver object lock while
+ joining the udev event thread, and that thread needs the same lock to
+ observe the quit flag and exit. Neither can proceed, so systemd waits
+ for TimeoutStopSec (30s) and then SIGKILLs libvirtd.
+
+ * It is a race (about 4% of shutdowns on Intel Panther Lake / Bartlett
+ Lake systems running Ubuntu 26.04), which delays shutdown/reboot and
+ breaks S4/S5 power-cycle validation.
+
+ * The fix (upstream commit 0c053ffd2c44, in v12.8.0) no longer holds the
+ lock while joining the thread. It applies to libvirt and libvirt-hwe in
+ Resolute, Stonking and Tenacious.
+
+ [ Test Plan ]
+
+ * The race is hard to hit, so reproduce by repeating the shutdown
+ scenario. Install the package under test and run:
+
+ sudo apt install -y libvirt-daemon-system libvirt-clients psmisc
+ sudo systemctl enable --now libvirtd.socket libvirtd.service
+ for i in $(seq 1 200); do
+ sudo systemctl start libvirtd.service
+ virsh -c qemu:///system list >/dev/null 2>&1 &
+ sudo systemctl kill -s SIGTERM libvirtd.service
+ start=$(date +%s)
+ while systemctl is-active --quiet libvirtd.service; do sleep 0.2; [ $((
$(date +%s) - start )) -gt 20 ] && { echo "HANG at iteration $i"; break 2; };
done
+ done
+ echo done
+
+ * Before the fix: a "HANG" is reported on some iterations (libvirtd stays
+ alive after SIGTERM; "gdb -p $(pidof libvirtd) -batch -ex 'thread apply
+ all bt'" shows daemon-shutdown joining udev-event in nodedev).
+
+ * After the fix: no hang in any iteration.
+
+ [ Where problems could occur ]
+
+ * The change is limited to nodeStateShutdownWait() in
+ src/node_device/node_device_udev.c: the udev thread is now joined without
+ holding the driver object lock.
+
+ * Possible regression: a race on priv->udevThread access during shutdown.
+ Upstream's analysis is that the field is only assigned at driver
+ initialisation and released on dispose, and the udev thread holds a
+ reference on priv, so the lock-free read is safe.
+
+ * Failure modes would be limited to libvirtd shutdown/restart (hang or
+ crash), and the nodedev driver (virsh nodedev-*).
+
+ [ Other Info ]
+
+ * Upstream commit:
https://gitlab.com/libvirt/libvirt/-/commit/0c053ffd2c44ef0820886ab79c1f812775b2cd15
+ * Introduced in v10.0.5; Noble (10.0.0) is not affected.
+ * Fixed upstream in v12.8.0; Tenacious/Stonking has 12.6.0-1ubuntu4,
Resolute has
+ 12.0.0-1ubuntu5.6 in -updates (12.0.0-1ubuntu5.7 in -proposed).
+
+ --- Original bug report
+
[Summary]
We found a libvirt bug during Ubuntu 26.04 upgrade. Since this issue only
happens on Ubuntu 26.04 and not found on Ubuntu 24.04, attached is our upstream
fix and seek help from Canonical to backport this fix ASAP to unblock this
ubuntu 26.04 upgrade issue in this 2 edge PTL & BTL platform.
[Steps to reproduce]
During device shutdown with Ubuntu 26.04 and libvirt(12.0.0) installed, we
see systemd wait 30s(default timeout) and then force kill libvirtd. This
happens randomly, about 4% chance to hit. This is libvirt bug(hang due to race
condition).
Logs as below.
[ ***] (2 of 2) Job libvirtd.service/stop running (18s / 30s)
[ **] (2 of 2) Job libvirtd.service/stop running (19s / 30s)
...
[*** ] (1 of 2) Job libvirt-guests.service/stop running (29s / no limit)
[* ] (1 of 2) Job libvirt-guests.service/stop running (30s / no limit)
[ OK ] Stopped libvirtd.service - libvirt legacy monolithic daemon.
[ 82.376607] libvirt-guests.sh[5468]: Can't connect to default. Skipping.
[ OK ] Stopped libvirt-guests.service - libvirt guests suspend/resume
service.
Fix
- Fix patch has been submitted to libvirt mail group. Patch has been accepted
and merged:
https://gitlab.com/libvirt/libvirt/-/commit/0c053ffd2c44ef0820886ab79c1f812775b2cd15
+ Fix patch has been submitted to libvirt mail group. Patch has been accepted
and merged:
https://gitlab.com/libvirt/libvirt/-/commit/0c053ffd2c44ef0820886ab79c1f812775b2cd15
But since this is master branch later than 12.0.0, need Canonical team to
evaluate how to mitigate this issue on Ubuntu 26.04.
Root Cause Analysis
There is a libvirt bug introduced in v10.0.5. Ubuntu 24.04 using v10.0.0 and
26.04 using v12.0.0. During our debug, we found a race condition in libvirt:
Sending SIGTERM to libvirtd while a client concurrently connects through the
activation socket reproduces the hang. Captured state of a hung daemon:
tid comm wchan
14039 libvirtd poll_schedule_timeout <- main event loop, still running
14108 udev-event futex_do_wait
14165 daemon-shutdown futex_do_wait
Thread "daemon-shutdown" (LWP 14165):
#5 __pthread_clockjoin_ex ()
#6 libvirt_driver_nodedev.so
#7 virStateShutdownWait () from libvirt.so.0
Thread "udev-event" (LWP 14108):
#1 __GI___lll_lock_wait (futex=0x7d55d001cc58)
#2 __pthread_mutex_cond_lock (mutex=0x7d55d001cc58)
#4 ___pthread_cond_wait ()
#5 virCondWait () from libvirt.so.0
#6 libvirt_driver_nodedev.so
The contended `pthread_mutex_t` names its owner:
(gdb) x/4dw 0x7d55d001cc58
0x7d55d001cc58: 2 0 14165 2
^ ^ ^ ^
__lock | __owner __nusers
__count
__owner = 14165 is the daemon-shutdown thread itself. So:
1. daemon-shutdown holds the nodedev driver's object lock and is blocked
in pthread_join() waiting for `udev-event` to exit.
2. udev-event is inside virCondWait(), blocked trying to re-acquire that
same lock so it can return from the wait and observe the quit flag.
Neither can proceed. The daemon never exits; systemd escalates to SIGKILL at
TimeoutStopSec.
-
Failed run screenshot:
Refer above log.
Passed run screenshot:
-
[Expected result]
[Actual result]
-
[Failure rate]
[Affected test cases]
[Additional information]
CID:
SKU: PTL & BTL
system-manufacturer: Intel Corporation
system-product-name: Panther Lake & Bartlett Lake platform
bios-version:
Ubuntu version: 26.04
Kernel: 6.18 & 7.0
BIOS: PTL & BTL BIOS-- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2168017 Title: [PTL][BTL]libvirt random hang issue on Ubuntu 26.04 To manage notifications about this bug go to: https://bugs.launchpad.net/kobuk/+bug/2168017/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
