Public bug reported: [ Impact ]
If a transient QEMU domain [1] takes a long time to shut down and cannot be killed with SIGKILL, libvirt will fail to remove the domain from the QEMU driver's linked-list of running domains. When this happens, the domain cannot be removed except by restarting libvirtd. It prevents new domains being created with the same name. It can also confuse API users of libvirt which expect a transient domain to eventually disappear. This has been observed when a domain has a very large memory (500GB+, cause of the slowness unclear) [2]. It can also happen if QEMU is stuck in D-state doing some block operation. [1] https://wiki.libvirt.org/VM_lifecycle.html#transient-guest-domains-vs-persistent-guest-domains [2] https://gitlab.com/libvirt/libvirt/-/work_items/853 [ Test Plan ] Use a frozen ext4 filesystem + blockcommit to artificially hold QEMUs worker threads in D state: ```sh lxc launch ubuntu:resolute r0 --vm -c limits.cpu=8 -c limits.memory=12GiB lxc storage volume create default libvirt-dstate size=128MiB --type=block lxc storage volume attach default libvirt-dstate r0 dstate ``` ```sh apt install -y libvirt-daemon-system qemu-system-x86 qemu-utils e2fsprogs util-linux mkfs.ext4 -F /dev/sdb mkdir /mnt/dstate mount /dev/sdb /mnt/dstate qemu-img create -f raw /mnt/dstate/base.raw 64M qemu-img create -f qcow2 \ -F raw \ -b /mnt/dstate/base.raw \ /var/lib/libvirt/images/overlay.qcow2 qemu-io -f qcow2 \ -c 'write -P 0x55 0 64M' \ /var/lib/libvirt/images/overlay.qcow2 cat >/root/transient-qemu.xml <<'EOF' <domain type='qemu'> <name>transient-qemu</name> <memory unit='MiB'>64</memory> <vcpu placement='static'>1</vcpu> <os> <type arch='x86_64' machine='pc'>hvm</type> </os> <devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none'/> <source file='/var/lib/libvirt/images/overlay.qcow2'/> <target dev='vda' bus='virtio'/> </disk> <console type='pty'/> </devices> </domain> EOF virsh create /root/transient-qemu.xml fsfreeze -f /mnt/dstate virsh blockcommit transient-qemu vda --active --pivot --wait \ >/root/blockcommit.out 2>&1 & echo $! >/root/blockcommit.pid # one or more worker threads in D state with WCHAN percpu_rwsem_wait ps -L -o pid,tid,stat,wchan:32,comm \ -p "$(cat /run/libvirt/qemu/transient-qemu.pid)" ``` `virsh destroy` will fail with "Device or resource busy" after ~40 seconds ```sh virsh destroy transient-qemu # transient-qemu "in shutdown" virsh list --all fsfreeze -u /mnt/dstate ``` Actual behavior (unpatched): - `transient-qemu` remains listed as "shut off" - recreation fails because `transient-qemu` already exists ```sh virsh list --all virsh create /root/transient-qemu.xml ``` Expected behavior (patched): - `transient-qemu` no longer listed - recreation succeeds ```sh virsh list --all virsh create /root/transient-qemu.xml virsh destroy transient-qemu ``` [ Where problems could occur ] The modified code is executed only when a QEMU_PROCESS_EVENT_SHUTDOWN_COMPLETED event is recieved; this event is only created when the pidfd for a QEMU process becomes readable (i.e. the process exits); see `qemuProcessInShutdownStartMonitor:qemu/qemu_process.c:8913`. If the code was broken, we would expect to see failures during domain shutdown. `qemuDomainRemoveInactive` is a no-op for persistent domains. [ Other info ] Upstream fix (v12.2.0): https://gitlab.com/libvirt/libvirt/-/commit/837463f83f2a750418674e32cb802cfbf13a977d Introduced in (v10.6.0): https://gitlab.com/libvirt/libvirt/-/commit/e62c26a20dced58ea342d9cb8f5e9164dc3bb023 Upstream bug report & reproducer: https://gitlab.com/libvirt/libvirt/-/work_items/853 Behavior: ``` root@r0:~# virsh list --all Id Name State --------------------------------- - transient-qemu shut off root@r0:~# virsh create /root/transient-qemu.xml error: Failed to create domain from /root/transient-qemu.xml error: operation failed: domain 'transient-qemu' already exists with uuid 0953da8d-539b-43c4-8953-25786ced29b8 root@r0:~# virsh list --all Id Name State --------------------------------- - transient-qemu shut off root@r0:~# virsh destroy transient-qemu error: Failed to destroy domain 'transient-qemu' error: Requested operation is not valid: domain is not running root@r0:~# virsh list --all Id Name State --------------------------------- - transient-qemu shut off root@r0:~# virsh undefine transient-qemu error: Failed to undefine domain 'transient-qemu' error: Requested operation is not valid: cannot undefine transient domain root@r0:~# virsh list --all Id Name State --------------------------------- - transient-qemu shut off ``` ** Affects: libvirt (Ubuntu) Importance: Undecided Status: Fix Released ** Affects: libvirt (Ubuntu Resolute) Importance: Medium Assignee: Wesley Hershberger (whershberger) Status: In Progress ** Affects: libvirt (Ubuntu Stonking) Importance: Undecided Status: Fix Released ** Affects: libvirt (Ubuntu Tenacious) Importance: Undecided Status: Fix Released ** Also affects: libvirt (Ubuntu Resolute) Importance: Undecided Status: New ** Also affects: libvirt (Ubuntu Stonking) Importance: Undecided Status: New ** Also affects: libvirt (Ubuntu Tenacious) Importance: Medium Status: In Progress ** Changed in: libvirt (Ubuntu Tenacious) Status: In Progress => Fix Released ** Changed in: libvirt (Ubuntu Stonking) Status: New => Fix Released ** Changed in: libvirt (Ubuntu Resolute) Status: New => In Progress ** Changed in: libvirt (Ubuntu Resolute) Assignee: (unassigned) => Wesley Hershberger (whershberger) ** Changed in: libvirt (Ubuntu Tenacious) Importance: Medium => Undecided ** Changed in: libvirt (Ubuntu Resolute) Importance: Undecided => Medium -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2173132 Title: Transient domains leaked if SIGKILL of QEMU fails To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/2173132/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
