** Description changed:

  [ Impact ]
  
  If a transient QEMU domain [1] takes a long time to shut down and cannot
- be killed with SIGKILL, libvirt will fail to remove the domain from the
- QEMU driver's linked-list of running domains.
+ be killed with SIGKILL, libvirt may fail to remove the domain from the
+ QEMU driver's linked-list of running domains. This only occurs if
+ libvirt processes events triggered by domain stop in a specific order
+ (see "Reproducibility" below).
  
  When this happens, the domain cannot be removed except by restarting
  libvirtd. It prevents new domains being created with the same name. It
  can also confuse API users of libvirt which expect a transient domain to
  eventually disappear.
  
+ In this case, libvirt also fails to emit a VIR_DOMAIN_EVENT_STOPPED,
+ which may confuse libvirt API users who expect transient domains to
+ eventually send that event [2].
+ 
  This has been observed when a domain has a very large memory (500GB+,
- cause of the slowness unclear) [2]. It can also happen if QEMU is stuck
+ cause of the slowness unclear) [3]. It can also happen if QEMU is stuck
  in D-state doing some block operation.
  
  [1] 
https://wiki.libvirt.org/VM_lifecycle.html#transient-guest-domains-vs-persistent-guest-domains
- [2] https://gitlab.com/libvirt/libvirt/-/work_items/853
+ [2] 
https://libvirt.org/html/libvirt-libvirt-domain.html#virConnectDomainEventRegisterAny
+ [3] https://gitlab.com/libvirt/libvirt/-/work_items/853
  
  [ Test Plan ]
  
- Use a frozen ext4 filesystem + blockcommit to artificially hold QEMUs
- worker threads in D state:
+ Use a frozen ext4 filesystem for RAM to artificially hold QEMUs worker
+ threads in D state:
  
  ```sh
  lxc launch ubuntu:resolute r0 --vm -c limits.cpu=8 -c limits.memory=12GiB
  
  lxc storage volume create default libvirt-dstate size=128MiB --type=block
  lxc storage volume attach default libvirt-dstate r0 dstate
  ```
  
  ```sh
- apt install -y libvirt-daemon-system qemu-system-x86 qemu-utils e2fsprogs 
util-linux
+ apt install -y libvirt-daemon-system qemu-system-x86
  
  mkfs.ext4 -F /dev/sdb
- mkdir /mnt/dstate
- mount /dev/sdb /mnt/dstate
- 
- qemu-img create -f raw /mnt/dstate/base.raw 64M
- qemu-img create -f qcow2 \
-     -F raw \
-     -b /mnt/dstate/base.raw \
-     /var/lib/libvirt/images/overlay.qcow2
- qemu-io -f qcow2 \
-     -c 'write -P 0x55 0 64M' \
-     /var/lib/libvirt/images/overlay.qcow2
+ mount /dev/sdb /var/lib/libvirt/qemu/ram
+ chown libvirt-qemu:kvm /var/lib/libvirt/qemu/ram
  
  cat >/root/transient-qemu.xml <<'EOF'
  <domain type='qemu'>
    <name>transient-qemu</name>
    <memory unit='MiB'>64</memory>
    <vcpu placement='static'>1</vcpu>
+   <memoryBacking>
+     <source type='file'/>
+   </memoryBacking>
    <os>
      <type arch='x86_64' machine='pc'>hvm</type>
    </os>
    <devices>
      <emulator>/usr/bin/qemu-system-x86_64</emulator>
-     <disk type='file' device='disk'>
-       <driver name='qemu' type='qcow2' cache='none'/>
-       <source file='/var/lib/libvirt/images/overlay.qcow2'/>
-       <target dev='vda' bus='virtio'/>
-     </disk>
      <console type='pty'/>
    </devices>
  </domain>
  EOF
  
  virsh create /root/transient-qemu.xml
  
- fsfreeze -f /mnt/dstate
+ virsh event --domain transient-qemu --event lifecycle --loop
+ >/root/events.log &
  
- virsh blockcommit transient-qemu vda --active --pivot --wait \
-     >/root/blockcommit.out 2>&1 &
- echo $! >/root/blockcommit.pid
- 
- # one or more worker threads in D state with WCHAN percpu_rwsem_wait
- ps -L -o pid,tid,stat,wchan:32,comm \
-     -p "$(cat /run/libvirt/qemu/transient-qemu.pid)"
+ # unlink to force inode eviction, otherwise QEMU won't hang
+ rm /var/lib/libvirt/qemu/ram/*-transient-qemu/pc.ram
+ fsfreeze -f /var/lib/libvirt/qemu/ram
  ```
  
  `virsh destroy` will fail with "Device or resource busy" after ~40 seconds
  ```sh
  virsh destroy transient-qemu
  
- # transient-qemu "in shutdown"
+ # transient-qemu should be "in shutdown"
+ # this might hang for a while due to the fsfreeze but does eventually return
  virsh list --all
  
- fsfreeze -u /mnt/dstate
+ # main thread should be in D state
+ ps -L -o pid,tid,stat,wchan:32,comm \
+     -p "$(cat /run/libvirt/qemu/transient-qemu.pid)"
+ 
+ fsfreeze -u /var/lib/libvirt/qemu/ram
  ```
  
  Actual behavior (unpatched):
  - `transient-qemu` remains listed as "shut off"
  - recreation fails because `transient-qemu` already exists
  
  ```sh
  virsh list --all
  virsh create /root/transient-qemu.xml
+ ```
+ 
+ `events.log` will not contain VIR_DOMAIN_EVENT_STOPPED:
+ ```
+ event 'lifecycle' for domain 'transient-qemu': Shutdown Finished after host 
request
  ```
  
  Expected behavior (patched):
  - `transient-qemu` no longer listed
  - recreation succeeds
  
  ```sh
  virsh list --all
  virsh create /root/transient-qemu.xml
  virsh destroy transient-qemu
  ```
  
+ `events.log` will contain VIR_DOMAIN_EVENT_STOPPED:
+ ```
+ event 'lifecycle' for domain 'transient-qemu': Shutdown Finished after host 
request
+ event 'lifecycle' for domain 'transient-qemu': Stopped Shutdown
+ ```
+ 
  [ Where problems could occur ]
  
- The modified code is executed only when a
+ The modified code in 837463f83f is executed only when a
  QEMU_PROCESS_EVENT_SHUTDOWN_COMPLETED event is recieved; this event is
  only created when the pidfd for a QEMU process becomes readable (i.e.
  the process exits); see
  `qemuProcessInShutdownStartMonitor:qemu/qemu_process.c:8913`.
  
- If the code was broken, we would expect to see failures during domain
- shutdown.
+ `qemuDomainRemoveInactive` is a no-op for persistent domains.
  
- `qemuDomainRemoveInactive` is a no-op for persistent domains.
+ bf9a96f85a6 affects both persistent and transient domains.
+ 
+ If the code was broken, we would expect to see failures during/after
+ domain shutdown.
  
  [ Other info ]
  
- Upstream fix (v12.2.0):
- 
https://gitlab.com/libvirt/libvirt/-/commit/837463f83f2a750418674e32cb802cfbf13a977d
+ Upstream fix (v12.2.0): 
https://gitlab.com/libvirt/libvirt/-/commit/837463f83f2a750418674e32cb802cfbf13a977d
+ Follow-up (v12.8.0): 
https://gitlab.com/libvirt/libvirt/-/commit/bf9a96f85a6a158ca7d5dd3a4ab02232e285c4e3
  
  Introduced in (v10.6.0):
  
https://gitlab.com/libvirt/libvirt/-/commit/e62c26a20dced58ea342d9cb8f5e9164dc3bb023
  
  Upstream bug report & reproducer:
  https://gitlab.com/libvirt/libvirt/-/work_items/853
  
- Behavior:
+ # Reproducibility
+ There are two libvirt-internal events which are emitted when a QEMU process 
exits:
+ - QEMU_PROCESS_EVENT_SHUTDOWN_COMPLETED was introduced by upstream e62c26a20 
and fires when the pidfd for the qemu process becomes readable
+ - QEMU_PROCESS_EVENT_MONITOR_EOF when the QMP socket gives EOF
+ 
+ This bug only occurs when QEMU_PROCESS_EVENT_SHUTDOWN_COMPLETED is
+ processed before QEMU_PROCESS_EVENT_MONITOR_EOF (since
+ processMonitorEOFEvent correctly calls qemuDomainRemoveInactive).
+ 
+ A previous version of the Test Plan used a blockcommit on a frozen
+ filesystem to reproduce the bug, but because QEMU hung before closing
+ its end of the QMP socket on blockcommit, it caused the two above events
+ to race.
+ 
+ In the "ram too slow" case described in the upstream bug,
+ QEMU_PROCESS_EVENT_MONITOR_EOF is emitted immediately after sigkill, so
+ the bug is reproducible deterministically.
+ 
+ # Behavior
  
  ```
  root@r0:~# virsh list --all
-  Id   Name             State
+  Id Name State
  ---------------------------------
-  -    transient-qemu   shut off
+  - transient-qemu shut off
  
  root@r0:~# virsh create /root/transient-qemu.xml
  error: Failed to create domain from /root/transient-qemu.xml
  error: operation failed: domain 'transient-qemu' already exists with uuid 
0953da8d-539b-43c4-8953-25786ced29b8
  
  root@r0:~# virsh list --all
-  Id   Name             State
+  Id Name State
  ---------------------------------
-  -    transient-qemu   shut off
+  - transient-qemu shut off
  
  root@r0:~# virsh destroy transient-qemu
  error: Failed to destroy domain 'transient-qemu'
  error: Requested operation is not valid: domain is not running
  
  root@r0:~# virsh list --all
-  Id   Name             State
+  Id Name State
  ---------------------------------
-  -    transient-qemu   shut off
+  - transient-qemu shut off
  
  root@r0:~# virsh undefine transient-qemu
  error: Failed to undefine domain 'transient-qemu'
  error: Requested operation is not valid: cannot undefine transient domain
  
  root@r0:~# virsh list --all
-  Id   Name             State
+  Id Name State
  ---------------------------------
-  -    transient-qemu   shut off
+  - transient-qemu shut off
  ```

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2173132

Title:
  Transient domains leaked if SIGKILL of QEMU fails

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/2173132/+subscriptions


-- 
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

Reply via email to