Steps to reproduce on Focal (shutdown-on-init) --- LXD virtual machine
lxc exec lp2059272-focal -- su - ubuntu lxc exec lp2059272-focal -- su - ubuntu Latest Packages and Debug Symbols: cat <<EOF | sudo tee /etc/apt/sources.list.d/proposed.list deb http://archive.ubuntu.com/ubuntu focal-proposed main universe deb http://ddebs.ubuntu.com focal-proposed main restricted EOF cat <<EOF | sudo tee /etc/apt/preferences.d/proposed Package: * Pin: release a=focal-proposed Pin-Priority: 400 EOF sudo apt install --yes --no-install-recommends gdb qemu-system-x86 ubuntu-dbgsym-keyring sudo apt install --yes --no-install-recommends -t focal-proposed libvirt{0,-daemon{,-driver-qemu,-system}}{,-dbgsym} libvirt-clients $ dpkg -l | awk '$2 ~ /^libvirt/ { print $2, $3 }' libvirt-clients 6.0.0-0ubuntu8.17 libvirt-daemon 6.0.0-0ubuntu8.17 libvirt-daemon-dbgsym 6.0.0-0ubuntu8.17 libvirt-daemon-driver-qemu 6.0.0-0ubuntu8.17 libvirt-daemon-driver-qemu-dbgsym 6.0.0-0ubuntu8.17 libvirt-daemon-system 6.0.0-0ubuntu8.17 libvirt-daemon-system-systemd 6.0.0-0ubuntu8.17 libvirt0:amd64 6.0.0-0ubuntu8.17 libvirt0-dbgsym:amd64 6.0.0-0ubuntu8.17 Start test VM cat <<-EOF >test-vm.xml <domain type='qemu'> <name>test-vm</name> <os> <type>hvm</type> </os> <memory unit='MiB'>32</memory> <vcpu>1</vcpu> </domain> EOF virsh define test-vm.xml virsh start test-vm $ virsh list Id Name State ------------------------- 1 test-vm running Stop libvirt systemd units sudo systemctl stop 'libvirtd*' Start libvirt in GDB sudo gdb \ -iex 'set confirm off' \ -iex 'set pagination off' \ -ex 'set non-stop on' \ -ex 'handle SIGTERM nostop noprint pass' \ -ex 'add-symbol-file /usr/sbin/libvirtd' \ -ex 'add-symbol-file /usr/lib/x86_64-linux-gnu/libvirt.so.0' \ -ex 'add-symbol-file /usr/lib/x86_64-linux-gnu/libvirt-qemu.so.0' \ -ex 'add-symbol-file /usr/lib/x86_64-linux-gnu/libvirt/connection-driver/libvirt_driver_qemu.so' \ /usr/sbin/libvirtd Add breakpoints for qemu driver cleanup and domain status XML save b qemuStateCleanup b virDomainObjSave run Thread 20 "libvirtd" hit Breakpoint 2, virDomainObjSave (obj=0x555cf5d83480, xmlopt=0x555cf5d7e6a0, statusDir=0x555cf5d26e70 "/run/libvirt/qemu") at ../../../src/conf/domain_conf.c:29157 Check the backtrace of the domain status XML save function, coming from QEMU process reconnect: t 20 (gdb) bt #0 virDomainObjSave (obj=0x555cf5d83480, xmlopt=0x555cf5d7e6a0, statusDir=0x555cf5d26e70 "/run/libvirt/qemu") at ../../../src/conf/domain_conf.c:29157 #1 0x00007f743b666268 in qemuProcessReconnect (opaque=<optimized out>) at ../../../src/qemu/qemu_process.c:8122 #2 0x00007f7460b9054a in virThreadHelper (data=<optimized out>) at ../../../src/util/virthread.c:196 #3 0x00007f7460851609 in start_thread (arg=<optimized out>) at pthread_create.c:477 #4 0x00007f7460776353 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 Leave the thread at this point Let's trigger the shutdown path $ sudo kill $(pidof libvirtd) Thread 1 "libvirtd" hit Breakpoint 1, qemuStateCleanup () at ../../../src/qemu/qemu_driver.c:1118 Check there are 2 threads: cleanup and domain status XML save (gdb) i th Id Target Id Frame 1 Thread 0x7f745cd37b40 (LWP 8029) "libvirtd" qemuStateCleanup () at ../../../src/qemu/qemu_driver.c:1118 2 Thread 0x7f745c8a9700 (LWP 8034) "libvirtd" (running) ... 18 Thread 0x7f7417fff700 (LWP 8100) "libvirtd" (running) * 20 Thread 0x7f7416ffd700 (LWP 8105) "libvirtd" virDomainObjSave (obj=0x555cf5d83480, xmlopt=0x555cf5d7e6a0, statusDir=0x555cf5d26e70 "/run/libvirt/qemu") at ../../../src/conf/domain_conf.c:29157 Confirm the qemu driver's domain xml formatter/options is set/referenced: t 20 (gdb) p xmlopt.privateData.format $1 = (virDomainXMLPrivateDataFormatFunc) 0x7f743b628810 <qemuDomainObjPrivateXMLFormat> (gdb) p xmlopt.parent.u.s.refs $2 = 1 (gdb) p/x xmlopt.parent $3 = {u = {dummy_align1 = 0x1cafe0027, dummy_align2 = 0x1cafe0027, s = {magic = 0xcafe0027, refs = 0x1}}, klass = 0x555cf5dbbbb0} Let the cleanup function finish t 1 finish Check the formatter/options again; it is now zeroed, and used after freed: t 20 (gdb) p xmlopt.privateData.format $5 = (virDomainXMLPrivateDataFormatFunc) 0x0 (gdb) p xmlopt.parent.u.s.refs $6 = 21852 (gdb) p/x xmlopt.parent $7 = {u = {dummy_align1 = 0x555cf5d39800, dummy_align2 = 0x555cf5d39800, s = {magic = 0xf5d39800, refs = 0x555c}}, klass = 0x555cf5d09010} The object data is zeroed in the last unreference in Focal, and its contents changed, as this is really an use-after-free (another thread might get/use that memory). Check the VM status XML *before* the save function finishes: $ sudo grep -e '<domstatus' -e '<domain' -e 'monitor path' /run/libvirt/qemu/test-vm.xml <domstatus state='running' reason='booted' pid='7932'> <monitor path='/var/lib/libvirt/qemu/domain-1-test-vm/monitor.sock' type='unix'/> <domain type='qemu' id='1'> Let the save function continue (gdb) c & Check the VM status XML *after*: $ sudo grep -e '<domstatus' -e '<domain' -e 'monitor path' /run/libvirt/qemu/test-vm.xml <domstatus state='running' reason='booted' pid='7932'> <domain type='qemu' id='1'> It no longer has the 'monitor path' tag/field. Let libvirt finish shutting down: (gdb) t 1 (gdb) c Continuing. ... [Inferior 1 (process 8029) exited normally] (gdb) quit Now, the next time libvirtd starts, it fails to parse that XML: $ sudo systemctl start libvirtd.service $ journalctl -b -u libvirtd.service | tail ... ... libvirtd[8297]: 8313: error : qemuDomainObjPrivateXMLParse:3678 : internal error: no monitor path ... libvirtd[8297]: 8313: error : virDomainObjListLoadAllConfigs:632 : Failed to load config for domain 'test-vm' And libvirt is not aware of the domain, and cannot manage it: $ virsh list Id Name State -------------------- $ virsh list --all Id Name State -------------------------- - test-vm shut off Even though it is still running: $ pgrep -af qemu-system-x86_64 | cut -d, -f1 7932 /usr/bin/qemu-system-x86_64 -name guest=test-vm Stop it manually $ sudo kill $(pgrep -f qemu-system-x86_64) $ sudo rm /run/libvirt/qemu/test-vm.{xml,pid} Start libvirt: sudo systemctl start libvirtd.service -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2059272 Title: libvirt domain is not listed/managed after libvirt restart with messages "internal error: no monitor path" and "Failed to load config for domain" To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/2059272/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs