Steps with test packages on Focal (shutdown-on-runtime)
---
Environment:
---
On top of LXD VM in comments #12/#13.
Enable PPA & debug symbols
sudo add-apt-repository -yn ppa:mfo/lp2059272
sudo sed '/^deb / s,$, main/debug,' -i
/etc/apt/sources.list.d/mfo-ubuntu-lp2059272-focal.list
sudo apt update
Install packages
sudo apt install --yes libvirt{0,-daemon{,-driver-
qemu}}{,-dbgsym} libvirt-clients gdb qemu-system-x86
$ dpkg -s libvirt-daemon | grep ^Version:
Version: 6.0.0-0ubuntu8.18~ppa1
Libvirtd debug logging
cat <<EOF | sudo tee -a /etc/libvirt/libvirtd.conf
log_filters="1:qemu 1:libvirt"
log_outputs="3:syslog:libvirtd
1:file:/var/log/libvirt/libvirtd-debug.log"
EOF
Follow `Steps to reproduce on Focal (shutdown-on-runtime)` in comment #12
---
Up to ...
Check the backtrace of the domain status XML save function, coming from
device deleted event
Thread 20 "libvirtd" hit Breakpoint 3, virDomainObjSave
(obj=0x7fb3202733b0, xmlopt=0x7fb3200400d0, statusDir=0x7fb32000a620
"/run/libvirt/qemu") at ../../../src/conf/domain_conf.c:29157
29157 ../../../src/conf/domain_conf.c: No such file or directory.
(gdb) bt
#0 virDomainObjSave (obj=0x7fb3202733b0, xmlopt=0x7fb3200400d0,
statusDir=0x7fb32000a620 "/run/libvirt/qemu") at
../../../src/conf/domain_conf.c:29157
#1 0x00007fb32c20d627 in processDeviceDeletedEvent
(devAlias=0x564ae9f032e0 "input0", vm=0x7fb3202733b0, driver=0x7fb320039ed0) at
../../../src/qemu/qemu_driver.c:4367
#2 qemuProcessEventHandler (data=0x564ae9fc88f0,
opaque=0x7fb320039ed0) at ../../../src/qemu/qemu_driver.c:4943
#3 0x00007fb333ed51af in virThreadPoolWorker
(opaque=opaque@entry=0x564ae9fd3080) at ../../../src/util/virthreadpool.c:163
#4 0x00007fb333ed451c in virThreadHelper (data=<optimized out>) at
../../../src/util/virthread.c:196
#5 0x00007fb333b95609 in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007fb333aba353 in clone () from /lib/x86_64-linux-gnu/libc.so.6
$ sudo kill $(pidof libvirtd)
Thread 1 "libvirtd" hit Breakpoint 1, qemuStateCleanup () at
../../../src/qemu/qemu_driver.c:1180
t 20
(gdb) p xmlopt.privateData.format
$1 = (virDomainXMLPrivateDataFormatFunc) 0x7fb32c167890
<qemuDomainObjPrivateXMLFormat>
t 1
c &
Check the formatter/options again; it is *STILL* referenced, not 0x0
anymore:
t 20
(gdb) p xmlopt.privateData.format
$2 = (virDomainXMLPrivateDataFormatFunc) 0x7fb32c167890
<qemuDomainObjPrivateXMLFormat>
Check the shutdown/cleanup thread is waiting for it,
in the path to free the worker thread pool:
(gdb) i th 1
Id Target Id Frame
1 Thread 0x7fb33007bb40 (LWP 6585) "libvirtd" (running)
t 1
interrupt
(gdb) bt
#0 0x00007fb333b9c376 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00007fb333ed479b in virCondWait (c=<optimized out>, m=<optimized
out>) at ../../../src/util/virthread.c:144
#2 0x00007fb333ed5438 in virThreadPoolFree (pool=<optimized out>) at
../../../src/util/virthreadpool.c:286
#3 0x00007fb32c1d89e3 in qemuStateCleanup () at
../../../src/qemu/qemu_driver.c:1186
#4 0x00007fb33407246f in virStateCleanup () at
../../../src/libvirt.c:669
#5 0x0000564ae98babc8 in main (argc=<optimized out>, argv=<optimized
out>) at ../../../src/remote/remote_daemon.c:1447
Let the save function continue, and libvirt finishes shutting down:
(gdb) c &
Continuing.
(gdb) t 20
(gdb) c
[Inferior 1 (process 6585) exited normally]
(gdb) q
Check the VM status XML *after*:
$ sudo grep -e '<domstatus' -e '<domain' -e 'monitor path'
/run/libvirt/qemu/test-vm.xml
<domstatus state='running' reason='booted' pid='6634'>
<monitor path='/var/lib/libvirt/qemu/domain-1-test-vm/monitor.sock'
type='unix'/>
<domain type='qemu' id='1'>
It *still* has the 'monitor path' tag/field.
Now, the next time libvirtd starts, it correctly parses that XML:
$ sudo systemctl start libvirtd.service
$ journalctl -b -u libvirtd.service | grep -A1 error
Mar 30 22:27:20 lp2059272-focal libvirtd[6670]: 6686: error :
dnsmasqCapsRefreshInternal:714 : Cannot check dnsmasq binary /usr/sbin/dnsmasq:
No such file or directory
And libvirt is aware of the domain, and can manage it:
$ virsh list
Id Name State
-------------------------
1 test-vm running
$ virsh destroy test-vm
Domain test-vm destroyed
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2059272
Title:
libvirt domain is not listed/managed after libvirt restart with
messages "internal error: no monitor path" and "Failed to load config
for domain"
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/2059272/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
