[Bug 2059272] Re: libvirt domain is not listed/managed after libvirt restart with messages "internal error: no monitor path" and "Failed to load config for domain"

This bug was fixed in the package libvirt - 6.0.0-0ubuntu8.20 --- libvirt (6.0.0-0ubuntu8.20) focal; urgency=medium * d/p/u/lp2059272-2-qemu-Wait-qemuProcessReconnect-threads-in-cleanup.patch: Remove patch. It is not possible to wait for qemuProcessReconnect() in cleanup:

[Bug 2059272] Re: libvirt domain is not listed/managed after libvirt restart with messages "internal error: no monitor path" and "Failed to load config for domain"

The packages in focal-proposed have also been verified successfully in real-world/non-synthetic tests by one of our Ubuntu Pro support customers. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2059272

[Bug 2059272] Re: libvirt domain is not listed/managed after libvirt restart with messages "internal error: no monitor path" and "Failed to load config for domain"

Verification done on focal-proposed, following comments 23, 24, 25, 26. Including in this comment a few key snippets from each test/comment. --- Environment --- LXD virtual machine lxc launch --vm ubuntu:focal lp2059272-focal lxc exec lp2059272-focal -- su - ubuntu Enable proposed & debug

[Bug 2059272] Re: libvirt domain is not listed/managed after libvirt restart with messages "internal error: no monitor path" and "Failed to load config for domain"

Hello Mauricio, or anyone else affected, Accepted libvirt into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libvirt/6.0.0-0ubuntu8.20 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See

[Bug 2059272] Re: libvirt domain is not listed/managed after libvirt restart with messages "internal error: no monitor path" and "Failed to load config for domain"

Mauricio identified that there are many symbols added in libvirt-daemon- driver-qemu.so, due to the inclusion of the RPC archives, associated with gcc/ld export-dynamic in the build (which includes all symbols, not just actually used). On consultation with both Mauricio and Sergio, we concluded

[Bug 2059272] Re: libvirt domain is not listed/managed after libvirt restart with messages "internal error: no monitor path" and "Failed to load config for domain"

SRU team: this is on hold to double check the library symbols (thanks, Robie!), please do not accept yet. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2059272 Title: libvirt domain is not

[Bug 2059272] Re: libvirt domain is not listed/managed after libvirt restart with messages "internal error: no monitor path" and "Failed to load config for domain"

Hi Sergio, Thanks for taking the time to read and review the patch and test cases! > this new code path could be reached even after the initialization, couldn't it? No, this code path should only be reached during initialization, AFAICT. That is because qemuProcessReconnect() can only be

[Bug 2059272] Re: libvirt domain is not listed/managed after libvirt restart with messages "internal error: no monitor path" and "Failed to load config for domain"

Hi Mauricio, Thanks for the detailed explanations, as usual. I took some time here to read your patch. It took me down the rabbit hole and I did some archaeology to find out what these "inhibit*" pointers are about. It was fun; I learned that they were introduced back in 2012 as a means to

[Bug 2059272] Re: libvirt domain is not listed/managed after libvirt restart with messages "internal error: no monitor path" and "Failed to load config for domain"

Steps with test packages on Focal (shutdown-on-runtime) --- Stop libvirtd systemd units sudo systemctl stop 'libvirtd*' Start libvirt in GDB sudo gdb \ -iex 'set confirm off' \ -iex 'set pagination off' \ -ex 'set non-stop on' \ -ex 'handle SIGTERM nostop noprint

[Bug 2059272] Re: libvirt domain is not listed/managed after libvirt restart with messages "internal error: no monitor path" and "Failed to load config for domain"

Steps with test packages on Focal (shutdown-on-init) --- Start test VM cat <<-EOF >test-vm.xml test-vm hvm 32 1 EOF virsh define test-vm.xml virsh start test-vm

[Bug 2059272] Re: libvirt domain is not listed/managed after libvirt restart with messages "internal error: no monitor path" and "Failed to load config for domain"

Steps with test packages on Focal (normal restarts) --- Restart libvirt 100 times with 10 QEMU domains. All domains continued to be managed by libvirt. Create 10 test VMs (test-vm-1, test-vm-2, ..., test-vm-10): for NAME in test-vm-{1..10}; do cat <<-EOF >test-vms.xml && virsh define

[Bug 2059272] Re: libvirt domain is not listed/managed after libvirt restart with messages "internal error: no monitor path" and "Failed to load config for domain"

Test packages with the new fix in ppa:mfo/lp2059272 built correctly in all supported architectures. The next 3 comments show steps to reproduce/verify the issue/fix with the test packages -- all passed. Environment --- LXD virtual machine lxc launch --vm ubuntu:focal lp2059272-focal

[Bug 2059272] Re: libvirt domain is not listed/managed after libvirt restart with messages "internal error: no monitor path" and "Failed to load config for domain"

The new fix for shutdown-on-init on Focal is simpler: Skip the XML update on init if libvirt is shutting down. This is based on 2 points: 1) The XML update on initialization will not be used at all in this run of libvirtd, since libvirtd is shutting down. 2) The XML update in _this_

[Bug 2059272] Re: libvirt domain is not listed/managed after libvirt restart with messages "internal error: no monitor path" and "Failed to load config for domain"

** Description changed: [ Impact ]  * If a race condition occurs on libvirtd shutdown,    a QEMU domain status XML (/run/libvirt/qemu/*.xml)    might lose the QEMU-driver specific information,    such as ''.    (The race condition details are in [Other Info].)  * On the next

[Bug 2059272] Re: libvirt domain is not listed/managed after libvirt restart with messages "internal error: no monitor path" and "Failed to load config for domain"

This bug was fixed in the package libvirt - 8.0.0-1ubuntu7.9 --- libvirt (8.0.0-1ubuntu7.9) jammy; urgency=medium * d/p/u/lp2059272-qemu-Fix-potential-crash-during-driver-cleanup.patch: On QEMU driver cleanup, release (stop) the worker thread pool _first_, before other data

[Bug 2059272] Re: libvirt domain is not listed/managed after libvirt restart with messages "internal error: no monitor path" and "Failed to load config for domain"

** Tags removed: verification-needed-focal ** Tags added: verification-failed-focal -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2059272 Title: libvirt domain is not listed/managed after libvirt

[Bug 2059272] Re: libvirt domain is not listed/managed after libvirt restart with messages "internal error: no monitor path" and "Failed to load config for domain"

(Un)fortunately, during focal-proposed testing today, a corner case was identified (late), which will require changes. Some of the qemuProcessReconnect() threads finished within the timeout but others didn't. The ones which didn't were waiting on a reply back from the QEMU monitor, but that

[Bug 2059272] Re: libvirt domain is not listed/managed after libvirt restart with messages "internal error: no monitor path" and "Failed to load config for domain"

Hello Mauricio, or anyone else affected, Accepted libvirt into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libvirt/6.0.0-0ubuntu8.18 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See

[Bug 2059272] Re: libvirt domain is not listed/managed after libvirt restart with messages "internal error: no monitor path" and "Failed to load config for domain"

Thanks for the reviews, Sergio! That certainly helps building additional confidence in the patch. With that and tests covering all options (comments #13 and #15) showing good/expected results, I'm happy to upload it to Focal. ** Changed in: libvirt (Ubuntu Focal) Status: Confirmed => In

[Bug 2059272] Re: libvirt domain is not listed/managed after libvirt restart with messages "internal error: no monitor path" and "Failed to load config for domain"

Mauricio asked me to review the debdiff for Focal as well. This debdiff is different because he had to implement some code to synchronize the destruction of the worker threads with the free'ing of the thread pool. I looked at the new code, analyzed it as best as I could, asked a few questions to

[Bug 2059272] Re: libvirt domain is not listed/managed after libvirt restart with messages "internal error: no monitor path" and "Failed to load config for domain"

Steps with test packages on Focal (shutdown-on-init) --- Environment: --- On top of LXD VM in comments #12/#13. Enable PPA & debug symbols sudo add-apt-repository -yn ppa:mfo/lp2059272 sudo sed '/^deb / s,$, main/debug,' -i

[Bug 2059272] Re: libvirt domain is not listed/managed after libvirt restart with messages "internal error: no monitor path" and "Failed to load config for domain"

Steps with test packages on Focal (shutdown-on-runtime) --- Environment: --- On top of LXD VM in comments #12/#13. Enable PPA & debug symbols sudo add-apt-repository -yn ppa:mfo/lp2059272 sudo sed '/^deb / s,$, main/debug,' -i

[Bug 2059272] Re: libvirt domain is not listed/managed after libvirt restart with messages "internal error: no monitor path" and "Failed to load config for domain"

Steps to reproduce on Focal (shutdown-on-init) --- LXD virtual machine lxc exec lp2059272-focal -- su - ubuntu lxc exec lp2059272-focal -- su - ubuntu Latest Packages and Debug Symbols: cat

[Bug 2059272] Re: libvirt domain is not listed/managed after libvirt restart with messages "internal error: no monitor path" and "Failed to load config for domain"

Steps to reproduce on Focal (shutdown-on-runtime) --- LXD virtual machine lxc exec lp2059272-focal -- su - ubuntu lxc exec lp2059272-focal -- su - ubuntu Latest Packages and Debug Symbols: cat

[Bug 2059272] Re: libvirt domain is not listed/managed after libvirt restart with messages "internal error: no monitor path" and "Failed to load config for domain"

Verification done on jammy-proposed. --- Part 1: comment #2, libvirt starts without errors, and can list and manage the domain. Part 2: comment #5, libvirt restarts without errors 100 times with 10 domains. Environment: --- LXD container lxc launch --vm ubuntu:jammy lp2059272-jammy

[Bug 2059272] Re: libvirt domain is not listed/managed after libvirt restart with messages "internal error: no monitor path" and "Failed to load config for domain"

** Description changed: [ Impact ]  * If a race condition occurs on libvirtd shutdown,    a QEMU domain status XML (/run/libvirt/qemu/*.xml)    might lose the QEMU-driver specific information,    such as ''.  * On the next libvirtd startup, the parsing of that    QEMU domain's

[Bug 2059272] Re: libvirt domain is not listed/managed after libvirt restart with messages "internal error: no monitor path" and "Failed to load config for domain"

> Please include the "normal" execution from comment #5 in the test plan, besides the synthetic one. Done; thanks! ** Description changed: [ Impact ]  * If a race condition occurs on libvirtd shutdown,    a QEMU domain status XML (/run/libvirt/qemu/*.xml)    might lose the

[Bug 2059272] Re: libvirt domain is not listed/managed after libvirt restart with messages "internal error: no monitor path" and "Failed to load config for domain"

Please include the "normal" execution from comment #5 in the test plan, besides the synthetic one. ** Changed in: libvirt (Ubuntu Jammy) Status: In Progress => Fix Committed ** Tags added: verification-needed verification-needed-jammy -- You received this bug notification because you

[Bug 2059272] Re: libvirt domain is not listed/managed after libvirt restart with messages "internal error: no monitor path" and "Failed to load config for domain"

Uploaded to Jammy. ** Description changed: [ Impact ] - * If a race condition occurs on libvirtd shutdown, -a QEMU domain status XML (/run/libvirt/qemu/*.xml) -might lose the QEMU-driver specific information, -such as ''. - - * On the next libvirtd startup, the parsing of

[Bug 2059272] Re: libvirt domain is not listed/managed after libvirt restart with messages "internal error: no monitor path" and "Failed to load config for domain"

OK, I tested the fixed package from your PPA and verified that it indeed solves the issue. Just to reiterate, then: LGTM, +1! -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2059272 Title: libvirt

[Bug 2059272] Re: libvirt domain is not listed/managed after libvirt restart with messages "internal error: no monitor path" and "Failed to load config for domain"

Test case for normal operations in this code path: Restart libvirt 100 times with 10 QEMU domains. All domains continued to be managed by libvirt. No error messages observed. $ for NAME in test-vm-{1..10}; do cat <<-EOF >test-vms.xml && virsh define test-vms.xml && virsh start $NAME; done

[Bug 2059272] Re: libvirt domain is not listed/managed after libvirt restart with messages "internal error: no monitor path" and "Failed to load config for domain"

Hah, I didn't see your comment before I posted mine. Mid air conflict! Anyway, thanks for providing the PPA. I'll take it for a spin :-). -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2059272

[Bug 2059272] Re: libvirt domain is not listed/managed after libvirt restart with messages "internal error: no monitor path" and "Failed to load config for domain"

Mauricio, Wow! Such an awesome reproducer. Very detailed and easy to follow. Thanks for providing it. I was able to verify the problem here, and looked at the upstream patch[1] that fixes it. The rationale makes sense to me, although that cleanup function is pretty involved and it's hard to

[Bug 2059272] Re: libvirt domain is not listed/managed after libvirt restart with messages "internal error: no monitor path" and "Failed to load config for domain"

Steps with test packages on Jammy --- Test packages built successfully in all architectures with -proposed enabled in Launchpad PPA mfo/lp2059272. https://launchpad.net/~mfo/+archive/ubuntu/lp2059272 Upgrade the libvirt packages and install debug symbols $ sudo add-apt-repository -y -c

[Bug 2059272] Re: libvirt domain is not listed/managed after libvirt restart with messages "internal error: no monitor path" and "Failed to load config for domain"

** Description changed: + [ Impact ] + + * If a race condition occurs on libvirtd shutdown, +a QEMU domain status XML (/run/libvirt/qemu/*.xml) +might lose the QEMU-driver specific information, +such as ''. + + * On the next libvirtd startup, the parsing of that +QEMU

[Bug 2059272] Re: libvirt domain is not listed/managed after libvirt restart with messages "internal error: no monitor path" and "Failed to load config for domain"

Steps to reproduce on Jammy --- Stop libvirt systemd units sudo systemctl stop 'libvirtd*' Start libvirt in GDB sudo gdb \ -iex 'set confirm off' \ -iex 'set pagination off' \ -iex 'set debuginfod enabled on' \ -iex 'set debuginfod urls