On 8/31/26 9:34 AM, Peter Krempa wrote:
On Fri, Aug 28, 2026 at 14:16:17 -0600, Jim Fehlig via Devel wrote:
From: Jim Fehlig <[email protected]>
When shutting down a VM, libvirt sends the associated QEMU process
SIGTERM, waits 10 seconds for it to disappear, optionally sends SIGKILL,
then waits up to another 30 seconds for the process to exit before
reporting and returning an error. Commit be2ca04447 added 2 seconds per
assigned host device to the total time libvirt waits for a QEMU process
to terminate. Other scenarios than the one described in be2ca04447 could
delay the clean exit of QEMU, e.g. reclaiming memory of VMs with large
memory allocations backed by 4k pages on the host.
Instead of trying to cover all such scenarios based on VM configuration,
introduce a 'process_exit_wait' setting in qemu.conf to control how much
additional time (in seconds) libvirt will wait for a QEMU process to
terminate before reporting an error.
You are stating that an error is reported. Can you please elaborate when
you are seeing such an error?
When e.g. destroying a 900GB VM whose memory is backed by 4k pages on the host
# virsh destroy 4ff7e1ea-a4d4-458b-8b27-a7b67d7b8011
error: Failed to destroy domain '4ff7e1ea-a4d4-458b-8b27-a7b67d7b8011'
error: Failed to terminate process 71785 with SIGKILL: Device or resource busy
Signed-off-by: Jim Fehlig <[email protected]>
---
src/qemu/libvirtd_qemu.aug | 1 +
src/qemu/qemu.conf.in | 12 ++++++++++++
src/qemu/qemu_conf.c | 3 +++
src/qemu/qemu_conf.h | 1 +
src/qemu/qemu_process.c | 13 ++++++++++---
src/qemu/test_libvirtd_qemu.aug.in | 1 +
6 files changed, 28 insertions(+), 3 deletions(-)
diff --git a/src/qemu/libvirtd_qemu.aug b/src/qemu/libvirtd_qemu.aug
index 311992e441..dc3d6d61df 100644
--- a/src/qemu/libvirtd_qemu.aug
+++ b/src/qemu/libvirtd_qemu.aug
@@ -132,6 +132,7 @@ module Libvirtd_qemu =
| bool_entry "dump_guest_core"
| str_entry "stdio_handler"
| int_entry "max_threads_per_process"
+ | int_entry "process_exit_wait"
| str_entry "sched_core"
let device_entry = bool_entry "mac_filter"
diff --git a/src/qemu/qemu.conf.in b/src/qemu/qemu.conf.in
index 97b0141cf6..2f24059053 100644
--- a/src/qemu/qemu.conf.in
+++ b/src/qemu/qemu.conf.in
@@ -844,6 +844,18 @@
#max_threads_per_process = 0
+# When shutting down a VM, libvirt will wait up to 40 seconds for
+# the associated QEMU process to exit before reporting an error.
+# For some VM configurations, QEMU might require more time to
+# cleanup and exit, e.g. VMs with very large memory allocations.
+#
+# If process_exit_wait is set to a positive interger, libvirt
+# will use the value as additional time to wait for the QEMU
+# process to exit before reporting it cannot be terminated.
+#
+#process_exit_wait = 0
I don't like this as a global option. The specifics of a VM can be
vastly different and setting this globally will possibly be unable to
satisfy new VMs without restart of the daemon.
I'm having difficulty understanding how that is possible...
This ... if required ... should be a per-VM setting.
Any suggestions on where such setting would be added? A 'destroy_timeout'
attribute on the <on_poweroff> element could be a possiblity. But I look at it
as encoding why the process exited, now how libvirt controls the exit. The only
other idea I have is a top-level 'destroy_timeout' element. E.g.
<domain>
...
<destroy_timeout seconds='120'/>
</domain>
+
+
# If max_core is set to a non-zero integer, then QEMU will be
# permitted to create core dumps when it crashes, provided its
# RAM size is smaller than the limit set.
diff --git a/src/qemu/qemu_conf.c b/src/qemu/qemu_conf.c
index e30b146634..adcba137cc 100644
--- a/src/qemu/qemu_conf.c
+++ b/src/qemu/qemu_conf.c
@@ -934,6 +934,9 @@ virQEMUDriverConfigLoadProcessEntry(virQEMUDriverConfig
*cfg,
cfg->schedCore = val;
}
+ if (virConfGetValueUInt(conf, "process_exit_wait", &cfg->processExitWait) < 0)
+ return -1;
+
return 0;
}
diff --git a/src/qemu/qemu_conf.h b/src/qemu/qemu_conf.h
index 1d29f35c5d..bc78982123 100644
--- a/src/qemu/qemu_conf.h
+++ b/src/qemu/qemu_conf.h
@@ -211,6 +211,7 @@ struct _virQEMUDriverConfig {
unsigned int maxProcesses;
unsigned int maxFiles;
unsigned int maxThreadsPerProc;
+ unsigned int processExitWait;
unsigned long long maxCore;
bool dumpGuestCore;
diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c
index b2506edce0..5042ffe339 100644
--- a/src/qemu/qemu_process.c
+++ b/src/qemu/qemu_process.c
@@ -9218,6 +9218,9 @@ qemuProcessInShutdownStartMonitor(virDomainObj *vm)
int
qemuProcessKill(virDomainObj *vm, unsigned int flags)
{
+ qemuDomainObjPrivate *priv = vm->privateData;
+ g_autoptr(virQEMUDriverConfig) cfg = virQEMUDriverGetConfig(priv->driver);
+ unsigned int delay = cfg->processExitWait;
int ret = -1;
VIR_DEBUG("vm=%p name=%s pid=%lld flags=0x%x",
@@ -9238,11 +9241,15 @@ qemuProcessKill(virDomainObj *vm, unsigned int flags)
return 0;
}
- /* Request an extra delay of two seconds per current nhostdevs
- * to be safe against stalls by the kernel freeing up the resources */
+ /* If the administrator has not requested an explicit shutdown wait period,
+ * add an extra delay of two seconds per current nhostdevs to be safe
against
+ * stalls by the kernel freeing up the resources */
+ if (delay == 0)
+ delay = vm->def->nhostdevs * 2;
+
ret = virProcessKillPainfullyDelay(vm->pid,
!!(flags &
VIR_QEMU_PROCESS_KILL_FORCE),
- vm->def->nhostdevs * 2,
+ delay,
false);
Can you please elaborate what the problem is?
Hitting the following error after waiting 40s for a qemu process to disappear
https://gitlab.com/libvirt/libvirt/-/blob/master/src/util/virprocess.c?ref_type=heads#L435
This function has additional code after this which waits for the process
to terminate on background if killing didn't work so the process should
be cleaned up.
We've observed qemu process cleanup/exit times exceeding the 40s total wait
time.
Adding extra delay will not cover all cases so that's why I'm not a fan
of another tunable which users don't have a reasonable way of setting.
Personally, I felt a qemu.conf option was a reasonable approach. It maintains
existing behavior and doesn't affect processes that exit within 40s, yet gives
users a mechansim to handle those with long cleanup/exit times.
I'm fine making this a per-VM setting if preferred. Also open to other options I
may be overlooking, although simply increasing the existing hardcoded value is
less appealing :-).
Regards,
Jim
