On Tue, Sep 01, 2026 at 12:48:32PM +0200, Claudio Fontana wrote:
> On 9/1/26 11:35, Daniel P. Berrangé wrote:
> > On Tue, Sep 01, 2026 at 10:11:04AM +0200, Claudio Fontana wrote:
> >> On 9/1/26 00:53, Jim Fehlig wrote:
> >>> On 8/31/26 9:34 AM, Peter Krempa wrote:
> >>>> On Fri, Aug 28, 2026 at 14:16:17 -0600, Jim Fehlig via Devel wrote:
> >>>>> From: Jim Fehlig <[email protected]>
> >>>>>
> >>>>> When shutting down a VM, libvirt sends the associated QEMU process
> >>>>> SIGTERM, waits 10 seconds for it to disappear, optionally sends SIGKILL,
> >>>>> then waits up to another 30 seconds for the process to exit before
> >>>>> reporting and returning an error. Commit be2ca04447 added 2 seconds per
> >>>>> assigned host device to the total time libvirt waits for a QEMU process
> >>>>> to terminate. Other scenarios than the one described in be2ca04447 could
> >>>>> delay the clean exit of QEMU, e.g. reclaiming memory of VMs with large
> >>>>> memory allocations backed by 4k pages on the host.
> >>>>>
> >>>>> Instead of trying to cover all such scenarios based on VM configuration,
> >>>>> introduce a 'process_exit_wait' setting in qemu.conf to control how much
> >>>>> additional time (in seconds) libvirt will wait for a QEMU process to
> >>>>> terminate before reporting an error.
> >>>>
> >>>> You are stating that an error is reported. Can you please elaborate when
> >>>> you are seeing such an error?
> >>>
> >>> When e.g. destroying a 900GB VM whose memory is backed by 4k pages on the
> >>> host
> >>
> >> We recommend to use hugepages to back large VMs, but for flexibility
> >> reasons and other
> >> constraints, users end up with these mid to large size VMs backed by
> >> normal memory.
> >>
> >> We have reports of this occuring on NUMA servers with even smaller VMs,
> >> f.e 512GiB.
> >>
> >> Normally with a fairly modern enterprise CPU and RAM this can take ~25s to
> >> terminate the QEMU process in isolation, but in certain conditions this
> >> can increase
> >> up to the 40 seconds.
> >>
> >> This was fine up until some time ago since the timeout was bugged and did
> >> not trigger: as the timeout was fixed, users started seeing their use
> >> cases break.
> >
> > FWIW, IBM developed a feature in QEMU/libvirt to deal with this
> >
> > <async-teardown enabled='yes'/>
> >
> > an implication of this approach though is that the RAM is not immediately
> > available for launching a new guest, and of course it requires per-guest
> > config knobs to be enabled.
> >
> >
> >>>>> diff --git a/src/qemu/qemu.conf.in b/src/qemu/qemu.conf.in
> >>>>> index 97b0141cf6..2f24059053 100644
> >>>>> --- a/src/qemu/qemu.conf.in
> >>>>> +++ b/src/qemu/qemu.conf.in
> >>>>> @@ -844,6 +844,18 @@
> >>>>> #max_threads_per_process = 0
> >>>>>
> >>>>>
> >>>>> +# When shutting down a VM, libvirt will wait up to 40 seconds for
> >>>>> +# the associated QEMU process to exit before reporting an error.
> >>>>> +# For some VM configurations, QEMU might require more time to
> >>>>> +# cleanup and exit, e.g. VMs with very large memory allocations.
> >>>>> +#
> >>>>> +# If process_exit_wait is set to a positive interger, libvirt
> >>>>> +# will use the value as additional time to wait for the QEMU
> >>>>> +# process to exit before reporting it cannot be terminated.
> >>>>> +#
> >>>>> +#process_exit_wait = 0
> >>>>
> >>>> I don't like this as a global option. The specifics of a VM can be
> >>>> vastly different and setting this globally will possibly be unable to
> >>>> satisfy new VMs without restart of the daemon.
> >>
> >> FWIW, in my view it would be both more convenient and more meaningful
> >> for the admin to configure this as a single global parameter when planning
> >> the host configuration, resources and workloads.
> >>
> >> A restart of the daemon if the admin doesn't get it "right" the first time
> >> is not that bad imo, existing VMs continue to run, and libvirt daemons
> >> restart fairly quickly.
> >>
> >> The thing to keep in mind for an admin / provisioning rule / agent /... is
> >> the max size of a normal memory backed VM that could land on this host,
> >> and the worst case pressure the workloads on the host / socket can generate
> >> on the memory system, given the host architecture and topology.
> >
> > The whole TERM, wait 10 seconds, KILL, wait 30 seconds approach was
> > designed from the POV that a normally behaving QEMU will "die" very
> > quickly. IOW, any scenario where we reached the KILL stage was almost
> > certainly a broken QEMU/kernel in some respect.
> >
> > Clearly this is no longer a valid assumption. When "normal" behaviour
> > or QEMU no longer matches libvirt's default mgmt action behaviour
> > then I don't think a global qemu.conf setting or a per-VM setting is
> > the ideal approach.
> >
> > We need to ensure libvirt "does the right thing" out of the box, as
> > best as we can.
>
> Hi Daniel,
>
> in principle I agree it would be the best outcome, just seems hard to get it
> right.
>
> >
> > IMHO, this suggests we need to dynamically increase our wait time
> > before KILL based on the guest RAM size. eg Add 5 seconds for each
> > 100 GB of small page RAM. I pulled that number out of the air,
> > you would need to pick something better based on a typical system,
> > plus some buffer/fuzz.
> >
> >
> > Also I've noticed that TDX guests are painfully slow to teardown,
> > even with tiny RAM sizes. So we might need to increase wait times
> > even more when using TDX.
>
> Indeed. And like this there are many more factors that can change the result
> substantially (hugepages settings, NUMA, cpu and load state @memory
> controller, ...).
Looking at the code I find we already introduced an extension
for PCI devidces
/* Request an extra delay of two seconds per current nhostdevs
* to be safe against stalls by the kernel freeing up the resources */
ret = virProcessKillPainfullyDelay(vm->pid,
!!(flags & VIR_QEMU_PROCESS_KILL_FORCE),
vm->def->nhostdevs * 2,
false);
>
> Each one of these factors would need to be extracted or sampled and
> multiplied for the total RAM size.
>
> In one case I have seen two CPUs of the same generation, ~ same # of sockets,
> same base clock,
> (but slightly different CPU model), same hugepages settings, comparable NUMA
> topology ...
> differ in terms of QEMU process termination times by a factor of at least 4,
> _presumeably_ due to
> a combination of slower memory speeds coupled with heavy memory-intensive
> workloads on the same socket.
Effectively there is no "right" number at all, because "right" means
wait long enough that a correctly operating QEMU exits, without waiting
so long that a QEMU stuck in uninterruptible sleep will hang the Destroy
API too long.
We had gone for a relatively short timeout to avoid hung QEMU's delaying
libvirt.
We also have VIR_DOMAIN_DESTROY_GRACEFUL flag which lets us send SIGTERM
but skip the SIGKILL part. IIUC the logic though, we still wait exactly
the same amount of time in total, just don't send KILL.
> I am concerned that getting to the "right" number might prove impractical?
Effectively we'll be forced to over-estimate the worst case timeout
by a very significant factor, to minimize chance of false errors.
I wonder what behaviour suits mgmt apps best ?
If virDomainDestroy reaches the timeout and returns an error, it
doesn't mean cleanup has failed.
If a mgmt app is monitoring the running VM list, it will eventually
see the VM go away.
IOW, an error from virDomainDestroy isn't really a serious error in
many cases - it is effectively more of a warning.
The biggest challenge I see is there there is no way to understand
if QEMU is "making progress" in shutting down, as opposed to become
stuck. If you look at the underlying process, if QEMU were stuck
in the "D" state that's bad, but if we've sent SIGKILL then it
should be reaped by the kernel in all other states, so it is just
a matter of waiting.
Should we offer a VIR_DOMAIN_DESTROY_ASYNC flag such that
* ASYNC -> send TERM, wait, send KILL, return success
* ASYNC|GRACEFUL -> send TERM, wait, return success
with the intent that applications simply monitor the VM list for
ongoing status ?
If it is stuck in "D" state it'll never go away, but that's something
we can never solve in libvirt. If it is in any other state then as
long as GRACEFUL was not set, you can be confident QEMU will go away
eventually.
With regards,
Daniel
--
|: https://berrange.com ~~ https://hachyderm.io/@berrange :|
|: https://libvirt.org ~~ https://entangle-photo.org :|
|: https://pixelfed.art/berrange ~~ https://fstop138.berrange.com :|
