On 9/1/26 5:35 AM, Daniel P. Berrangé wrote:
On Tue, Sep 01, 2026 at 12:48:32PM +0200, Claudio Fontana wrote:
[...]
In one case I have seen two CPUs of the same generation, ~ same # of sockets,
same base clock,
(but slightly different CPU model), same hugepages settings, comparable NUMA
topology ...
differ in terms of QEMU process termination times by a factor of at least 4,
_presumeably_ due to
a combination of slower memory speeds coupled with heavy memory-intensive
workloads on the same socket.
Effectively there is no "right" number at all, because "right" means
wait long enough that a correctly operating QEMU exits, without waiting
so long that a QEMU stuck in uninterruptible sleep will hang the Destroy
API too long.
We had gone for a relatively short timeout to avoid hung QEMU's delaying
libvirt.
I think this is a worthy goal and we should avoid extending it if possible.
We also have VIR_DOMAIN_DESTROY_GRACEFUL flag which lets us send SIGTERM
but skip the SIGKILL part. IIUC the logic though, we still wait exactly
the same amount of time in total, just don't send KILL.
I am concerned that getting to the "right" number might prove impractical?
Effectively we'll be forced to over-estimate the worst case timeout
by a very significant factor, to minimize chance of false errors.
I wonder what behaviour suits mgmt apps best ?
If virDomainDestroy reaches the timeout and returns an error, it
doesn't mean cleanup has failed.
If a mgmt app is monitoring the running VM list, it will eventually
see the VM go away.
This is effectively the current behavior, which I think is fine.
IOW, an error from virDomainDestroy isn't really a serious error in
many cases - it is effectively more of a warning.
I suspect others, if encountering the EBUSY error, have treated it as a warning.
The biggest challenge I see is there there is no way to understand
if QEMU is "making progress" in shutting down, as opposed to become
stuck. If you look at the underlying process, if QEMU were stuck
in the "D" state that's bad, but if we've sent SIGKILL then it
should be reaped by the kernel in all other states, so it is just
a matter of waiting.
Right. And we actually already do that in the shutdown case, which btw was the
case for the reporter of the issue influencing this patch. Due to some
miscommunication, we mistakenly started chasing timeouts in the destroy
operation, when in fact the reporter (using libvirt 11.0.0) hit
https://gitlab.com/libvirt/libvirt/-/work_items/853
by issuing 'systemctl poweroff' within the guest. In the end, huge facepalm on
my part. Sorry for the disruption, but thanks for being vigilant against
unneeded hacks :-). Self NACK for this patch.
But perhaps not all time is wasted. During shutdown, a long-exiting QEMU could
exceed the timeout, causing qemuProcessBeginStopJob() to fail and return early
in processMonitorEOFEvent()
https://gitlab.com/libvirt/libvirt/-/blob/master/src/qemu/qemu_driver.c?ref_type=heads#L4187
None of the code from line 4190 through the rest of the function is executed.
When the QEMU process finally disappears, processShutdownCompletedEvent() is
eventually executed, where we only call qemuDomainRemoveInactive(). Do we miss
sending a lifecycle event in this case?
Regards,
Jim
