On Fri, Sep 04, 2026 at 03:07:30PM -0600, Jim Fehlig wrote: > On 9/1/26 5:35 AM, Daniel P. Berrangé wrote: > > On Tue, Sep 01, 2026 at 12:48:32PM +0200, Claudio Fontana wrote: > [...] > > > In one case I have seen two CPUs of the same generation, ~ same # of > > > sockets, same base clock, > > > (but slightly different CPU model), same hugepages settings, comparable > > > NUMA topology ... > > > differ in terms of QEMU process termination times by a factor of at least > > > 4, _presumeably_ due to > > > a combination of slower memory speeds coupled with heavy memory-intensive > > > workloads on the same socket. > > > > Effectively there is no "right" number at all, because "right" means > > wait long enough that a correctly operating QEMU exits, without waiting > > so long that a QEMU stuck in uninterruptible sleep will hang the Destroy > > API too long. > > > > We had gone for a relatively short timeout to avoid hung QEMU's delaying > > libvirt. > > I think this is a worthy goal and we should avoid extending it if possible. > > > We also have VIR_DOMAIN_DESTROY_GRACEFUL flag which lets us send SIGTERM > > but skip the SIGKILL part. IIUC the logic though, we still wait exactly > > the same amount of time in total, just don't send KILL. > > > > > I am concerned that getting to the "right" number might prove impractical? > > > > Effectively we'll be forced to over-estimate the worst case timeout > > by a very significant factor, to minimize chance of false errors. > > > > > > I wonder what behaviour suits mgmt apps best ? > > > > > > If virDomainDestroy reaches the timeout and returns an error, it > > doesn't mean cleanup has failed. > > > > If a mgmt app is monitoring the running VM list, it will eventually > > see the VM go away. > > This is effectively the current behavior, which I think is fine. > > > > > IOW, an error from virDomainDestroy isn't really a serious error in > > many cases - it is effectively more of a warning. > > I suspect others, if encountering the EBUSY error, have treated it as a > warning. > > > > > The biggest challenge I see is there there is no way to understand > > if QEMU is "making progress" in shutting down, as opposed to become > > stuck. If you look at the underlying process, if QEMU were stuck > > in the "D" state that's bad, but if we've sent SIGKILL then it > > should be reaped by the kernel in all other states, so it is just > > a matter of waiting. > > Right. And we actually already do that in the shutdown case, which btw was > the case for the reporter of the issue influencing this patch. Due to some > miscommunication, we mistakenly started chasing timeouts in the destroy > operation, when in fact the reporter (using libvirt 11.0.0) hit > > https://gitlab.com/libvirt/libvirt/-/work_items/853 > > by issuing 'systemctl poweroff' within the guest. In the end, huge facepalm > on my part. Sorry for the disruption, but thanks for being vigilant against > unneeded hacks :-). Self NACK for this patch. > > But perhaps not all time is wasted. During shutdown, a long-exiting QEMU > could exceed the timeout, causing qemuProcessBeginStopJob() to fail and > return early in processMonitorEOFEvent() > > https://gitlab.com/libvirt/libvirt/-/blob/master/src/qemu/qemu_driver.c?ref_type=heads#L4187 > > None of the code from line 4190 through the rest of the function is > executed. When the QEMU process finally disappears, > processShutdownCompletedEvent() is eventually executed, where we only call > qemuDomainRemoveInactive(). Do we miss sending a lifecycle event in this > case?
Not sure, it would have to be tested to validate it. With regards, Daniel -- |: https://berrange.com ~~ https://hachyderm.io/@berrange :| |: https://libvirt.org ~~ https://entangle-photo.org :| |: https://pixelfed.art/berrange ~~ https://fstop138.berrange.com :|
