DV <[email protected]> writes: > Hi Vitaly, Alex, all, > > On Mon, 26 Jan 2026, Vitaly Kuznetsov wrote: >> My bet goes to the clocksource switch, e.g. the guest disables (or >> just stops using, good luck detecting that :-) ) TSC page and uses >> raw TSC for some period or something. > > I think I've hit a close relative of this, with a reproducer that > works on every boot. A Windows guest running its own Hyper-V computes > direct-mode stimer deadlines from the raw TSC, while KVM's reference > counter lags the TSC because of how QEMU restores kvmclock. Here the > lag is positive, so the guest's timers fire late instead of storming. > > Setup > ----- > > Host: Intel Core Ultra 9 285K (invariant TSC, 3686.4 MHz), Linux > 7.3.0, QEMU 10.2.4 (pc-q35-10.2), libvirt 12.2.0. > > Guest: Windows 11 Pro 26H2 (build 26300.9711), 8 vCPUs, with > virtualization-based security on. So Windows runs as the root > partition on its own Hyper-V (L1), nested on KVM. > > -cpu host,migratable=off,hv-time=on,hv-relaxed=on,hv-vapic=on, > hv-spinlocks=0xfff,hv-vpindex=on,hv-runtime=on,hv-synic=on, > hv-stimer=on,hv-stimer-direct=on,hv-frequencies=on, > hv-tlbflush=on,hv-tlbflush-direct=on,hv-tlbflush-ext=on, > hv-evmcs=on,hv-avic=on,hv-emsr-bitmap=on,hv-xmm-input=on,... > > Symptom > ------- > > Every timer inside Windows fired 0.3-0.7 s late. A 1 ms wait on a > high-resolution waitable timer took up to 466 ms, and an indirect > display driver paced only by a timer refreshed at 6-8 Hz instead of > 120. Things only moved when some other interrupt woke a vCPU. > > What L1 does > ------------ > > With both hv-stimer-direct and hv-frequencies, L1 arms stimer0 in > direct mode (config 0x1ef8: vector 0xef, direct, auto-enable) and > writes absolute counts. Call X the amount by which the reference > counter lags guest TSC / HV_X64_MSR_TSC_FREQUENCY. L1's counts are > X too far out. A timer that should fire right away looks like this: > > kvm_hv_stimer_start_one_shot: vcpu_id 6 timer 0 > time_now 40095770231 count 40100417164 > > That count is 464.69 ms ahead of time_now, and X on that boot was > 464.7 ms. L1 writes the count again on every L2 timer exit, so such a > timer keeps moving to now + X and only fires once the vCPU is quiet > for that long. > > Without hv-stimer-direct, without hv-frequencies, or without > hv-stimer, L1 uses the local APIC timer instead (TMICT/TDCR writes, > a relative countdown), and all timers are on time. I've turned direct > mode off as a workaround: a 1 ms wait is now late by at most 1.24 ms. > > Why the reference counter lags > ------------------------------ > > get_time_ref_counter() follows kvmclock on both paths. The fallback > divides get_kvmclock_ns() by 100, and compute_tsc_page_parameters() > takes the TSC page's tsc_offset from the pvclock system_time. > > QEMU's kvmclock_realize() calls KVM_GET_CLOCK when the VM is created. > kvmclock_vm_state_change() writes the saved value back with > KVM_SET_CLOCK when the VM starts running, and saves a new value > whenever it stops. So kvmclock, and with it the reference counter, > stands still from creation to first run and during every pause. The > guest TSC keeps counting through both. On six boots X was 297, 465, > 570, 495, 669 and 512 ms. > > Two measurements tie X to this: > > - In a trace from QEMU startup, extrapolated back to zero, the guest > TSC started at host monotonic 265208.1505 s and the reference > counter at 265208.8205 s, 0.670 s later. X on that boot was > 669.2 ms. > > - virsh suspend for 1.045 s, then resume: X grew from 669.2 to > 1689.3 ms (+1020.1 ms), and L1's far stimer deadline moved out by > the same amount. > > hv-evmcs off (X = 570.1 ms) and hv-reenlightenment on (X = 494.7 ms) > didn't change the behaviour. Host load and power management were > ruled out too. > > Connection to your thread > ------------------------- > > If the reference counter were ahead of TSC time instead, L1's > deadlines would land in the past. stimer_start() expires a one-shot > whose count is in the past at once, L1 arms it again on its next > exit, and that would look like the expired-stimer storm Alex's patch > delays. > > I haven't been able to try that here, because libvirt is set up to > start a fresh QEMU whenever the guest reboots. But a reset inside the > same QEMU process sets the guest TSC back to 0 while kvmclock keeps > counting, so I'd expect it to flip the sign, possibly by the whole > previous uptime. A resume path that restores the two clocks > differently could do the same. > > Questions > --------- > > 1. Should KVM keep the Hyper-V reference counter equal to guest TSC / > TSC frequency when hv-frequencies is exposed? L1 seems to assume > it is. > > 2. If so, where does the fix belong: in QEMU's kvmclock save/restore, > or in KVM deriving the reference counter (and TSC page) from the > guest TSC? > > 3. If not, should hv-stimer-direct be documented as unsafe for guests > that run their own Hyper-V? >
>From what I remember, direct mode synthetic timers were added specifically to aid Hyper-V-on-KVM case as non-nested Windows was preferring non-direct version anyway. So I believe we need a fix and not 3). I'm wondering if the issue is somewhat related to the nasty hack we put in commit 0469f2f7ab4c6a6cae4b74c4f981c4da6d909411 Author: Vitaly Kuznetsov <[email protected]> Date: Tue Mar 16 15:37:36 2021 +0100 KVM: x86: hyper-v: Don't touch TSC page values when guest opted for re-enlightenment but maybe hibernation is somewhat different. Or maybe we misunderstood the problem in the first place and only fixed one sympthom of it? > I can share the full traces and the Windows-side timer and frame > probes if they'd help. > > Claude Code, an AI coding assistant, did the tracing and analysis > while I was looking into why the VM's virtual display ran at a few > frames per second. > > Thanks, > David > -- Vitaly
