On Tue, Oct 25, 2022 at 09:00:33PM -0700, James J. Lippard wrote:
> On Tue, Oct 25, 2022 at 09:20:05PM -0500, Scott Cheloha wrote:
> > On Tue, Oct 25, 2022 at 02:24:24PM -0700, James J. Lippard wrote:
> > > I'm one of several people experiencing this issue with OpenBSD 7.2 on
> > > VMware ESXi 7.5. Scott C. has given me help in trying to track the issue
> > > down; a patched -current kernel to remove the acpi_delay code added in
> > > 7.2 makes the issue go away.
> >
> > Thanks for your report.
> >
> > I have one more patch for you to try. Attached at the end. Hopefully
> > it will confirm the root problem. Send the resulting dmesg and we'll
> > see whether the problem is actually the acpitimer(4).
>
> >[...]
> > Okay, here is the third patch. Revert the earlier one and boot this.
>
> Here's the dmesg output running with this new patch:
Thank you for testing, let's take a look.
> OpenBSD 7.2-current (GENERIC.MP) #1: Tue Oct 25 20:09:51 MST 2022
> lipp...@chaos.int.discord.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> [snip]
> acpitimer0 at acpi0: 3579545 Hz, 24 bits
> acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
> cpu0 at mainbus0: apid 0 (boot processor)
> cpu0: Intel(R) Xeon(R) CPU D-1528 @ 1.90GHz, 1899.77 MHz, 06-56-03
> cpu0:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,RDSEED,ADX,SMAP,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,ARAT,XSAVEOPT,MELTDOWN
> cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB
> 64b/line 8-way L2 cache, 9MB 64b/line 12-way L3 cache
> measure_tsc_freq: indirect calibration with acpitimer0(1000), 3579545 Hz:
> count 12840801 13198720 tsc 8350048970 8540029885 usecs 99990: 1899999149 Hz
> measure_tsc_freq: direct calibration with acpitimer0(1000), 3579545 Hz:
> cycles 357969 tsc 62919804: 629172553 Hz
> measure_tsc_freq: indirect calibration with acpitimer0(1000), 3579545 Hz:
> count 13562994 13686416 tsc 8608912502 8798895525 usecs 34479: (failed)
> measure_tsc_freq: indirect calibration with acpitimer0(1000), 3579545 Hz:
> count 13692684 14050605 tsc 8802222961 8992204988 usecs 99990: 1900010271 Hz
> measure_tsc_freq: direct calibration with acpitimer0(1000), 3579545 Hz:
> cycles 357969 tsc 64754894: 647522710 Hz
When we do "indirect calibration," we're using delay(9) to spin for
~100,000 microseconds in between reads of the reference timer and the
TSC. In this case, the underlying delay(9) implementation is
i8254_delay(). This method calibrates the TSC to 1900 MHz. For
example, in the first indirect calibration round we get:
(tsc2 - tsc1) * acpitimer-frequency / (acpitimer2 - acpitimer1)
= (8540029885 - 8350048970) * 3579545 / (13198720 - 12840801)
= 1899997581
or roughly 1900 MHz. The result printed in the dmesg (1899999149) is
a little different because the math in the kernel is a little
different. The third indirect calibration round yields basically the
same result (1900010271).
When we do "direct calibration," we're reading the reference timer
itself repeatedly to spin for ~100,000 microseconds and accumulating a
count of reference timer cycles and TSC cycles as we spin. This
method calibrates the TSC to ~630 MHz. For example, in the first
direct calibration round we get:
tsc-cycles * acpitimer-frequency / acpitimer-cycles
= 62919804 * 3579545 / 357969
= 629172553
or roughly 630 MHz. The second indirect calibration round yields a
similar result (647522710).
Based on these numbers, I think the virtual ACPI PM Timer on this ESXi
VM accelerates beyond 3579545 Hz when it is read repeatedly and then
decelerates back down to the nominal frequency when it is read less
frequently.
I don't think the TSC itself has a non-constant frequency. When we
calibrate it later with the HPET, both indirect calibration using the
local apic timer to spin and direct calibration using only the HPET
yield a TSC frequency of ~1900 MHz:
> [snip]
> cpu0: apic clock running at 65MHz
> delay_init: changing delay implementation: 0 -> 3000
(Here we switch from i8254_delay() to lapic_delay().)
> [snip]
> acpihpet0 at acpi0: 14318179 Hz
> measure_tsc_freq: indirect calibration with acpihpet0(1000), 14318179 Hz:
> count 7984 1439544 tsc 11218877272 11408843078 usecs 99981: 1900019063 Hz
> measure_tsc_freq: direct calibration with acpihpet0(1000), 14318179 Hz:
> cycles 1431817 tsc 189999744: 1899998634 Hz
> measure_tsc_freq: indirect calibration with acpihpet0(1000), 14318179 Hz:
> count 2894172 4325743 tsc 11601869571 11791837035 usecs 99982: 1900016642 Hz
> measure_tsc_freq: direct calibration with acpihpet0(1000), 14318179 Hz:
> cycles 1431826 tsc 190000912: 1899998371 Hz
> measure_tsc_freq: indirect calibration with acpihpet0(1000), 14318179 Hz:
> count 5780812 7212468 tsc 11984921805 12174900576 usecs 99988: 1900015711 Hz
> measure_tsc_freq: direct calibration with acpihpet0(1000), 14318179 Hz:
> cycles 1431877 tsc 190007695: 1899998525 Hz
And here you can see the results of both indirect and direct
calibration. In all six rounds we get about ~1900 MHz.
--
I don't know how to explain this. Maybe another developer will read
this and spot something I'm missing. Or maybe this is a known issue
and I'm just not finding a reference to it online.
The simplest workaround is to skip installing acpitimer_delay() with
delay_init() during acpitimerattach(). The attached patch does this.
I don't know if this problem persists after boot. If it does, using
the acpitimer0 timecounter may yield strange results in the VM. I
recommend not using the acpitimer0 timecounter until the problem is
better understood. A calibrated TSC is going to be a better
timecounter anyway.
There might be a second workaround. Kalabic mentions here in the
other thread about this problem:
https://marc.info/?l=openbsd-bugs&m=166664949825616&w=2
... that changing the ESXi option "Guest OS Version" from "FreeBSD
(32-bit)" to "FreeBSD (64-bit)" seemed to fix the problem on his
version of ESXi. Does that work for you? I don't know what the other
consequences of that configuration change are, but it might be worth a
try if you prefer to run 7.2-RELEASE or 7.2-STABLE instead of patching
-current.
Do you have VMware support? Is there any way for you to report this
problem to them? It's unlikely they explicitly support running an
OpenBSD guest, but it's plausible this issue could affect other
operating systems. I can't imagine OpenBSD is reading the ACPI PM
timer differently than Linux or FreeBSD.
Index: dev/acpi/acpitimer.c
===================================================================
RCS file: /cvs/src/sys/dev/acpi/acpitimer.c,v
retrieving revision 1.16
diff -u -p -r1.16 acpitimer.c
--- dev/acpi/acpitimer.c 25 Aug 2022 17:43:34 -0000 1.16
+++ dev/acpi/acpitimer.c 26 Oct 2022 09:13:45 -0000
@@ -102,8 +102,6 @@ acpitimerattach(struct device *parent, s
acpi_timecounter.tc_name = sc->sc_dev.dv_xname;
tc_init(&acpi_timecounter);
- delay_init(acpitimer_delay, 1000);
-
#if defined(__amd64__)
extern void cpu_recalibrate_tsc(struct timecounter *);
cpu_recalibrate_tsc(&acpi_timecounter);
