Re: 7.2: tsc timecounter running too fast on ESXi 7.5
On Tue, Oct 25, 2022 at 09:20:05PM -0500, Scott Cheloha wrote: > On Tue, Oct 25, 2022 at 02:24:24PM -0700, James J. Lippard wrote: > > I'm one of several people experiencing this issue with OpenBSD 7.2 on > > VMware ESXi 7.5. Scott C. has given me help in trying to track the issue > > down; a patched -current kernel to remove the acpi_delay code added in > > 7.2 makes the issue go away. > > Thanks for your report. > > I have one more patch for you to try. Attached at the end. Hopefully > it will confirm the root problem. Send the resulting dmesg and we'll > see whether the problem is actually the acpitimer(4). >[...] > Okay, here is the third patch. Revert the earlier one and boot this. Here's the dmesg output running with this new patch: OpenBSD 7.2-current (GENERIC.MP) #1: Tue Oct 25 20:09:51 MST 2022 lipp...@chaos.int.discord.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 6424494080 (6126MB) avail mem = 6212374528 (5924MB) random: good seed from bootblocks mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xe0010 (242 entries) bios0: vendor Phoenix Technologies LTD version "6.00" date 11/12/2020 bios0: VMware, Inc. VMware Virtual Platform acpi0 at bios0: ACPI 4.0 acpi0: sleep states S0 S1 S4 S5 acpi0: tables DSDT FACP BOOT APIC MCFG SRAT HPET WAET acpi0: wakeup devices PCI0(S3) USB_(S1) P2P0(S3) S1F0(S3) S2F0(S3) S8F0(S3) S16F(S3) S18F(S3) S22F(S3) S23F(S3) S24F(S3) S25F(S3) PE40(S3) S1F0(S3) PE50(S3) S1F0(S3) [...] acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Xeon(R) CPU D-1528 @ 1.90GHz, 1899.77 MHz, 06-56-03 cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,RDSEED,ADX,SMAP,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,ARAT,XSAVEOPT,MELTDOWN cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB 64b/line 8-way L2 cache, 9MB 64b/line 12-way L3 cache measure_tsc_freq: indirect calibration with acpitimer0(1000), 3579545 Hz: count 12840801 13198720 tsc 8350048970 8540029885 usecs 0: 189149 Hz measure_tsc_freq: direct calibration with acpitimer0(1000), 3579545 Hz: cycles 357969 tsc 62919804: 629172553 Hz measure_tsc_freq: indirect calibration with acpitimer0(1000), 3579545 Hz: count 13562994 13686416 tsc 8608912502 8798895525 usecs 34479: (failed) measure_tsc_freq: indirect calibration with acpitimer0(1000), 3579545 Hz: count 13692684 14050605 tsc 880961 8992204988 usecs 0: 1900010271 Hz measure_tsc_freq: direct calibration with acpitimer0(1000), 3579545 Hz: cycles 357969 tsc 64754894: 647522710 Hz cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges cpu0: apic clock running at 65MHz delay_init: changing delay implementation: 0 -> 3000 cpu1 at mainbus0: apid 2 (application processor) cpu1: Intel(R) Xeon(R) CPU D-1528 @ 1.90GHz, 1899.68 MHz, 06-56-03 cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,RDSEED,ADX,SMAP,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,ARAT,XSAVEOPT,MELTDOWN cpu1: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB 64b/line 8-way L2 cache, 9MB 64b/line 12-way L3 cache cpu1: smt 0, core 0, package 2 ioapic0 at mainbus0: apid 1 pa 0xfec0, version 20, 24 pins acpimcfg0 at acpi0 acpimcfg0: addr 0xf000, bus 0-127 acpihpet0 at acpi0: 14318179 Hz measure_tsc_freq: indirect calibration with acpihpet0(1000), 14318179 Hz: count 7984 1439544 tsc 11218877272 11408843078 usecs 99981: 1900019063 Hz measure_tsc_freq: direct calibration with acpihpet0(1000), 14318179 Hz: cycles 1431817 tsc 18744: 188634 Hz measure_tsc_freq: indirect calibration with acpihpet0(1000), 14318179 Hz: count 2894172 4325743 tsc 11601869571 11791837035 usecs 99982: 1900016642 Hz measure_tsc_freq: direct calibration with acpihpet0(1000), 14318179 Hz: cycles 1431826 tsc 19912: 188371 Hz measure_tsc_freq: indirect calibration with acpihpet0(1000), 14318179 Hz: count 5780812 7212468 tsc 11984921805 12174900576 usecs 99988: 1900015711 Hz measure_tsc_freq: direct calibration with acpihpet0(1000), 14318179 Hz: cycles 1431877 tsc 190007695: 188525 Hz acpiprt0 at acpi0: bus 0 (PCI0) acpipci0 at acpi0 PCI0: 0x 0x0011 0x0001 acpicmos0 at acpi0 "PNP0A05" at acpi0 not configured acpiac0 at acpi0: AC unit online acpicpu0 at acpi0: C1(@1 halt!) acpicpu1 at acpi0: C1(@1 halt!) cpu0: using VERW MDS workaround pvbus0 at
Re: 7.2: tsc timecounter running too fast on ESXi 7.5
On Tue, Oct 25, 2022 at 02:24:24PM -0700, James J. Lippard wrote: > I'm one of several people experiencing this issue with OpenBSD 7.2 on > VMware ESXi 7.5. Scott C. has given me help in trying to track the issue > down; a patched -current kernel to remove the acpi_delay code added in > 7.2 makes the issue go away. Thanks for your report. I have one more patch for you to try. Attached at the end. Hopefully it will confirm the root problem. Send the resulting dmesg and we'll see whether the problem is actually the acpitimer(4). > Below is output from sysctl machdep, sysctl hw, and dmesg: > > [...] > machdep.tscfreq=1900013052 > machdep.invarianttsc=1 This is probably the true TSC frequency... > sysctl hw: > hw.machine=amd64 > hw.model=Intel(R) Xeon(R) CPU D-1528 @ 1.90GHz ... it would match the nominal CPU frequency listed in the CPU string. I'm going to snip some of these dmesgs for reference, too. > dmesg (which includes 7.1, post-upgrade, and with patched -current > kernel): > > OpenBSD 7.1 (GENERIC.MP) #3: Sun May 15 10:27:01 MDT 2022 > > r...@syspatch-71-amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP > real mem = 6424494080 (6126MB) > avail mem = 6212485120 (5924MB) > random: good seed from bootblocks > mpath0 at root > scsibus0 at mpath0: 256 targets > mainbus0 at root > bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xe0010 (242 entries) > bios0: vendor Phoenix Technologies LTD version "6.00" date 11/12/2020 > bios0: VMware, Inc. VMware Virtual Platform > acpi0 at bios0: ACPI 4.0 > acpi0: sleep states S0 S1 S4 S5 > acpi0: tables DSDT FACP BOOT APIC MCFG SRAT HPET WAET > acpi0: wakeup devices PCI0(S3) USB_(S1) P2P0(S3) S1F0(S3) S2F0(S3) S8F0(S3) > S16F(S3) S18F(S3) S22F(S3) S23F(S3) S24F(S3) S25F(S3) PE40(S3) S1F0(S3) > PE50(S3) S1F0(S3) [...] > acpitimer0 at acpi0: 3579545 Hz, 24 bits > acpimadt0 at acpi0 addr 0xfee0: PC-AT compat > cpu0 at mainbus0: apid 0 (boot processor) > cpu0: Intel(R) Xeon(R) CPU D-1528 @ 1.90GHz, 1899.75 MHz, 06-56-03 > cpu0: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,RDSEED,ADX,SMAP,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,ARAT,XSAVEOPT,MELTDOWN > cpu0: 256KB 64b/line 8-way L2 cache > cpu0: smt 0, core 0, package 0 > mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges > cpu0: apic clock running at 65MHz > cpu1 at mainbus0: apid 2 (application processor) > cpu1: Intel(R) Xeon(R) CPU D-1528 @ 1.90GHz, 1899.62 MHz, 06-56-03 > cpu1: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,RDSEED,ADX,SMAP,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,ARAT,XSAVEOPT,MELTDOWN > cpu1: 256KB 64b/line 8-way L2 cache > cpu1: disabling user TSC (skew=-2507) > cpu1: smt 0, core 0, package 2 > ioapic0 at mainbus0: apid 1 pa 0xfec0, version 20, 24 pins > acpimcfg0 at acpi0 > acpimcfg0: addr 0xf000, bus 0-127 > acpihpet0 at acpi0: 14318179 Hz 7.1-release. This dmesg worked right. > OpenBSD 7.2 (GENERIC.MP) #758: Tue Sep 27 11:57:54 MDT 2022 > dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP > real mem = 6424494080 (6126MB) > avail mem = 6212378624 (5924MB) > random: good seed from bootblocks > mpath0 at root > scsibus0 at mpath0: 256 targets > mainbus0 at root > bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xe0010 (242 entries) > bios0: vendor Phoenix Technologies LTD version "6.00" date 11/12/2020 > bios0: VMware, Inc. VMware Virtual Platform > acpi0 at bios0: ACPI 4.0 > acpi0: sleep states S0 S1 S4 S5 > acpi0: tables DSDT FACP BOOT APIC MCFG SRAT HPET WAET > acpi0: wakeup devices PCI0(S3) USB_(S1) P2P0(S3) S1F0(S3) S2F0(S3) S8F0(S3) > S16F(S3) S18F(S3) S22F(S3) S23F(S3) S24F(S3) S25F(S3) PE40(S3) S1F0(S3) > PE50(S3) S1F0(S3) [...] > acpitimer0 at acpi0: 3579545 Hz, 24 bits > acpimadt0 at acpi0 addr 0xfee0: PC-AT compat > cpu0 at mainbus0: apid 0 (boot processor) > cpu0: Intel(R) Xeon(R) CPU D-1528 @ 1.90GHz, 586.43 MHz, 06-56-03 > cpu0: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,RDSEED,ADX,SMAP,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,ARAT,XSAVEOPT,MELTDOWN > cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB > 64b/line 8-way L2 cache, 9MB 64b/line 12-way L3 cache > cpu0: smt 0, core 0, package 0 > mtrr: Pentium Pro MTRR support, 8 var ranges, 88
ACPI 6.4 Could not convert 1 to 4 panic
For a new project, we are trying a Lenovo Thinkstation M70s Gen 3 running OpenBSD. We hit a hurdle straight up when validating the hardware, the kernel doesn't get past the ACPI initialisation. Specifically: acpi0: at bios0: ACPI 6.4Could not convert 1 to 4 panic: aml_die aml_convert:2095 The operating system has halted. Please press any key to reboot. Whole kernel banner can be seen here: https://pbs.twimg.com/media/Ff8ilONakAMI6gQ?format=jpg Running 7.2-current (20221024) Sorry, there isn't much extra information I can give. The usual secure boot is off. FWIW, the machine was shipped from Lenovo with Ubuntu pre-installed. Thanks, Jason.
7.2: tsc timecounter running too fast on ESXi 7.5
I'm one of several people experiencing this issue with OpenBSD 7.2 on VMware ESXi 7.5. Scott C. has given me help in trying to track the issue down; a patched -current kernel to remove the acpi_delay code added in 7.2 makes the issue go away. Below is output from sysctl machdep, sysctl hw, and dmesg: sysctl machdep: machdep.console_device=ttyC0 machdep.bios.diskinfo.128=bootdev = 0xa204, cylinders = 1024, heads = 255, sectors = 63 machdep.bios.diskinfo.129=bootdev = 0xa0010204, cylinders = 1024, heads = 255, sectors = 63 machdep.bios.diskinfo.130=bootdev = 0xa0020204, cylinders = 1024, heads = 255, sectors = 63 machdep.bios.cksumlen=2 machdep.allowaperture=0 machdep.cpuvendor=GenuineIntel machdep.cpuid=0x50663 machdep.cpufeature=0xf9bfbff machdep.kbdreset=0 machdep.xcrypt=0 machdep.lidaction=1 machdep.forceukbd=0 machdep.tscfreq=1900013052 machdep.invarianttsc=1 machdep.pwraction=1 sysctl hw: hw.machine=amd64 hw.model=Intel(R) Xeon(R) CPU D-1528 @ 1.90GHz hw.ncpu=2 hw.byteorder=1234 hw.pagesize=4096 hw.disknames=cd0:,sd0:e0a47e78ea955d63,sd1:0c3277666ef919fc,sd2:bdec30edfe97d02b hw.diskcount=4 hw.sensors.acpiac0.indicator0=On (power supply) hw.sensors.vmt0.timedelta0=0.000371 secs, OK, Tue Oct 25 14:18:36.273 hw.cpuspeed=1899 hw.vendor=VMware, Inc. hw.product=VMware Virtual Platform hw.version=None hw.serialno=VMware-56 4d 2b c8 07 85 8b b7-28 1b a0 5d d5 cf 8d fd hw.uuid=564d2bc8-0785-8bb7-281b-a05dd5cf8dfd hw.physmem=6424494080 hw.usermem=6424477696 hw.ncpufound=2 hw.allowpowerdown=1 hw.smt=0 hw.ncpuonline=2 hw.power=1 dmesg (which includes 7.1, post-upgrade, and with patched -current kernel): OpenBSD 7.1 (GENERIC.MP) #3: Sun May 15 10:27:01 MDT 2022 r...@syspatch-71-amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 6424494080 (6126MB) avail mem = 6212485120 (5924MB) random: good seed from bootblocks mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xe0010 (242 entries) bios0: vendor Phoenix Technologies LTD version "6.00" date 11/12/2020 bios0: VMware, Inc. VMware Virtual Platform acpi0 at bios0: ACPI 4.0 acpi0: sleep states S0 S1 S4 S5 acpi0: tables DSDT FACP BOOT APIC MCFG SRAT HPET WAET acpi0: wakeup devices PCI0(S3) USB_(S1) P2P0(S3) S1F0(S3) S2F0(S3) S8F0(S3) S16F(S3) S18F(S3) S22F(S3) S23F(S3) S24F(S3) S25F(S3) PE40(S3) S1F0(S3) PE50(S3) S1F0(S3) [...] acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Xeon(R) CPU D-1528 @ 1.90GHz, 1899.75 MHz, 06-56-03 cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,RDSEED,ADX,SMAP,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,ARAT,XSAVEOPT,MELTDOWN cpu0: 256KB 64b/line 8-way L2 cache cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges cpu0: apic clock running at 65MHz cpu1 at mainbus0: apid 2 (application processor) cpu1: Intel(R) Xeon(R) CPU D-1528 @ 1.90GHz, 1899.62 MHz, 06-56-03 cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,RDSEED,ADX,SMAP,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,ARAT,XSAVEOPT,MELTDOWN cpu1: 256KB 64b/line 8-way L2 cache cpu1: disabling user TSC (skew=-2507) cpu1: smt 0, core 0, package 2 ioapic0 at mainbus0: apid 1 pa 0xfec0, version 20, 24 pins acpimcfg0 at acpi0 acpimcfg0: addr 0xf000, bus 0-127 acpihpet0 at acpi0: 14318179 Hz acpiprt0 at acpi0: bus 0 (PCI0) acpipci0 at acpi0 PCI0: 0x 0x0011 0x0001 acpicmos0 at acpi0 "PNP0A05" at acpi0 not configured acpiac0 at acpi0: AC unit online acpicpu0 at acpi0: C1(@1 halt!) acpicpu1 at acpi0: C1(@1 halt!) cpu0: using VERW MDS workaround pvbus0 at mainbus0: VMware vmt0 at pvbus0 pci0 at mainbus0 bus 0 pchb0 at pci0 dev 0 function 0 "Intel 82443BX AGP" rev 0x01 ppb0 at pci0 dev 1 function 0 "Intel 82443BX AGP" rev 0x01 pci1 at ppb0 bus 1 pcib0 at pci0 dev 7 function 0 "Intel 82371AB PIIX4 ISA" rev 0x08 pciide0 at pci0 dev 7 function 1 "Intel 82371AB IDE" rev 0x01: DMA, channel 0 configured to compatibility, channel 1 configured to compatibility pciide0: channel 0 disabled (no drives) pciide0: channel 1 disabled (no drives) piixpm0 at pci0 dev 7 function 3 "Intel 82371AB Power" rev 0x08: SMBus disabled "VMware VMCI" rev 0x10 at pci0 dev 7 function 7 not configured vga1 at pci0 dev 15 function 0 "VMware SVGA II" rev 0x00 wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) wsdisplay0: screen 1-5 added (80x25, vt100 emulation) ppb1 at
Re: Had to set 'kern.timecounter.hardware' to 'acpitimer0' to fix system clock going too fast
On 24/10/2022 15:26, Scott Cheloha wrote: My guess is that we are miscalibrating the TSC frequency at least once. The only major thing that changed in the frequency calibration code between 7.1 and 7.2 is the introduction of additional delay(9) implementations. Let's see if removing them from the picture changes the result we get. Can you compile and boot a -current kernel with the attached patch and send the resulting dmesg? Patch has fixed the issue! Attaching resulting dmesg and sysctl outputs from recently compiled kernel. Cheers, - Kalabic S.OpenBSD 7.2-current (GENERIC.MP) #1: Tue Oct 25 15:57:21 UTC 2022 r...@pc118.lan4.home:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 1055850496 (1006MB) avail mem = 1006477312 (959MB) random: good seed from bootblocks mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xe0010 (556 entries) bios0: vendor Phoenix Technologies LTD version "6.00" date 09/21/2015 bios0: VMware, Inc. VMware Virtual Platform acpi0 at bios0: ACPI 4.0 acpi0: sleep states S0 S1 S4 S5 acpi0: tables DSDT FACP BOOT APIC MCFG SRAT HPET WAET acpi0: wakeup devices PCI0(S3) USB_(S1) P2P0(S3) S1F0(S3) S2F0(S3) S8F0(S3) S16F(S3) S17F(S3) S18F(S3) S22F(S3) S23F(S3) S24F(S3) S25F(S3) PE40(S3) S1F0(S3) PE50(S3) [...] acpitimer0 at acpi0: 3579545 Hz, 24 bits acpitimerattach: 6087399 21645243246 acpitimerattach: 6087421 21645247716 acpitimerattach: 6087442 21645252182 acpitimerattach: 6087463 21645256655 acpitimerattach: 6087485 21645261118 acpitimerattach: 6087506 21645265584 acpitimerattach: 6087528 21645270054 acpitimerattach: 6087549 21645274520 acpitimerattach: 6087571 21645278980 acpitimerattach: 6087592 21645283456 acpitimerattach: 6087614 21645287916 acpitimerattach: 6087635 21645292392 acpitimerattach: 6087657 21645296855 acpitimerattach: 6087678 21645301321 acpitimerattach: 6087700 21645305791 acpitimerattach: 6087721 21645310257 acpitimerattach: 6087743 21645314730 acpitimerattach: 6087764 21645319196 acpitimerattach: 6087786 21645323669 acpitimerattach: 6087807 21645328129 acpitimerattach: 6087829 21645332595 acpitimerattach: 6087850 21645337052 acpitimerattach: 6087871 21645341528 acpitimerattach: 6087893 21645345994 acpitimerattach: 6087914 21645350467 acpitimerattach: 6087936 21645354930 acpitimerattach: 6087957 21645359403 acpitimerattach: 6087979 21645363869 acpitimerattach: 6088000 21645368332 acpitimerattach: 6088022 21645372808 acpitimerattach: 6088043 21645377268 acpitimerattach: 6088065 21645381744 acpitimerattach: 6088086 21645386204 acpitimerattach: 6088108 21645390670 acpitimerattach: 6088129 21645395140 acpitimerattach: 6088151 21645399609 acpitimerattach: 6088172 21645404079 acpitimerattach: 6088194 21645408545 acpitimerattach: 6088215 21645413008 acpitimerattach: 6088237 21645417481 acpitimerattach: 6088258 21645421947 acpitimerattach: 6088280 21645426420 acpitimerattach: 6088301 21645430886 acpitimerattach: 6088322 21645435346 acpitimerattach: 6088344 21645439819 acpitimerattach: 6088365 21645444276 acpitimerattach: 6088387 21645448745 acpitimerattach: 6088408 21645875421 acpitimerattach: 6088430 21645879881 acpitimerattach: 6088451 21645884341 acpitimerattach: 6088473 2164504 acpitimerattach: 6088494 21645893280 acpitimerattach: 6088516 21645897743 acpitimerattach: 6088537 21645902200 acpitimerattach: 6088559 21645906670 acpitimerattach: 6088580 21645911130 acpitimerattach: 6088602 21645915593 acpitimerattach: 6088623 21645920050 acpitimerattach: 6088645 21645924523 acpitimerattach: 6088666 21645928980 acpitimerattach: 6088688 21645933440 acpitimerattach: 6088709 21645937903 acpitimerattach: 6088731 21645942373 acpitimerattach: 6088752 21645946836 acpitimerattach: 6088773 21645951293 acpitimerattach: 6088795 21645955753 acpitimerattach: 6088816 21645960217 acpitimerattach: 6088838 21645964680 acpitimerattach: 6088859 21645969140 acpitimerattach: 601 21645973613 acpitimerattach: 6088902 21645978895 acpitimerattach: 6088924 21645983371 acpitimerattach: 6088945 21645987837 acpitimerattach: 6088967 21645992303 acpitimerattach: 6088988 21645996770 acpitimerattach: 6089010 21646001236 acpitimerattach: 6089031 21646005696 acpitimerattach: 6089053 21646010169 acpitimerattach: 6089074 21646014635 acpitimerattach: 6089096 21646019111 acpitimerattach: 6089117 21646023574 acpitimerattach: 6089139 21646028037 acpitimerattach: 6089160 21646032510 acpitimerattach: 6089181 21646036976 acpitimerattach: 6089203 21646041449 acpitimerattach: 6089224 21646045912 acpitimerattach: 6089246 21646050385 acpitimerattach: 6089267 21646054845 acpitimerattach: 6089289 21646059311 acpitimerattach: 6089310 21646063781 acpitimerattach: 6089332 21646068247 acpitimerattach: 6089353 21646072713 acpitimerattach: 6089375 21646077186 acpitimerattach: 6089396 21646081649 acpitimerattach: 6089418 21646086119 acpitimerattach: 6089439 21646090588 acpitimerattach: 6089461 21646095061 acpitimerattach: 6089482 21646099527
Re: packet forwarding issues with TSO
On Tue, Oct 25, 2022 at 12:43:48PM +0100, Pedro Caetano wrote: > Hi @bugs, > > I've a pair of boxes peering bgp with isp infrastructure and on premises > infrastructure > One of the routers was upgraded to 7.2 last night without any issues. > > This is a 'border router' announcing private prefixes via bgp to isp > equipment and customer on a vpn that is pushing ~5tb a day. (upload and > download aggregated) > It has a few rules on pf.conf setup with sloppy state. (only one router was > upgraded, the traffic flow is assymetric) > > > After upgrading, TSO was enabled on this NICs using ifconfig ix0 tso; > ifconfig ix1 tso. > It was not possible to detect any issues on the maintenance window, yet > today with more users using the network we had reports of issues. > > Some traffic would be very *slow*, some traffic would flow normally. > After disabling TSO, the packet forwarding resumed its normal state. > Unfortunately I do not have pcaps to support this issue. > > The router is running 7.2 with latest patches available > > Please let me know what further information would be useful in this > scenario. Do not enable TSO on forwading infrastructure. It is only useful on servers and clients. And even then receive TCP segment offloading does not scale on systems handling many concurrent TCP sessions. The cards just don't have enough RAM to reassemble that many streams. So I'm not surprised that some connections show strange behaviour. -- :wq Claudio
packet forwarding issues with TSO
Hi @bugs, I've a pair of boxes peering bgp with isp infrastructure and on premises infrastructure One of the routers was upgraded to 7.2 last night without any issues. This is a 'border router' announcing private prefixes via bgp to isp equipment and customer on a vpn that is pushing ~5tb a day. (upload and download aggregated) It has a few rules on pf.conf setup with sloppy state. (only one router was upgraded, the traffic flow is assymetric) After upgrading, TSO was enabled on this NICs using ifconfig ix0 tso; ifconfig ix1 tso. It was not possible to detect any issues on the maintenance window, yet today with more users using the network we had reports of issues. Some traffic would be very *slow*, some traffic would flow normally. After disabling TSO, the packet forwarding resumed its normal state. Unfortunately I do not have pcaps to support this issue. The router is running 7.2 with latest patches available Please let me know what further information would be useful in this scenario. Thank your your support, Pedro Dmesg below: OpenBSD 7.2 (GENERIC.MP) #758: Tue Sep 27 11:57:54 MDT 2022 dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 8437354496 (8046MB) avail mem = 8164241408 (7786MB) random: good seed from bootblocks mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.8 @ 0x788cb000 (262 entries) bios0: vendor HP version "P89" date 06/02/2016 bios0: HP ProLiant DL380 Gen9 acpi0 at bios0: ACPI 5.0 acpi0: sleep states S0 S5 acpi0: tables DSDT FACP UEFI MCEJ SSDT HEST BERT ERST EINJ BGRT HPET PMCT WDDT APIC MCFG SLIT SRAT SPMI RASF SPCR MSCT BDAT PCCT DMAR SSDT SSDT SSDT acpi0: wakeup devices PEX4(S4) BR05(S4) BR03(S4) BR07(S4) BR09(S4) BR11(S4) BR13(S4) BR17(S4) acpitimer0 at acpi0: 3579545 Hz, 24 bits acpihpet0 at acpi0: 14318179 Hz acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Xeon(R) CPU E5-2603 v3 @ 1.60GHz, 1598.17 MHz, 06-3f-02 cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,PQM,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB 64b/line 8-way L2 cache, 15MB 64b/line 20-way L3 cache cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges cpu0: apic clock running at 99MHz cpu0: mwait min=64, max=64, C-substates=0.2.1.2, IBE cpu1 at mainbus0: apid 4 (application processor) cpu1: Intel(R) Xeon(R) CPU E5-2603 v3 @ 1.60GHz, 1598.17 MHz, 06-3f-02 cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,PQM,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN cpu1: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB 64b/line 8-way L2 cache, 15MB 64b/line 20-way L3 cache cpu1: smt 0, core 2, package 0 cpu2 at mainbus0: apid 8 (application processor) cpu2: Intel(R) Xeon(R) CPU E5-2603 v3 @ 1.60GHz, 1598.17 MHz, 06-3f-02 cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,PQM,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN cpu2: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB 64b/line 8-way L2 cache, 15MB 64b/line 20-way L3 cache cpu2: smt 0, core 4, package 0 cpu3 at mainbus0: apid 16 (application processor) cpu3: Intel(R) Xeon(R) CPU E5-2603 v3 @ 1.60GHz, 1598.18 MHz, 06-3f-02 cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,PQM,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN cpu3: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB 64b/line 8-way L2 cache, 15MB 64b/line 20-way L3 cache cpu3: smt 0, core 0, package 1 cpu4 at mainbus0: apid 20 (application processor)
