Re: [v5] amd64: simplify TSC sync testing
Hi, Scott. Sorry, I missed v5 patch. I tested v5 patch on my Ryzen7 box. And I got failed message: $ sysctl -a | grep tsc kern.timecounter.choice=i8254(0) acpihpet0(1000) tsc(2000) acpitimer0(1000) machdep.tscfreq=3593259787 machdep.invarianttsc=1 $ sysctl kern.timecounter kern.timecounter.tick=1 kern.timecounter.timestepwarnings=0 kern.timecounter.hardware=acpihpet0 kern.timecounter.choice=i8254(0) acpihpet0(1000) tsc(2000) acpitimer0(1000) $ dmesg | grep failed tsc: cpu0/cpu1: sync test round 1/2 failed tsc: cpu0/cpu2: sync test round 1/2 failed tsc: cpu0/cpu3: sync test round 2/2 failed tsc: cpu0/cpu4: sync test round 1/2 failed tsc: cpu0/cpu5: sync test round 1/2 failed tsc: cpu0/cpu6: sync test round 1/2 failed tsc: cpu0/cpu7: sync test round 1/2 failed $ dmesg: OpenBSD 7.2-beta (GENERIC.MP) #11: Mon Aug 1 14:26:45 JST 2022 a...@g2-obsd.my.domain:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 34256752640 (32669MB) avail mem = 33201164288 (31663MB) random: good seed from bootblocks mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xdb64 (63 entries) bios0: vendor American Megatrends Inc. version "9015" date 03/03/2020 bios0: MouseComputer Co.,Ltd. LM-AG400 acpi0 at bios0: ACPI 6.0 acpi0: sleep states S0 S3 S4 S5 acpi0: tables DSDT FACP APIC FPDT FIDT SSDT MSDM SSDT SSDT MCFG HPET UEFI BGRT TPM2 IVRS PCCT SSDT CRAT CDIT SSDT SSDT SSDT acpi0: wakeup devices GPP0(S4) GPP1(S4) GPP3(S4) GPP4(S4) GPP5(S4) GPP6(S4) GPP7(S4) GPP8(S4) X161(S4) GPP9(S4) X162(S4) GPPA(S4) GPPB(S4) GPPC(S4) GPPD(S4) GPPE(S4) [...] acpitimer0 at acpi0: 3579545 Hz, 32 bits acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: AMD Ryzen 7 3700X 8-Core Processor, 3593.75 MHz, 17-71-00 cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,PQM,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,UMIP,IBPB,STIBP,SSBD,XSAVEOPT,XSAVEC,XGETBV1,XSAVES cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 512KB 64b/line 8-way L2 cache, 16MB 64b/line 16-way L3 cache cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges cpu0: apic clock running at 99MHz cpu0: mwait min=64, max=64, C-substates=1.1, IBE cpu1 at mainbus0: apid 2 (application processor) tsc: cpu0/cpu1: sync test round 1/2 failed tsc: cpu0/cpu1: cpu1: 1 lags 36 cycles cpu1: AMD Ryzen 7 3700X 8-Core Processor, 3593.25 MHz, 17-71-00 cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,PQM,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,UMIP,IBPB,STIBP,SSBD,XSAVEOPT,XSAVEC,XGETBV1,XSAVES cpu1: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 512KB 64b/line 8-way L2 cache, 16MB 64b/line 16-way L3 cache cpu1: smt 0, core 1, package 0 cpu2 at mainbus0: apid 4 (application processor) tsc: cpu0/cpu2: sync test round 1/2 failed tsc: cpu0/cpu2: cpu2: 82 lags 36 cycles cpu2: AMD Ryzen 7 3700X 8-Core Processor, 3593.25 MHz, 17-71-00 cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,PQM,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,UMIP,IBPB,STIBP,SSBD,XSAVEOPT,XSAVEC,XGETBV1,XSAVES cpu2: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 512KB 64b/line 8-way L2 cache, 16MB 64b/line 16-way L3 cache cpu2: smt 0, core 2, package 0 cpu3 at mainbus0: apid 6 (application processor) tsc: cpu0/cpu3: sync test round 2/2 failed tsc: cpu0/cpu3: cpu3: 13 lags 36 cycles cpu3: AMD Ryzen 7 3700X 8-Core Processor, 3593.25 MHz, 17-71-00 cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,PQM,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,UMIP,IBPB,STIBP,SSBD,XSAVEOPT,XSAVEC,XGETBV1,XSAVES cpu3: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 512KB 64b/line 8-way L2 cache, 16MB 64b/line 16-way L3 cache cpu3: smt 0, core
Re: bioctl.8: noauto flag has no effect in bootloaders
Klemens Nanni [k...@openbsd.org] wrote: > Reading "at boot time" may come off as "in the bootloader" > -Do not automatically assemble this volume at boot time. > +Do not automatically assemble this volume at system startup time. I don't think this reads any different. If you want to be very clear, talk about the kernel. I'd use "kernel start" instead of "boot time" or "system startup time" (which are both vague to me.)
Re: [v4] amd64: simplify TSC sync testing
Hi, Scott I tested your patch on my on ESXi on Ryzen7 box. It works fine for me. $ sysctl -a | grep tsc kern.timecounter.hardware=tsc kern.timecounter.choice=i8254(0) acpihpet0(1000) tsc(2000) acpitimer0(1000) machdep.tscfreq=3593261949 machdep.invarianttsc=1 $ sysctl kern.timecounter kern.timecounter.tick=1 kern.timecounter.timestepwarnings=0 kern.timecounter.hardware=tsc kern.timecounter.choice=i8254(0) acpihpet0(1000) tsc(2000) acpitimer0(1000) $ dmesg | grep failed $ dmesg: OpenBSD 7.2-beta (GENERIC.MP) #1: Mon Aug 1 14:01:28 JST 2022 a...@vmamd-obsd-curr.my.domain:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 17161912320 (16366MB) avail mem = 16624422912 (15854MB) random: good seed from bootblocks mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xe0010 (260 entries) bios0: vendor Phoenix Technologies LTD version "6.00" date 12/12/2018 bios0: VMware, Inc. VMware Virtual Platform acpi0 at bios0: ACPI 4.0 acpi0: sleep states S0 S1 S4 S5 acpi0: tables DSDT FACP BOOT APIC MCFG SRAT HPET WAET acpi0: wakeup devices PCI0(S3) USB_(S1) P2P0(S3) S1F0(S3) S2F0(S3) S8F0(S3) S16F(S3) S17F(S3) S18F(S3) S22F(S3) S23F(S3) S24F(S3) S25F(S3) PE40(S3) S1F0(S3) PE50(S3) [...] acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: AMD Ryzen 7 3700X 8-Core Processor, 3592.99 MHz, 17-71-00 cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,PCLMUL,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,IBPB,XSAVEOPT,XSAVEC,XSAVES cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache cpu0: 512KB 64b/line 8-way L2 cache cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges cpu0: apic clock running at 65MHz cpu1 at mainbus0: apid 2 (application processor) cpu1: AMD Ryzen 7 3700X 8-Core Processor, 3592.74 MHz, 17-71-00 cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,PCLMUL,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,IBPB,XSAVEOPT,XSAVEC,XSAVES cpu1: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache cpu1: 512KB 64b/line 8-way L2 cache cpu1: smt 0, core 0, package 2 cpu2 at mainbus0: apid 4 (application processor) cpu2: AMD Ryzen 7 3700X 8-Core Processor, 3592.75 MHz, 17-71-00 cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,PCLMUL,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,IBPB,XSAVEOPT,XSAVEC,XSAVES cpu2: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache cpu2: 512KB 64b/line 8-way L2 cache cpu2: smt 0, core 0, package 4 cpu3 at mainbus0: apid 6 (application processor) cpu3: AMD Ryzen 7 3700X 8-Core Processor, 3592.76 MHz, 17-71-00 cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,PCLMUL,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,IBPB,XSAVEOPT,XSAVEC,XSAVES cpu3: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache cpu3: 512KB 64b/line 8-way L2 cache cpu3: smt 0, core 0, package 6 cpu4 at mainbus0: apid 8 (application processor) cpu4: AMD Ryzen 7 3700X 8-Core Processor, 3592.74 MHz, 17-71-00 cpu4: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,PCLMUL,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,IBPB,XSAVEOPT,XSAVEC,XSAVES cpu4: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache cpu4: 512KB 64b/line 8-way L2 cache cpu4: smt 0, core 0, package 8 cpu5 at mainbus0: apid 10 (application processor) cpu5: AMD Ryzen 7 3700X 8-Core Processor, 3592.73 MHz, 17-71-00 cpu5:
Re: [v4] amd64: simplify TSC sync testing
> On Jul 31, 2022, at 23:48, Masato Asou wrote: > > Hi, Scott > > I tested your patch on my Ryzen7 box. > And I got failed message: > > $ sysctl -a | grep tsc > kern.timecounter.choice=i8254(0) acpihpet0(1000) tsc(2000) > acpitimer0(1000) > machdep.tscfreq=3593244667 > machdep.invarianttsc=1 > $ sysctl kern.timecounter > kern.timecounter.tick=1 > kern.timecounter.timestepwarnings=0 > kern.timecounter.hardware=acpihpet0 > kern.timecounter.choice=i8254(0) acpihpet0(1000) tsc(2000) > acpitimer0(1000) > $ dmesg | grep failed > tsc: cpu0/cpu2: sync test round 1/2 failed > tsc: cpu0/cpu4: sync test round 1/2 failed > tsc: cpu0/cpu5: sync test round 1/2 failed > tsc: cpu0/cpu6: sync test round 1/2 failed > tsc: cpu0/cpu7: sync test round 1/2 failed > $ Thank you for testing. Please try with the latest patch. v5 is posted on tech@ now. > dmesg: > > OpenBSD 7.2-beta (GENERIC.MP) #10: Mon Aug 1 13:12:06 JST 2022 >a...@g2-obsd.my.domain:/usr/src/sys/arch/amd64/compile/GENERIC.MP > real mem = 34256752640 (32669MB) > avail mem = 33201152000 (31663MB) > random: good seed from bootblocks > mpath0 at root > scsibus0 at mpath0: 256 targets > mainbus0 at root > bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xdb64 (63 entries) > bios0: vendor American Megatrends Inc. version "9015" date 03/03/2020 > bios0: MouseComputer Co.,Ltd. LM-AG400 You may also want to try updating your BIOS.
Re: [v4] amd64: simplify TSC sync testing
Hi, Scott I tested your patch on my Ryzen7 box. And I got failed message: $ sysctl -a | grep tsc kern.timecounter.choice=i8254(0) acpihpet0(1000) tsc(2000) acpitimer0(1000) machdep.tscfreq=3593244667 machdep.invarianttsc=1 $ sysctl kern.timecounter kern.timecounter.tick=1 kern.timecounter.timestepwarnings=0 kern.timecounter.hardware=acpihpet0 kern.timecounter.choice=i8254(0) acpihpet0(1000) tsc(2000) acpitimer0(1000) $ dmesg | grep failed tsc: cpu0/cpu2: sync test round 1/2 failed tsc: cpu0/cpu4: sync test round 1/2 failed tsc: cpu0/cpu5: sync test round 1/2 failed tsc: cpu0/cpu6: sync test round 1/2 failed tsc: cpu0/cpu7: sync test round 1/2 failed $ dmesg: OpenBSD 7.2-beta (GENERIC.MP) #10: Mon Aug 1 13:12:06 JST 2022 a...@g2-obsd.my.domain:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 34256752640 (32669MB) avail mem = 33201152000 (31663MB) random: good seed from bootblocks mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xdb64 (63 entries) bios0: vendor American Megatrends Inc. version "9015" date 03/03/2020 bios0: MouseComputer Co.,Ltd. LM-AG400 acpi0 at bios0: ACPI 6.0 acpi0: sleep states S0 S3 S4 S5 acpi0: tables DSDT FACP APIC FPDT FIDT SSDT MSDM SSDT SSDT MCFG HPET UEFI BGRT TPM2 IVRS PCCT SSDT CRAT CDIT SSDT SSDT SSDT acpi0: wakeup devices GPP0(S4) GPP1(S4) GPP3(S4) GPP4(S4) GPP5(S4) GPP6(S4) GPP7(S4) GPP8(S4) X161(S4) GPP9(S4) X162(S4) GPPA(S4) GPPB(S4) GPPC(S4) GPPD(S4) GPPE(S4) [...] acpitimer0 at acpi0: 3579545 Hz, 32 bits acpimadt0 at acpi0 addr 0xfee0: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: AMD Ryzen 7 3700X 8-Core Processor, 3593.68 MHz, 17-71-00 cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,PQM,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,UMIP,IBPB,STIBP,SSBD,XSAVEOPT,XSAVEC,XGETBV1,XSAVES cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 512KB 64b/line 8-way L2 cache, 16MB 64b/line 16-way L3 cache cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges cpu0: apic clock running at 99MHz cpu0: mwait min=64, max=64, C-substates=1.1, IBE cpu1 at mainbus0: apid 2 (application processor) cpu1: AMD Ryzen 7 3700X 8-Core Processor, 3593.24 MHz, 17-71-00 cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,PQM,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,UMIP,IBPB,STIBP,SSBD,XSAVEOPT,XSAVEC,XGETBV1,XSAVES cpu1: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 512KB 64b/line 8-way L2 cache, 16MB 64b/line 16-way L3 cache cpu1: smt 0, core 1, package 0 cpu2 at mainbus0: apid 4 (application processor) tsc: cpu0/cpu2: sync test round 1/2 failed tsc: cpu0/cpu2: cpu2: 161 lags 72 cycles cpu2: AMD Ryzen 7 3700X 8-Core Processor, 3593.24 MHz, 17-71-00 cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,PQM,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,UMIP,IBPB,STIBP,SSBD,XSAVEOPT,XSAVEC,XGETBV1,XSAVES cpu2: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 512KB 64b/line 8-way L2 cache, 16MB 64b/line 16-way L3 cache cpu2: smt 0, core 2, package 0 cpu3 at mainbus0: apid 6 (application processor) cpu3: AMD Ryzen 7 3700X 8-Core Processor, 3593.24 MHz, 17-71-00 cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,PQM,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,UMIP,IBPB,STIBP,SSBD,XSAVEOPT,XSAVEC,XGETBV1,XSAVES cpu3: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 512KB 64b/line 8-way L2 cache, 16MB 64b/line 16-way L3 cache cpu3: smt 0, core 3, package 0 cpu4 at mainbus0: apid 8 (application processor) tsc: cpu0/cpu4: sync test round 1/2 failed tsc: cpu0/cpu4: cpu4: 129 lags 1296 cycles cpu4: AMD Ryzen 7 3700X 8-Core Processor, 3593.25 MHz, 17-71-00 cpu4:
riscv64: trigger deferred timer interrupts from splx(9)
Hi,
I am unsure how to properly mask RISC-V timer interrupts in hardware
at or above IPL_CLOCK. I think that would be cleaner than doing
another timer interrupt deferral thing.
But, just to get the ball rolling, here a first attempt at the timer
interrupt deferral thing for riscv64. The motivation, as with every
other platform, is to eventually make it unnecessary for the machine
dependent code to know anything about the clock interrupt schedule.
The thing I'm most unsure about is where to retrigger the timer in the
PLIC code. It seems right to do it from plic_setipl() because I want
to retrigger it before doing soft interrupt work, but I'm not sure.
Unless I'm missing something, I don't think I need to do anything in
the default interrupt handler code, i.e. riscv64_dflt_setipl(), right?
I have no riscv64 machine, so this is untested. Would appreciate
tests and feedback.
-Scott
Index: include/cpu.h
===
RCS file: /cvs/src/sys/arch/riscv64/include/cpu.h,v
retrieving revision 1.12
diff -u -p -r1.12 cpu.h
--- include/cpu.h 10 Jun 2022 21:34:15 - 1.12
+++ include/cpu.h 1 Aug 2022 01:13:38 -
@@ -92,7 +92,7 @@ struct cpu_info {
uint64_tci_lasttb;
uint64_tci_nexttimerevent;
uint64_tci_nextstatevent;
- int ci_statspending;
+ volatile intci_timer_deferred;
uint32_tci_cpl;
uint32_tci_ipending;
Index: riscv64/clock.c
===
RCS file: /cvs/src/sys/arch/riscv64/riscv64/clock.c,v
retrieving revision 1.3
diff -u -p -r1.3 clock.c
--- riscv64/clock.c 24 Jul 2021 22:41:09 - 1.3
+++ riscv64/clock.c 1 Aug 2022 01:13:38 -
@@ -21,6 +21,7 @@
#include
#include
#include
+#include
#include
#include
@@ -106,6 +107,17 @@ clock_intr(void *frame)
int s;
/*
+* If the clock interrupt is masked, defer all clock interrupt
+* work until the clock interrupt is unmasked from splx(9).
+*/
+ if (ci->ci_cpl >= IPL_CLOCK) {
+ ci->ci_timer_deferred = 1;
+ sbi_set_timer(UINT64_MAX);
+ return 0;
+ }
+ ci->ci_timer_deferred = 0;
+
+ /*
* Based on the actual time delay since the last clock interrupt,
* we arrange for earlier interrupt next time.
*/
@@ -132,30 +144,23 @@ clock_intr(void *frame)
sbi_set_timer(nextevent);
- if (ci->ci_cpl >= IPL_CLOCK) {
- ci->ci_statspending += nstats;
- } else {
- nstats += ci->ci_statspending;
- ci->ci_statspending = 0;
-
- s = splclock();
- intr_enable();
-
- /*
-* Do standard timer interrupt stuff.
-*/
- while (ci->ci_lasttb < prevtb) {
- ci->ci_lasttb += tick_increment;
- clock_count.ec_count++;
- hardclock((struct clockframe *)frame);
- }
+ s = splclock();
+ intr_enable();
- while (nstats-- > 0)
- statclock((struct clockframe *)frame);
-
- intr_disable();
- splx(s);
+ /*
+* Do standard timer interrupt stuff.
+*/
+ while (ci->ci_lasttb < prevtb) {
+ ci->ci_lasttb += tick_increment;
+ clock_count.ec_count++;
+ hardclock((struct clockframe *)frame);
}
+
+ while (nstats-- > 0)
+ statclock((struct clockframe *)frame);
+
+ intr_disable();
+ splx(s);
return 0;
}
Index: dev/plic.c
===
RCS file: /cvs/src/sys/arch/riscv64/dev/plic.c,v
retrieving revision 1.10
diff -u -p -r1.10 plic.c
--- dev/plic.c 6 Apr 2022 18:59:27 - 1.10
+++ dev/plic.c 1 Aug 2022 01:13:38 -
@@ -557,6 +557,10 @@ plic_setipl(int new)
/* higher values are higher priority */
plic_set_threshold(ci->ci_cpuid, new);
+ /* trigger deferred timer interrupt if cpl is now low enough */
+ if (ci->ci_timer_deferred && new < IPL_CLOCK)
+ sbi_set_timer(0);
+
intr_restore(sie);
}
mips64: trigger deferred timer interrupt from splx(9)
Hi,
Apparently mips64, i.e. octeon and loongson, has the same problem as
powerpc/macppc and powerpc64. The timer interrupt is normally only
logically masked, not physically masked in the hardware, when we're
running at or above IPL_CLOCK. If we arrive at cp0_int5() when the
clock interrupt is logically masked we postpone all work until the
next tick. This is a problem for my WIP clock interrupt work.
So, this patch is basically the same as what I did for macppc and what
I have proposed for powerpc64.
- Add a new member, ci_timer_deferred, to mips64's cpu_info struct.
While here, remove ci_pendingticks. We don't need it anymore.
- If we get to cp0_int5() and our IPL is too high, set
cpu_info.ci_timer_deferred and return.
- If we get to cp0_int5() and our IPL is low enough, clear
cpu_info.ci_timer_deferred and do clock interrupt work.
- In splx(9), if the new IPL is low enough and cpu_info.ci_timer_deferred
is set, trigger the clock interrupt.
The only big difference is that mips64 uses an equality comparison
when deciding whether to arm the timer interrupt, so it's really easy
to "miss" CP0.count when you're setting CP0.compare.
To address this I've written a function, cp0_raise_int5(), that spins
until it is sure the timer interrupt will go off. The function needed
a bit of new code for reading CP0.cause, which I've added to
cp0access.S. I am using an initial offset of 16 cycles based on
experimentation with the machine I have access to, a 500Mhz CN50xx.
Values lower than 16 require more than one loop to arm the timer. If
that value is insufficient for other machines we can try passing the
initial offset as an argument to the function.
I wasn't sure where to put the prototype for cp0_raise_int5() so I
stuck it in mips64/cpu.h. If there's a better place for it, just say
so.
I also left some atomic counters for you to poke at with pstat(8) if
you want to see what the machine is doing in cp0_raise_int5(), i.e.
how often we defer clock interrupt work and how many loops you take to
arm the timer interrupt. Those will be removed before commit.
I'm running a `make build` on my EdgeRouter PoE. It only has 512MB of
RAM, so I can't do a parallel build without hanging the machine when
attempting to compile LLVM. The build has been running for four days
and the machine has not yet hung, so I think this patch is correct-ish.
I will holler if it hangs.
visa: Assuming this code looks right, could you test this on a
beefier octeon machine? Preferably a parallel build?
miod: I'm unclear whether loongson uses cp0_int5(). Am I missing
code here, or are my changes in arch/loongson sufficient?
If it's sufficient, could you test this?
I have no loongson hardware, so this is uncompiled there.
Sorry in advance if it does not compile.
Thoughts?
Index: mips64/mips64/clock.c
===
RCS file: /cvs/src/sys/arch/mips64/mips64/clock.c,v
retrieving revision 1.45
diff -u -p -r1.45 clock.c
--- mips64/mips64/clock.c 6 Apr 2022 18:59:26 - 1.45
+++ mips64/mips64/clock.c 31 Jul 2022 18:18:05 -
@@ -92,13 +92,13 @@ clockattach(struct device *parent, struc
* Interrupt handler for targets using the internal count register
* as interval clock. Normally the system is run with the clock
* interrupt always enabled. Masking is done here and if the clock
- * can not be run the tick is just counted and handled later when
- * the clock is logically unmasked again.
+ * can not be run the is tick handled later when the clock is logically
+ * unmasked again.
*/
uint32_t
cp0_int5(uint32_t mask, struct trapframe *tf)
{
- u_int32_t clkdiff;
+ u_int32_t clkdiff, pendingticks = 0;
struct cpu_info *ci = curcpu();
/*
@@ -113,15 +113,26 @@ cp0_int5(uint32_t mask, struct trapframe
}
/*
+* If the clock interrupt is masked we can't do any work until
+* it is unmasked.
+*/
+ if (tf->ipl >= IPL_CLOCK) {
+ ci->ci_timer_deferred = 1;
+ cp0_set_compare(cp0_get_count() - 1);
+ return CR_INT_5;
+ }
+ ci->ci_timer_deferred = 0;
+
+ /*
* Count how many ticks have passed since the last clock interrupt...
*/
clkdiff = cp0_get_count() - ci->ci_cpu_counter_last;
while (clkdiff >= ci->ci_cpu_counter_interval) {
ci->ci_cpu_counter_last += ci->ci_cpu_counter_interval;
clkdiff = cp0_get_count() - ci->ci_cpu_counter_last;
- ci->ci_pendingticks++;
+ pendingticks++;
}
- ci->ci_pendingticks++;
+ pendingticks++;
ci->ci_cpu_counter_last += ci->ci_cpu_counter_interval;
/*
@@ -132,32 +143,64 @@ cp0_int5(uint32_t mask, struct trapframe
clkdiff = cp0_get_count() - ci->ci_cpu_counter_last;
if ((int)clkdiff >= 0) {
Re: [v5] amd64: simplify TSC sync testing
Scott Cheloha writes: > Hi, > > At the urging of sthen@ and dv@, here is v5. > > Two major changes from v4: > > - Add the function tc_reset_quality() to kern_tc.c and use it > to lower the quality of the TSC timecounter if we fail the > sync test. > > tc_reset_quality() will choose a new active timecounter if, > after the quality change, the given timecounter is no longer > the best timecounter. > > The upshot is: if you fail the TSC sync test you should boot > with the HPET as your active timecounter. If you don't have > an HPET you'll be using something else. > > - Drop the SMT accomodation from the hot loop. It hasn't been > necessary since last year when I rewrote the test to run without > a mutex. In the rewritten test, the two CPUs in the hot loop > are not competing for any resources so they should not be able > to starve one another. > > dv: Could you double-check that this still chooses the right > timecounter on your machine? If so, I will ask deraadt@ to > put this into snaps to replace v4. > Yes, looks like it's choosing acpihpet0 still with this diff. No issues after zzz/ZZZ either. $ sysctl | grep hpet kern.timecounter.hardware=acpihpet0 kern.timecounter.choice=i8254(0) tsc(-1000) acpihpet0(1000) acpitimer0(1000) I'm going to try getting the latest bios on this machine and see if anything changes. -dv
Re: interface media without netlock
On Sun, Jul 31, 2022 at 12:24:01PM +0100, Stuart Henderson wrote: > On 2022/07/28 13:30, Alexander Bluhm wrote: > > Problem is that smtpd(8) periodically checks media status. > > Really?! I bet it is the other simple protocol daemon, snmpd(8) -- :wq Claudio
Re: interface media without netlock
On 2022/07/28 13:30, Alexander Bluhm wrote: > Problem is that smtpd(8) periodically checks media status. Really?!
Re: Consistency and cleanup in /share/misc/airport
On Sat, 2022-07-30 at 22:44 +0100, Stuart Henderson wrote: > Due to the rule for this file mentioned in the header, I think you'll > need to find a developer who has been to the added airports that have > replaced the metro areas, i.e. KCK MFW QDV QSF, either that or just > remove them. You are right. I'll give it a try: Anyone been on one of those? KCK:Kirensk, Russia MFW:Magaruque Island, Mozambique QDV:Comte. Rolim Adolfo Amaro, Jundiai, Brazil QSF:8 May 1945, Setif, Algeria If no response I'll submit a revised patch with them removed next weekend.
Re: [v5] amd64: simplify TSC sync testing
Scott Cheloha [2022-07-30, 22:13 -0500]:
> Hi,
>
> At the urging of sthen@ and dv@, here is v5.
>
> Two major changes from v4:
>
> - Add the function tc_reset_quality() to kern_tc.c and use it
> to lower the quality of the TSC timecounter if we fail the
> sync test.
>
> tc_reset_quality() will choose a new active timecounter if,
> after the quality change, the given timecounter is no longer
> the best timecounter.
>
> The upshot is: if you fail the TSC sync test you should boot
> with the HPET as your active timecounter. If you don't have
> an HPET you'll be using something else.
>
> - Drop the SMT accomodation from the hot loop. It hasn't been
> necessary since last year when I rewrote the test to run without
> a mutex. In the rewritten test, the two CPUs in the hot loop
> are not competing for any resources so they should not be able
> to starve one another.
>
> dv: Could you double-check that this still chooses the right
> timecounter on your machine? If so, I will ask deraadt@ to
> put this into snaps to replace v4.
>
> Additional test reports are welcome. Include your dmesg.
>
> --
>
> I do not see much more I can do to improve this patch.
>
> I am seeking patch review and OKs.
>
> I am especially interested in whether my assumptions in tsc_ap_test()
> and tsc_bp_test() are correct. The whole patch depends on those
> assumptions. Is this a valid way to test for TSC desync? Or am I
> missing membar_producer()/membar_consumer() calls?
>
> Here is the long version of "what" and "why" for this patch.
>
> The patch is attached at the end.
>
> - Computing a per-CPU TSC skew value is error-prone, especially
> on multisocket machines and VMs. My best guess is that larger
> latencies appear to the skew measurement test as TSC desync,
> and so the TSC is demoted to a kernel timecounter on these
> machines or marked non-monotonic.
>
> This patch eliminates per-CPU TSC skew values. Instead of trying
> to measure and correct for TSC desync we only try to detect desync,
> which is less error-prone. This approach should allow a wider
> variety of machines to use the TSC as a timecounter when running
> OpenBSD.
>
> - In the new sync test, both CPUs repeatedly try to detect whether
> their TSC is trailing the other CPU's TSC. The upside to this
> approach is that it yields no false positives (if my assumptions
> about AMD64 memory access and instruction serialization are correct).
> The downside to this approach is that it takes more time than the
> current skew measurement test. Each test round takes 1ms, and
> we run up to two rounds per CPU, so this patch slows boot down
> by 2ms per AP.
>
> - If any CPU fails the sync test, the TSC is marked non-monotonic
> and a different timecounter is activated. The TC_USER flag
> remains intact. There is no "middle ground" where we fall back
> to only using the TSC in the kernel.
>
> - Because there is no per-CPU skew value, there is also no concept
> of TSC drift anymore.
>
> - Before running the test, we check for the IA32_TSC_ADJUST
> register and reset it if necessary. This is a trivial way
> to work around firmware bugs that desync the TSC before we
> reach the kernel.
>
> Unfortunately, at the moment this register appears to only
> be available on Intel processors and I cannot find an equivalent
> but differently-named MSR for AMD processors.
>
> --
>
> Index: sys/arch/amd64/amd64/tsc.c
> ===
> RCS file: /cvs/src/sys/arch/amd64/amd64/tsc.c,v
> retrieving revision 1.24
> diff -u -p -r1.24 tsc.c
> --- sys/arch/amd64/amd64/tsc.c31 Aug 2021 15:11:54 - 1.24
> +++ sys/arch/amd64/amd64/tsc.c31 Jul 2022 03:06:39 -
> @@ -36,13 +36,6 @@ inttsc_recalibrate;
> uint64_t tsc_frequency;
> int tsc_is_invariant;
>
> -#define TSC_DRIFT_MAX 250
> -#define TSC_SKEW_MAX 100
> -int64_t tsc_drift_observed;
> -
> -volatile int64_t tsc_sync_val;
> -volatile struct cpu_info *tsc_sync_cpu;
> -
> u_inttsc_get_timecount(struct timecounter *tc);
> void tsc_delay(int usecs);
>
> @@ -236,22 +229,12 @@ cpu_recalibrate_tsc(struct timecounter *
> u_int
> tsc_get_timecount(struct timecounter *tc)
> {
> - return rdtsc_lfence() + curcpu()->ci_tsc_skew;
> + return rdtsc_lfence();
> }
>
> void
> tsc_timecounter_init(struct cpu_info *ci, uint64_t cpufreq)
> {
> -#ifdef TSC_DEBUG
> - printf("%s: TSC skew=%lld observed drift=%lld\n", ci->ci_dev->dv_xname,
> - (long long)ci->ci_tsc_skew, (long long)tsc_drift_observed);
> -#endif
> - if (ci->ci_tsc_skew < -TSC_SKEW_MAX || ci->ci_tsc_skew > TSC_SKEW_MAX) {
> - printf("%s: disabling user TSC (skew=%lld)\n",
> - ci->ci_dev->dv_xname, (long long)ci->ci_tsc_skew);
> - tsc_timecounter.tc_user =
