Re: Zenbleed
Manawyrm wrote: > (Hetzner engineer here, but speaking as a private individual) > > Hetzner Cloud is using regular mainline QEMU on Linux as the hypervisor, > so while I'd agree that faulting when the MSR is set isn't ideal, this > behaviour will probably also occur on a lot of other machines/setups. > > Another solution might be preferrable. Linux has started to do a very > similar thing and I haven't yet checked if it runs into a similar > situation/fault. > > At least on Hetzner Cloud, there is a lot of indications that the > machine is running as a virtual machine (ACPI/SMBIOS, etc.). If > possible, maybe skip setting the flag when running virtualized? Indeed. We now check the cpuid ecx HV flag before performing this specific wrmsr. This kind of thing seems to keep coming up. In our codebase, we have a way to handle a faulting rdmsr. We don't have a maching way to handle a faulting wrmsr. We sort of don't want the latter..
Re: Zenbleed
Hi, (Hetzner engineer here, but speaking as a private individual) Hetzner Cloud is using regular mainline QEMU on Linux as the hypervisor, so while I'd agree that faulting when the MSR is set isn't ideal, this behaviour will probably also occur on a lot of other machines/setups. Another solution might be preferrable. Linux has started to do a very similar thing and I haven't yet checked if it runs into a similar situation/fault. At least on Hetzner Cloud, there is a lot of indications that the machine is running as a virtual machine (ACPI/SMBIOS, etc.). If possible, maybe skip setting the flag when running virtualized? So long, Manawyrm -- https://tbspace.de
Re: Zenbleed
I hit the same trap as Lucas with my HV (netcup.de).
bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xf5900 (10 entries)
bios0: vendor netcup version "RS 1000 G9 Plus" date 12/17/2020
bios0: netcup KVM Server
acpi0 at bios0: ACPI 1.0
acpi0: sleep states S5
acpi0: tables DSDT FACP APIC
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee0: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: AMD EPYC 7702P 64-Core Processor, 1996.52 MHz, 17-31-00
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,CPCTR,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,UMIP,IBRS,IBPB,STIBP,SSBD,IBPB,IBRS,STIBP,SSBD,VIRTSSBD,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
cpu0: 64KB 64b/line 2-way D-cache, 64KB 64b/line 2-way I-cache
cpu0: 512KB 64b/line 16-way L2 cache
cpu0: smt 0, core 0, package 0
https://ibb.co/gZPtLZb
https://ibb.co/sgnTsPb
Based on your diff below the following diff applies on -stable and works
for me.
Index: cpu.c
===
RCS file: /cvs/src/sys/arch/amd64/amd64/cpu.c,v
retrieving revision 1.165
diff -u -p -r1.165 cpu.c
--- cpu.c 9 Mar 2023 13:17:28 - 1.165
+++ cpu.c 25 Jul 2023 06:17:59 -
@@ -1181,7 +1181,8 @@ cpu_fix_msrs(struct cpu_info *ci)
* This MSR is available on all AMD families >= 10h, except 11h
* where LFENCE is always serializing.
*/
- if (family >= 0x10 && family != 0x11) {
+ if (family >= 0x10 && family != 0x11 &&
+ (cpu_ecxfeature & CPUIDECX_HV) == 0) {
msr = rdmsr(MSR_DE_CFG);
if ((msr & DE_CFG_SERIALIZE_LFENCE) == 0) {
msr |= DE_CFG_SERIALIZE_LFENCE;
On Mon Jul 24, 2023 at 10:25:29PM -0600, Theo de Raadt wrote:
> Would like to know if this patch helps anyone with this type of
> problem.
>
> Index: sys/arch/amd64/amd64/cpu.c
> ===
> RCS file: /cvs/src/sys/arch/amd64/amd64/cpu.c,v
> retrieving revision 1.172
> diff -u -p -u -r1.172 cpu.c
> --- sys/arch/amd64/amd64/cpu.c24 Jul 2023 14:53:58 - 1.172
> +++ sys/arch/amd64/amd64/cpu.c25 Jul 2023 03:28:35 -
> @@ -1216,7 +1216,8 @@ cpu_fix_msrs(struct cpu_info *ci)
> if (msr != nmsr)
> wrmsr(MSR_DE_CFG, nmsr);
> }
> - if (family == 0x17 && ci->ci_model >= 0x31) {
> + if (family == 0x17 && ci->ci_model >= 0x31 &&
> + (cpu_ecxfeature & CPUIDECX_HV) == 0) {
> nmsr = msr = rdmsr(MSR_DE_CFG);
> nmsr |= DE_CFG_SERIALIZE_9;
> if (msr != nmsr)
> Index: sys/arch/i386/i386/machdep.c
> ===
> RCS file: /cvs/src/sys/arch/i386/i386/machdep.c,v
> retrieving revision 1.664
> diff -u -p -u -r1.664 machdep.c
> --- sys/arch/i386/i386/machdep.c 24 Jul 2023 14:54:00 - 1.664
> +++ sys/arch/i386/i386/machdep.c 25 Jul 2023 03:28:29 -
> @@ -1993,7 +1993,8 @@ identifycpu(struct cpu_info *ci)
> if (msr != nmsr)
> wrmsr(MSR_DE_CFG, nmsr);
> }
> - if (family == 0x17 && ci->ci_model >= 0x31) {
> + if (family == 0x17 && ci->ci_model >= 0x31 &&
> + (cpu_ecxfeature & CPUIDECX_HV) == 0) {
> nmsr = msr = rdmsr(MSR_DE_CFG);
> nmsr |= DE_CFG_SERIALIZE_9;
> if (msr != nmsr)
>
Re: Zenbleed
Would like to know if this patch helps anyone with this type of
problem.
Index: sys/arch/amd64/amd64/cpu.c
===
RCS file: /cvs/src/sys/arch/amd64/amd64/cpu.c,v
retrieving revision 1.172
diff -u -p -u -r1.172 cpu.c
--- sys/arch/amd64/amd64/cpu.c 24 Jul 2023 14:53:58 - 1.172
+++ sys/arch/amd64/amd64/cpu.c 25 Jul 2023 03:28:35 -
@@ -1216,7 +1216,8 @@ cpu_fix_msrs(struct cpu_info *ci)
if (msr != nmsr)
wrmsr(MSR_DE_CFG, nmsr);
}
- if (family == 0x17 && ci->ci_model >= 0x31) {
+ if (family == 0x17 && ci->ci_model >= 0x31 &&
+ (cpu_ecxfeature & CPUIDECX_HV) == 0) {
nmsr = msr = rdmsr(MSR_DE_CFG);
nmsr |= DE_CFG_SERIALIZE_9;
if (msr != nmsr)
Index: sys/arch/i386/i386/machdep.c
===
RCS file: /cvs/src/sys/arch/i386/i386/machdep.c,v
retrieving revision 1.664
diff -u -p -u -r1.664 machdep.c
--- sys/arch/i386/i386/machdep.c24 Jul 2023 14:54:00 - 1.664
+++ sys/arch/i386/i386/machdep.c25 Jul 2023 03:28:29 -
@@ -1993,7 +1993,8 @@ identifycpu(struct cpu_info *ci)
if (msr != nmsr)
wrmsr(MSR_DE_CFG, nmsr);
}
- if (family == 0x17 && ci->ci_model >= 0x31) {
+ if (family == 0x17 && ci->ci_model >= 0x31 &&
+ (cpu_ecxfeature & CPUIDECX_HV) == 0) {
nmsr = msr = rdmsr(MSR_DE_CFG);
nmsr |= DE_CFG_SERIALIZE_9;
if (msr != nmsr)
Re: Zenbleed
I am not aware of a feature flag we can check for if we can
write to MSR_DE_CFG, and my guess is their VM needs to silently
ignore the bits we modify, and not generate a fault.
I'm not sure what we can do here, but I suspect some developers will
think about it.
Anyways, they decided to fault. That seems wrong. I think they will
get other complaints in the near future. I think you should let them
know. It is an easy fix on their side.
Lucas wrote:
> Hey,
>
> I'm running this on a Hetzner VPS, with the dmesg below. I ran syspatch
> followed by installboot -v sd0. After each boot, 100% I get
>
> cpu0 at mainbus0: apid 0 (boot processor)
> cpu0: AMD EPYC Processor, 2445.60 MHz, 17-31-00
> cpu0: FPU,... (proc capabilities)
> cpu0: 32KB ... (cache info)
> cpu0: smt 0, core 0, package 0
> kernel: protection fault trap, code=0
> Stopped at cpu_fix_msrs+0x1a4: wrmsr
> ddb{0}> show panic
> the kernel did not panic
> ddb{0}> show reg
> rdi 0x8243dff0cpu_info_full_primary+0x1ff0
> rsi0x2
> rbp 0x829277f0end+0x3277f0
> rbx 0x80043880
> rdx 0
> rcx 0xc0011029
> rax 0x202
> r8 0x101010101010101
> r9 0x8080808080808080
> r10 0xf74eb8da9a13c7be
> r11 0x4299783767f1ac05
> r12 0x800438a4
> r13 0x800020a11000
> r14 0x8243dff0cpu_info_full_primary+0x1ff0
> r15 0x17
> rip 0x811b50f4cpu_fix_msrs+0x1a4
> cs 0x8
> rflags 0x10202__ALIGN_SIZE+0xf202
> rsp 0x829277c0end+0x3277c0
> ss0x10
> cpu_fix_msrs+0x1a4: wrmsr
> ddb{0}> bt
> cpu_fix_msrs(...) at cpu_fix_msrs+0x1a4
> cpu_attach(...) at cpu_attach+0x3fd
> config_attach(...) at config_attach+0x1f4
> acpimadt_attach(...) at acpimadt_attach+0x349
> config_attach(...) at config_attach+0x1f4
> acpi_attach_common(...) at acpi_attach_common+0x5db
> config_attach(...) at config_attach+0x1f4
> bios_attach(...) at bios_attach+0x77e
> config_attach(...) at config_attach+0x1f4
> mainbus_attach(...) at mainbus_attach+0x778
> config_attach(...) at config_attach+0x1f4
> cpu_configure(...) at cpu_configure+0x33
> main(0,0,0,0,0,1) at main+0x3d3
> end trace fram: 0x0, count: 13
> ddb{0}> mach ddbcpu 1
> Invalid cpu 1
> ddb{0}>
>
>
> I've screenshots of the backtrace, should the function parameters be
> required.
>
> If other people are lucky like, remember that you can `boot /obsd` at
> the boot prompt to boot an older kernel.
>
> After booting the old kernel, I successfully ran fw_update and fetched
> the amd firmware package. Nevertheless, the problem persists. I tried
> rerunning `syspatch` and `installboot -v sd0` again, and the problem
> still persists. In case it's relevant, the installboot output is
>
> Using / as root
> installing bootstrap on /dev/rsd0c
> using first-stage /usr/mdec/biosboot, second-stage /usr/mdec/boot
> copying /usr/mdec/boot to //boot
> looking for superblock at 65536
> found valid ffs2 superblock
> //boot is 6 blocks x 16384 bytes
> fs block shift 2; part offset 64; inode block 56, offset 5232
> expecting 64-bit fs blocks (incr 4)
> master boot record (MBR) at sector 0
> partition 3: type 0xA6 offset 64 size 80003008
> /usr/mdec/biosboot will be written at sector 64
>
>
> OpenBSD 7.3 (GENERIC.MP) #0: Wed Jul 12 05:09:49 MDT 2023
>
> r...@syspatch-73-amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> real mem = 2080227328 (1983MB)
> avail mem = 1997848576 (1905MB)
> random: good seed from bootblocks
> mpath0 at root
> scsibus0 at mpath0: 256 targets
> mainbus0 at root
> bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xf59d0 (10 entries)
> bios0: vendor Hetzner version "2017" date 11/11/2017
> bios0: Hetzner vServer
> acpi0 at bios0: ACPI 3.0
> acpi0: sleep states S5
> acpi0: tables DSDT FACP APIC HPET MCFG WAET
> acpi0: wakeup devices
> acpitimer0 at acpi0: 3579545 Hz, 24 bits
> acpimadt0 at acpi0 addr 0xfee0: PC-AT compat
> cpu0 at mainbus0: apid 0 (boot processor)
> cpu0: AMD EPYC Processor, 2445.58 MHz, 17-31-00
> cpu0:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,TOPEXT,CPCTR,FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,UMIP,IBPB,IBRS,STIBP,SSBD,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
> cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 512KB
> 64b/line 8-way L2 cache, 16MB 64b/line 16-way L3 cache
> cpu0: smt 0, core 0, package 0
> mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
> cpu0: apic clock running at 1000
Re: Zenbleed
Hey,
I'm running this on a Hetzner VPS, with the dmesg below. I ran syspatch
followed by installboot -v sd0. After each boot, 100% I get
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: AMD EPYC Processor, 2445.60 MHz, 17-31-00
cpu0: FPU,... (proc capabilities)
cpu0: 32KB ... (cache info)
cpu0: smt 0, core 0, package 0
kernel: protection fault trap, code=0
Stopped at cpu_fix_msrs+0x1a4: wrmsr
ddb{0}> show panic
the kernel did not panic
ddb{0}> show reg
rdi 0x8243dff0cpu_info_full_primary+0x1ff0
rsi 0x2
rbp 0x829277f0end+0x3277f0
rbx 0x80043880
rdx0
rcx 0xc0011029
rax0x202
r8 0x101010101010101
r90x8080808080808080
r10 0xf74eb8da9a13c7be
r11 0x4299783767f1ac05
r12 0x800438a4
r13 0x800020a11000
r14 0x8243dff0cpu_info_full_primary+0x1ff0
r15 0x17
rip 0x811b50f4cpu_fix_msrs+0x1a4
cs 0x8
rflags 0x10202__ALIGN_SIZE+0xf202
rsp 0x829277c0end+0x3277c0
ss 0x10
cpu_fix_msrs+0x1a4: wrmsr
ddb{0}> bt
cpu_fix_msrs(...) at cpu_fix_msrs+0x1a4
cpu_attach(...) at cpu_attach+0x3fd
config_attach(...) at config_attach+0x1f4
acpimadt_attach(...) at acpimadt_attach+0x349
config_attach(...) at config_attach+0x1f4
acpi_attach_common(...) at acpi_attach_common+0x5db
config_attach(...) at config_attach+0x1f4
bios_attach(...) at bios_attach+0x77e
config_attach(...) at config_attach+0x1f4
mainbus_attach(...) at mainbus_attach+0x778
config_attach(...) at config_attach+0x1f4
cpu_configure(...) at cpu_configure+0x33
main(0,0,0,0,0,1) at main+0x3d3
end trace fram: 0x0, count: 13
ddb{0}> mach ddbcpu 1
Invalid cpu 1
ddb{0}>
I've screenshots of the backtrace, should the function parameters be
required.
If other people are lucky like, remember that you can `boot /obsd` at
the boot prompt to boot an older kernel.
After booting the old kernel, I successfully ran fw_update and fetched
the amd firmware package. Nevertheless, the problem persists. I tried
rerunning `syspatch` and `installboot -v sd0` again, and the problem
still persists. In case it's relevant, the installboot output is
Using / as root
installing bootstrap on /dev/rsd0c
using first-stage /usr/mdec/biosboot, second-stage /usr/mdec/boot
copying /usr/mdec/boot to //boot
looking for superblock at 65536
found valid ffs2 superblock
//boot is 6 blocks x 16384 bytes
fs block shift 2; part offset 64; inode block 56, offset 5232
expecting 64-bit fs blocks (incr 4)
master boot record (MBR) at sector 0
partition 3: type 0xA6 offset 64 size 80003008
/usr/mdec/biosboot will be written at sector 64
OpenBSD 7.3 (GENERIC.MP) #0: Wed Jul 12 05:09:49 MDT 2023
r...@syspatch-73-amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 2080227328 (1983MB)
avail mem = 1997848576 (1905MB)
random: good seed from bootblocks
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xf59d0 (10 entries)
bios0: vendor Hetzner version "2017" date 11/11/2017
bios0: Hetzner vServer
acpi0 at bios0: ACPI 3.0
acpi0: sleep states S5
acpi0: tables DSDT FACP APIC HPET MCFG WAET
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee0: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: AMD EPYC Processor, 2445.58 MHz, 17-31-00
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,TOPEXT,CPCTR,FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,UMIP,IBPB,IBRS,STIBP,SSBD,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 512KB 64b/line
8-way L2 cache, 16MB 64b/line 16-way L3 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 1000MHz
cpu1 at mainbus0: apid 1 (application processor)
cpu1: AMD EPYC Processor, 2445.89 MHz, 17-31-00
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,TOPEXT,CPCTR,FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,UMIP,IBPB,IBRS,STIBP,SSBD,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
cpu1: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 512KB 64b/line
8-way L2 cache, 16MB 64b/line 16-way L3 cache
Zenbleed
Zenbleed errata for 7.2 and 7.3 will come out soon. sysupgrade of the -current snapshot already contains a fix. I wanted to share some notes on impact: OpenBSD does not use the AVX instructions to the same extent that Linux and Microsoft do, so this is not as important. On Linux, glibc has AVX-based optimizations for simple functions (string and memory copies) which will store secrets into the register file which can be extracted trivially, so the impact on glibc-based systems is HUGE. While working on our fixes, I ran the test programs for quite a while and I never saw anything resembling a 'text' string. However when I ran a browser I saw streams of what was probably graphics-related fragments flowing past. The base system clearly uses AVX very rarely by itself. In summary: in OpenBSD, this isn't a big deal today. However, attacks built upon primitives always get better over time, so I urge everyone to install these workarounds as soon as our errata ship. -- ps. If you use syspatch for these new errata, you must install the bootblocks yourself! syspatch cannot install them for you. So you must run this yourself, before the last reboot: installboot -v sd0 or installboot -v wd0 Our cpu firmware update mechanism uses the bootblocks to load the firmware from disk and provides it to the kernel, so if you don't have new bootblocks you won't be protected.
