Kernel panic on current amd64 running under Ubuntu KVM (patch included)
Hello bugs@, I'm have problem with run OpenBSD current amd64 as guest in KVM hypervisor on Ubuntu server with AMD CPU. Host OS: Ubuntu 12.04 TLS Host CPU: AMD Phenom(tm) II X4 975 Processor stepping 03 Host kernel: Linux M720-US3 3.2.0-43-generic #68-Ubuntu SMP Wed May 15 03:33:33 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux qemu-kvm: 1.2.0+dfsg-0~12.04~ppa0 seabios: 1.7.0-1 OpenBSD crash: acpiprt0 at acpi0: bus 0 (PCI0) mpbios0 at bios0: Intel MP Specification 1.4 cpu0 at mainbus0: apid 0 (boot processor) cpu0: AMD Phenom(tm) 9550 Quad-Core Processor, 3600.53 MHz cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CF LUSH,MMX,FXSR,SSE,SSE2,SSE3,CX16,POPCNT,NXE,MMXX,FFXSR,LONG,3DNOW2,3DNOW,LAHF,CM PLEG,SVM,AMCR8,ABM,SSE4A,MASSE,3DNOWP cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB 64b/line 1 6-way L2 cache cpu0: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped cpu0: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped kernel: protection fault trap, code=0 Stopped at aesni_setup+0x1a0: rdmsr aesni_setup() at aesni setup+0x1a0 amd64_errata() at amd64 errata+0xc9 identifycpu() at identifycpu+0x729 cpu attach() at cpu_attach+0x2ce config_attach() at config_attach+0x1d4 mpbios_cpu() at mpbios_cpu+0x5b mpbios_scan() at mpbios_scan+0x355 config_attach() at config_attach+0x1d4 bios_attach() at bios_attach+0x296 config_attach() at config_attach+0x1d4 end trace frame: 0x81de9e30, count: 0 ddb{0}> ddb{0}> trace aesni_setup() at aesni_setup+0x1a0 amd64_errata() at amd64_errata+0xc9 identifycpu() at identifycpu+0x729 cpu_attach() at cpu_attach+0x2ce config_attach() at config_attach+0x1d4 mpbios_cpu() at mpbios_cpu+0x5b mpbios_scan() at mpbios_scan+0x355 config_attach() at config_attach+0x1d4 bios_attach() at bios_attach+0x296 config_attach() at config_attach+0x1d4 mainbus_attach() at mainbus_attach+0x5b config_attach() at config_attach+0x1d4 cpu_configure() at cpu_configure+0x17 main() at main+0x3f5 end trace frame: 0x0, count: -14 ddb{0}> ddb{0}>ps PID PPID PGRP UID S FLAGS WAIT COMMAND *0 -10 0 7 0x200 swapper ddb{0}> After ported and applied patch from NetBSD ( http://cvsweb.netbsd.org/bsdweb.cgi/src/sys/arch/x86/x86/errata.c.diff?r1=1.20&r2=1.21&only_with_tag=MAIN&f=h ), OpenBSD run without crash: OpenBSD 5.3-current (GENERIC.MP) #0: Mon May 20 02:56:29 EEST 2013 root@buildbox.local:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 1056956416 (1007MB) avail mem = 1021165568 (973MB) mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xfd970 (11 entries) bios0: vendor Bochs version "Bochs" date 01/01/2007 bios0: Bochs Bochs acpi0 at bios0: rev 0 acpi0: sleep states S3 S4 S5 acpi0: tables DSDT FACP SSDT APIC HPET SSDT acpi0: wakeup devices acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee0: PC-AT compat acpihpet0 at acpi0: 1 Hz acpiprt0 at acpi0: bus 0 (PCI0) mpbios0 at bios0: Intel MP Specification 1.4 cpu0 at mainbus0: apid 0 (boot processor) cpu0: AMD Phenom(tm) 9550 Quad-Core Processor, 3600.47 MHz cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,CX16,POPCNT,RAZ,NXE,MMXX,FFXSR,LONG,3DNOW2,3DNOW,LAHF,CMPLEG,SVM,AMCR8,ABM,SSE4A,MASSE,3DNOWP cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB 64b/line 16-way L2 cache cpu0: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped cpu0: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped cpu0: apic clock running at 999MHz cpu1 at mainbus0: apid 1 (application processor) cpu1: AMD Phenom(tm) 9550 Quad-Core Processor, 3600.30 MHz cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,CX16,POPCNT,RAZ,NXE,MMXX,FFXSR,LONG,3DNOW2,3DNOW,LAHF,CMPLEG,SVM,AMCR8,ABM,SSE4A,MASSE,3DNOWP cpu1: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB 64b/line 16-way L2 cache cpu1: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped cpu1: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped mpbios0: bus 0 is type PCI mpbios0: bus 1 is type ISA ioapic0 at mainbus0: apid 2 pa 0xfec0, version 11, 24 pins ioapic0: misconfigured as apic 0, remapped to apid 2 pci0 at mainbus0 bus 0 pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02 pcib0 at pci0 dev 1 function 0 "Intel 82371SB ISA" rev 0x00 pciide0 at pci0 dev 1 function 1 "Intel 82371SB IDE" rev 0x00: DMA, channel 0 wired to compatibility, channel 1 wired to compatibility pciide0: channel 0 disabled (no drives) atapiscsi0 at pciide0 channel 1 drive 0 scsibus0 at atapiscsi0: 2 targets cd0 at scsibus0 targ 0 lun 0: ATAPI 5/cdrom removable cd0(pciide0:1:0): using PIO mode 4, DMA mode 2 uhci0 at pci0 dev 1 function 2 "Intel 82371SB USB" rev 0x01: apic 2 int 11 piixpm0 at pci0 dev 1 function 3 "Intel 82371AB Power" rev 0x03: apic 2 int 9 iic0 at piixpm0 iic0: addr 0x4c 4
Re: Kernel panic on current amd64 running under Ubuntu KVM (patch included)
patch_amd64errata.diff Index: amd64/amd64/amd64errata.c === RCS file: /cvs/src/sys/arch/amd64/amd64/amd64errata.c,v retrieving revision 1.3 diff -u -p -u -p -r1.3 amd64errata.c --- amd64/amd64/amd64errata.c27 Mar 2012 05:59:46 -1.3 +++ amd64/amd64/amd64errata.c19 May 2013 23:48:11 - @@ -293,6 +293,9 @@ amd64_errata(struct cpu_info *ci) int found = 0; int corrected = 0; +if (ci->ci_feature_eflags & CPUIDECX_RAZ) +return; + CPUID(0x8001, code, dummy, dummy, dummy); for (i = 0; ; i += 2) { Index: amd64/amd64/identcpu.c === RCS file: /cvs/src/sys/arch/amd64/amd64/identcpu.c,v retrieving revision 1.47 diff -u -p -u -p -r1.47 identcpu.c --- amd64/amd64/identcpu.c6 May 2013 04:32:12 -1.47 +++ amd64/amd64/identcpu.c19 May 2013 23:48:11 - @@ -129,6 +129,7 @@ const struct { { CPUIDECX_AVX,"AVX" }, { CPUIDECX_F16C,"F16C" }, { CPUIDECX_RDRAND,"RDRAND" }, +{ CPUIDECX_RAZ,"RAZ" } }, cpu_ecpuid_ecxfeatures[] = { { CPUIDECX_LAHF,"LAHF" }, { CPUIDECX_CMPLEG,"CMPLEG" }, Index: amd64/include/specialreg.h === RCS file: /cvs/src/sys/arch/amd64/include/specialreg.h,v retrieving revision 1.25 diff -u -p -u -p -r1.25 specialreg.h --- amd64/include/specialreg.h6 May 2013 04:32:12 -1.25 +++ amd64/include/specialreg.h19 May 2013 23:48:11 - @@ -158,6 +158,7 @@ #defineCPUIDECX_AVX0x1000/* Advanced Vector Extensions */ #defineCPUIDECX_F16C0x2000/* 16bit fp conversion */ #defineCPUIDECX_RDRAND0x4000/* RDRAND instruction */ +#defineCPUIDECX_RAZ0x8000/* RAZ. Indicates guest state. */ /* * "Structured Extended Feature Flags Parameters" (CPUID function 0x7, leaf 0)
Re: Kernel panic on current amd64 running under Ubuntu KVM (patch included)
$ cat patch_amd64errata.diff Index: amd64/amd64/amd64errata.c === RCS file: /cvs/src/sys/arch/amd64/amd64/amd64errata.c,v retrieving revision 1.3 diff -u -p -u -p -r1.3 amd64errata.c --- amd64/amd64/amd64errata.c27 Mar 2012 05:59:46 -1.3 +++ amd64/amd64/amd64errata.c19 May 2013 23:48:11 - @@ -293,6 +293,9 @@ amd64_errata(struct cpu_info *ci) int found = 0; int corrected = 0; +if (ci->ci_feature_eflags & CPUIDECX_RAZ) +return; + CPUID(0x8001, code, dummy, dummy, dummy); for (i = 0; ; i += 2) { Index: amd64/amd64/identcpu.c === RCS file: /cvs/src/sys/arch/amd64/amd64/identcpu.c,v retrieving revision 1.47 diff -u -p -u -p -r1.47 identcpu.c --- amd64/amd64/identcpu.c6 May 2013 04:32:12 -1.47 +++ amd64/amd64/identcpu.c19 May 2013 23:48:11 - @@ -129,6 +129,7 @@ const struct { { CPUIDECX_AVX,"AVX" }, { CPUIDECX_F16C,"F16C" }, { CPUIDECX_RDRAND,"RDRAND" }, +{ CPUIDECX_RAZ,"RAZ" } }, cpu_ecpuid_ecxfeatures[] = { { CPUIDECX_LAHF,"LAHF" }, { CPUIDECX_CMPLEG,"CMPLEG" }, Index: amd64/include/specialreg.h === RCS file: /cvs/src/sys/arch/amd64/include/specialreg.h,v retrieving revision 1.25 diff -u -p -u -p -r1.25 specialreg.h --- amd64/include/specialreg.h6 May 2013 04:32:12 -1.25 +++ amd64/include/specialreg.h19 May 2013 23:48:11 - @@ -158,6 +158,7 @@ #defineCPUIDECX_AVX0x1000/* Advanced Vector Extensions */ #defineCPUIDECX_F16C0x2000/* 16bit fp conversion */ #defineCPUIDECX_RDRAND0x4000/* RDRAND instruction */ +#defineCPUIDECX_RAZ0x8000/* RAZ. Indicates guest state. */ /* * "Structured Extended Feature Flags Parameters" (CPUID function 0x7, leaf 0) 2013/5/20 Roman Kravchuk > Hello bugs@, > > I'm have problem with run OpenBSD current amd64 as guest in KVM hypervisor > on Ubuntu server with AMD CPU. > > Host OS: Ubuntu 12.04 TLS > Host CPU: AMD Phenom(tm) II X4 975 Processor stepping 03 > Host kernel: Linux M720-US3 3.2.0-43-generic #68-Ubuntu SMP Wed May 15 > 03:33:33 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux > qemu-kvm: 1.2.0+dfsg-0~12.04~ppa0 > seabios: 1.7.0-1 > > > OpenBSD crash: > > acpiprt0 at acpi0: bus 0 (PCI0) > mpbios0 at bios0: Intel MP Specification 1.4 > cpu0 at mainbus0: apid 0 (boot processor) > cpu0: AMD Phenom(tm) 9550 Quad-Core Processor, 3600.53 MHz > cpu0: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CF > > LUSH,MMX,FXSR,SSE,SSE2,SSE3,CX16,POPCNT,NXE,MMXX,FFXSR,LONG,3DNOW2,3DNOW,LAHF,CM > PLEG,SVM,AMCR8,ABM,SSE4A,MASSE,3DNOWP > cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB > 64b/line 1 > 6-way L2 cache > cpu0: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped > cpu0: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped > kernel: protection fault trap, code=0 > Stopped at aesni_setup+0x1a0: rdmsr > aesni_setup() at aesni setup+0x1a0 > amd64_errata() at amd64 errata+0xc9 > identifycpu() at identifycpu+0x729 > cpu attach() at cpu_attach+0x2ce > config_attach() at config_attach+0x1d4 > mpbios_cpu() at mpbios_cpu+0x5b > mpbios_scan() at mpbios_scan+0x355 > config_attach() at config_attach+0x1d4 > bios_attach() at bios_attach+0x296 > config_attach() at config_attach+0x1d4 > end trace frame: 0x81de9e30, count: 0 > ddb{0}> > > > ddb{0}> trace > aesni_setup() at aesni_setup+0x1a0 > amd64_errata() at amd64_errata+0xc9 > identifycpu() at identifycpu+0x729 > cpu_attach() at cpu_attach+0x2ce > config_attach() at config_attach+0x1d4 > mpbios_cpu() at mpbios_cpu+0x5b > mpbios_scan() at mpbios_scan+0x355 > config_attach() at config_attach+0x1d4 > bios_attach() at bios_attach+0x296 > config_attach() at config_attach+0x1d4 > mainbus_attach() at mainbus_attach+0x5b > config_attach() at config_attach+0x1d4 > cpu_configure() at cpu_configure+0x17 > main() at main+0x3f5 > end trace frame: 0x0, count: -14 > ddb{0}> > > ddb{0}>ps > PID PPID PGRP UID S FLAGS WAIT COMMAND > *0 -10 0 7 0x200 swapper > ddb{0}> > > > > After ported and applied patch from NetBSD ( > http://cvsweb.netbsd.org/bsdweb.cgi/src/sys/arch/x86/x86/errata.c.diff?r1=1.20&r2=1.21&only_with_tag=MAIN&f=h > ), > OpenBSD run without crash: > > OpenBSD 5.3-current (GENERIC.MP) #0: Mon May 20 02:56:29 EEST 2013 > root@buildbox.local:/usr/src/sys/arch/amd64/compile/GENERIC.MP > real mem = 1056956416 (1007MB) > avail mem = 1021165568 (973MB) > mainbus0 at root > bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xfd970 (11 entries) > bios0: vendor Bochs version "Bochs" date 01/01/2007 > bios0: Bochs Bochs > acpi0 at bios0: rev 0 > acpi0: sleep states S3 S4 S5 > acpi0: tables DSDT FACP SSDT APIC HPET SSDT > acpi0: wakeup
Re: Kernel panic on current amd64 running under Ubuntu KVM (patch included)
That is quite a large hammer. It would be preferable to find out which msr it objects to and guard it with a specific cpuid check, and/or fix the hypervisor. On Mon, May 20, 2013 at 10:34:43AM +0300, Roman Kravchuk wrote: > patch_amd64errata.diff > > Index: amd64/amd64/amd64errata.c > === > RCS file: /cvs/src/sys/arch/amd64/amd64/amd64errata.c,v > retrieving revision 1.3 > diff -u -p -u -p -r1.3 amd64errata.c > --- amd64/amd64/amd64errata.c27 Mar 2012 05:59:46 -1.3 > +++ amd64/amd64/amd64errata.c19 May 2013 23:48:11 - > @@ -293,6 +293,9 @@ amd64_errata(struct cpu_info *ci) > int found = 0; > int corrected = 0; > > +if (ci->ci_feature_eflags & CPUIDECX_RAZ) > +return; > + > CPUID(0x8001, code, dummy, dummy, dummy); > > for (i = 0; ; i += 2) { > Index: amd64/amd64/identcpu.c > === > RCS file: /cvs/src/sys/arch/amd64/amd64/identcpu.c,v > retrieving revision 1.47 > diff -u -p -u -p -r1.47 identcpu.c > --- amd64/amd64/identcpu.c6 May 2013 04:32:12 -1.47 > +++ amd64/amd64/identcpu.c19 May 2013 23:48:11 - > @@ -129,6 +129,7 @@ const struct { > { CPUIDECX_AVX,"AVX" }, > { CPUIDECX_F16C,"F16C" }, > { CPUIDECX_RDRAND,"RDRAND" }, > +{ CPUIDECX_RAZ,"RAZ" } > }, cpu_ecpuid_ecxfeatures[] = { > { CPUIDECX_LAHF,"LAHF" }, > { CPUIDECX_CMPLEG,"CMPLEG" }, > Index: amd64/include/specialreg.h > === > RCS file: /cvs/src/sys/arch/amd64/include/specialreg.h,v > retrieving revision 1.25 > diff -u -p -u -p -r1.25 specialreg.h > --- amd64/include/specialreg.h6 May 2013 04:32:12 -1.25 > +++ amd64/include/specialreg.h19 May 2013 23:48:11 - > @@ -158,6 +158,7 @@ > #defineCPUIDECX_AVX0x1000/* Advanced Vector Extensions */ > #defineCPUIDECX_F16C0x2000/* 16bit fp conversion */ > #defineCPUIDECX_RDRAND0x4000/* RDRAND instruction */ > +#defineCPUIDECX_RAZ0x8000/* RAZ. Indicates guest state. */ > > /* > * "Structured Extended Feature Flags Parameters" (CPUID function 0x7, > leaf 0)
Re: Kernel panic on current amd64 running under Ubuntu KVM (patch included)
Jonathan Gray jsg.id.au> writes: > That is quite a large hammer. It would be preferable to > find out which msr it objects to and guard it with a specific cpuid > check, and/or fix the hypervisor. >From NetBSD PR/47677 (http://gnats.netbsd.org/47677) "I think x86_errata should be avoided if NetBSD running on virtual machine because accesses to MSR may be restricted" While above statement is an assumption, it is highly possible for hypervisor to restrict operations on model-specific regs. For me it's kinda grounded approach, because erratas applied on "host level" may conflict with what guests want, increasing a possibility of screwing up things. So what is the point for guest to run erratas if hypervisor may already done that upwards?
Re: Kernel panic on current amd64 running under Ubuntu KVM (patch included)
On Mon, May 20, 2013 at 01:50:39PM +, Alexey E. Suslikov wrote: > Jonathan Gray jsg.id.au> writes: > > > That is quite a large hammer. It would be preferable to > > find out which msr it objects to and guard it with a specific cpuid > > check, and/or fix the hypervisor. > > >From NetBSD PR/47677 (http://gnats.netbsd.org/47677) > > "I think x86_errata should be avoided if NetBSD running > on virtual machine because accesses to MSR may be restricted" > > While above statement is an assumption, it is highly possible > for hypervisor to restrict operations on model-specific regs. > > For me it's kinda grounded approach, because erratas applied > on "host level" may conflict with what guests want, increasing > a possibility of screwing up things. > > So what is the point for guest to run erratas if hypervisor > may already done that upwards? Filling the kernel full of if not really running on hardware do something else paths is asking for trouble. If a hypervisor claims to be a specific model of a processor it should not be surprised when it gets msrs that processor can handle.
Re: Kernel panic on current amd64 running under Ubuntu KVM (patch included)
Jonathan Gray jsg.id.au> writes: > > So what is the point for guest to run erratas if hypervisor > > may already done that upwards? > > Filling the kernel full of if not really running on hardware > do something else paths is asking for trouble. If a hypervisor > claims to be a specific model of a processor it should not be > surprised when it gets msrs that processor can handle. Have you noticed kernel: protection fault trap, code=0 Stopped in pid 0.1 (system) at netbsd:rdmsr_locked+0xb: rdmsr in original NetBSD PR? Is there a possibility to tell if MSR access in locked or not without receiving a protection fault trap?
Re: Kernel panic on current amd64 running under Ubuntu KVM (patch included)
Jonathan Gray jsg.id.au> writes: > Filling the kernel full of if not really running on hardware > do something else paths is asking for trouble. If a hypervisor > claims to be a specific model of a processor it should not be > surprised when it gets msrs that processor can handle. > > FreeBSD do --- head/sys/amd64/amd64/initcpu.c 2012/03/30 16:32:41 233702 +++ head/sys/amd64/amd64/initcpu.c 2012/08/07 08:36:10 239125 @@ -91,11 +91,17 @@ * * http://support.amd.com/us/Processor_TechDocs/41322_10h_Rev_Gd.pdf * http://support.amd.com/us/Processor_TechDocs/44739_12h_Rev_Gd.pdf +* +* Hypervisors do not provide access to the errata MSR, +* causing #GP exception on attempt to apply the errata. The +* MSR write shall be done on host and persist globally +* anyway, so do not try to do it when under virtualization. */ switch (CPUID_TO_FAMILY(cpu_id)) { case 0x10: case 0x12: - wrmsr(0xc0011029, rdmsr(0xc0011029) | 1); + if ((cpu_feature2 & CPUID2_HV) == 0) + wrmsr(0xc0011029, rdmsr(0xc0011029) | 1); break; } }
Re: Kernel panic on current amd64 running under Ubuntu KVM (patch included)
On Mon, May 20, 2013 at 09:16:21AM +0300, Roman Kravchuk wrote: > I'm have problem with run OpenBSD current amd64 as guest in KVM hypervisor > on Ubuntu server with AMD CPU. .. > cpu0 at mainbus0: apid 0 (boot processor) > cpu0: AMD Phenom(tm) 9550 Quad-Core Processor, 3600.53 MHz > cpu0: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CF > LUSH,MMX,FXSR,SSE,SSE2,SSE3,CX16,POPCNT,NXE,MMXX,FFXSR,LONG,3DNOW2,3DNOW,LAHF,CM > PLEG,SVM,AMCR8,ABM,SSE4A,MASSE,3DNOWP > cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB > 64b/line 1 > 6-way L2 cache > cpu0: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped > cpu0: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped > kernel: protection fault trap, code=0 > Stopped at aesni_setup+0x1a0: rdmsr > aesni_setup() at aesni setup+0x1a0 > amd64_errata() at amd64 errata+0xc9 > identifycpu() at identifycpu+0x729 > cpu attach() at cpu_attach+0x2ce > config_attach() at config_attach+0x1d4 > mpbios_cpu() at mpbios_cpu+0x5b > mpbios_scan() at mpbios_scan+0x355 > config_attach() at config_attach+0x1d4 > bios_attach() at bios_attach+0x296 > config_attach() at config_attach+0x1d4 > end trace frame: 0x81de9e30, count: 0 > ddb{0}> This is another KVM bug. It's pretending to be an AMD CPU, but not emulating one that actually exists. -Bryan.
Re: Kernel panic on current amd64 running under Ubuntu KVM (patch included)
On Mon, May 20, 2013 at 02:33:07PM +, Alexey E. Suslikov wrote: > Jonathan Gray jsg.id.au> writes: > > > Filling the kernel full of if not really running on hardware > > do something else paths is asking for trouble. If a hypervisor > > claims to be a specific model of a processor it should not be > > surprised when it gets msrs that processor can handle. > > > > > > FreeBSD do Who cares? > > --- head/sys/amd64/amd64/initcpu.c2012/03/30 16:32:41 233702 > +++ head/sys/amd64/amd64/initcpu.c2012/08/07 08:36:10 239125 > @@ -91,11 +91,17 @@ >* >* http://support.amd.com/us/Processor_TechDocs/41322_10h_Rev_Gd.pdf >* http://support.amd.com/us/Processor_TechDocs/44739_12h_Rev_Gd.pdf > + * > + * Hypervisors do not provide access to the errata MSR, > + * causing #GP exception on attempt to apply the errata. The > + * MSR write shall be done on host and persist globally > + * anyway, so do not try to do it when under virtualization. >*/ > switch (CPUID_TO_FAMILY(cpu_id)) { > case 0x10: > case 0x12: > - wrmsr(0xc0011029, rdmsr(0xc0011029) | 1); > + if ((cpu_feature2 & CPUID2_HV) == 0) > + wrmsr(0xc0011029, rdmsr(0xc0011029) | 1); > break; > } > } > You're kidding, right?
Re: Kernel panic on current amd64 running under Ubuntu KVM (patch included)
Mike Larkin azathoth.net> writes: > You're kidding, right? Could you guys tell what exactly wrong with FreeBSD/NetBSD approach?
Re: Kernel panic on current amd64 running under Ubuntu KVM (patch included)
> > You're kidding, right? > > Could you guys tell what exactly wrong with FreeBSD/NetBSD approach? So basically, we're told that these VM's are emulating real machines. Except when we try to manage the hardware -- in the specific way that the specific hardware must be handled -- the VM fails to behave like the real machine which it says it is. Do you simply lack comprehension of where we'll end up a decade of following that road? If these VM's are real VM's the should start emulating the machines they claim to be emulating correctly, or they should start advertising that they are something "different", so that we can isolate the bullshit factor.
Re: Kernel panic on current amd64 running under Ubuntu KVM (patch included)
On Mon, May 20, 2013 at 6:15 PM, Alexey E. Suslikov wrote: > Mike Larkin azathoth.net> writes: > >> You're kidding, right? > > Could you guys tell what exactly wrong with FreeBSD/NetBSD approach? > Adding some silly workarounds to the kernel for a bug that is a *KVM bug*.
Re: Kernel panic on current amd64 running under Ubuntu KVM (patch included)
Theo de Raadt cvs.openbsd.org> writes: > If these VM's are real VM's the should start emulating the machines > they claim to be emulating correctly, or they should start advertising > that they are something "different", so that we can isolate the bullshit > factor. Ok. I see. Could we trim that down to the following? --- sys/arch/amd64/amd64/identcpu.c.origMon May 20 19:58:06 2013 +++ sys/arch/amd64/amd64/identcpu.c Mon May 20 20:01:08 2013 @@ -127,6 +127,7 @@ { CPUIDECX_AVX, "AVX" }, { CPUIDECX_F16C,"F16C" }, { CPUIDECX_RDRAND, "RDRAND" }, + { CPUIDECX_HV, "HV" }, }, cpu_ecpuid_ecxfeatures[] = { { CPUIDECX_LAHF,"LAHF" }, { CPUIDECX_CMPLEG, "CMPLEG" }, --- sys/arch/amd64/include/specialreg.h.origMon May 20 20:01:56 2013 +++ sys/arch/amd64/include/specialreg.h Mon May 20 20:06:09 2013 @@ -158,6 +158,7 @@ #defineCPUIDECX_AVX0x1000 /* Advanced Vector Extensions */ #defineCPUIDECX_F16C 0x2000 /* 16bit fp conversion */ #defineCPUIDECX_RDRAND 0x4000 /* RDRAND instruction */ +#defineCPUIDECX_HV 0x8000 /* Hypervisor presence */ /* * "Structured Extended Feature Flags Parameters" (CPUID function 0x7, leaf 0)
Re: Kernel panic on current amd64 running under Ubuntu KVM (patch included)
On Mon, May 20, 2013 at 05:11:35PM +, Alexey E. Suslikov wrote: > Theo de Raadt cvs.openbsd.org> writes: > > > If these VM's are real VM's the should start emulating the machines > > they claim to be emulating correctly, or they should start advertising > > that they are something "different", so that we can isolate the bullshit > > factor. > > Ok. I see. > > Could we trim that down to the following? > > --- sys/arch/amd64/amd64/identcpu.c.orig Mon May 20 19:58:06 2013 > +++ sys/arch/amd64/amd64/identcpu.c Mon May 20 20:01:08 2013 > @@ -127,6 +127,7 @@ > { CPUIDECX_AVX, "AVX" }, > { CPUIDECX_F16C,"F16C" }, > { CPUIDECX_RDRAND, "RDRAND" }, > + { CPUIDECX_HV, "HV" }, > }, cpu_ecpuid_ecxfeatures[] = { > { CPUIDECX_LAHF,"LAHF" }, > { CPUIDECX_CMPLEG, "CMPLEG" }, > --- sys/arch/amd64/include/specialreg.h.orig Mon May 20 20:01:56 2013 > +++ sys/arch/amd64/include/specialreg.h Mon May 20 20:06:09 2013 > @@ -158,6 +158,7 @@ > #define CPUIDECX_AVX0x1000 /* Advanced Vector Extensions */ > #define CPUIDECX_F16C 0x2000 /* 16bit fp conversion */ > #define CPUIDECX_RDRAND 0x4000 /* RDRAND instruction */ > +#define CPUIDECX_HV 0x8000 /* Hypervisor presence > */ > > /* > * "Structured Extended Feature Flags Parameters" (CPUID function 0x7, leaf > 0) > That's certainly less objectionable but I'm not sure what useful information this diff provides.
Re: Kernel panic on current amd64 running under Ubuntu KVM (patch included)
On Mon, May 20, 2013 at 8:42 PM, Mike Larkin wrote: > On Mon, May 20, 2013 at 05:11:35PM +, Alexey E. Suslikov wrote: >> Theo de Raadt cvs.openbsd.org> writes: >> >> > If these VM's are real VM's the should start emulating the machines >> > they claim to be emulating correctly, or they should start advertising >> > that they are something "different", so that we can isolate the bullshit >> > factor. >> >> Ok. I see. >> >> Could we trim that down to the following? >> >> --- sys/arch/amd64/amd64/identcpu.c.orig Mon May 20 19:58:06 2013 >> +++ sys/arch/amd64/amd64/identcpu.c Mon May 20 20:01:08 2013 >> @@ -127,6 +127,7 @@ >> { CPUIDECX_AVX, "AVX" }, >> { CPUIDECX_F16C,"F16C" }, >> { CPUIDECX_RDRAND, "RDRAND" }, >> + { CPUIDECX_HV, "HV" }, >> }, cpu_ecpuid_ecxfeatures[] = { >> { CPUIDECX_LAHF,"LAHF" }, >> { CPUIDECX_CMPLEG, "CMPLEG" }, >> --- sys/arch/amd64/include/specialreg.h.orig Mon May 20 20:01:56 2013 >> +++ sys/arch/amd64/include/specialreg.h Mon May 20 20:06:09 2013 >> @@ -158,6 +158,7 @@ >> #define CPUIDECX_AVX0x1000 /* Advanced Vector Extensions >> */ >> #define CPUIDECX_F16C 0x2000 /* 16bit fp conversion */ >> #define CPUIDECX_RDRAND 0x4000 /* RDRAND instruction */ >> +#define CPUIDECX_HV 0x8000 /* Hypervisor presence >> */ >> >> /* >> * "Structured Extended Feature Flags Parameters" (CPUID function 0x7, leaf >> 0) >> > > That's certainly less objectionable but I'm not sure what useful information > this diff provides. Seen in dmesg, HV flag will indicate operating system is run under hypervisor and weird things are possible while running kernel code which depends on CPU features. After all, it is kinda documented by AMD on page 570 of http://support.amd.com/us/Processor_TechDocs/24594_APM_v3.pdf (AMD named it RAZ, but I put meaningful name like in FreeBSD - should we put a reference to above mentioned document near the define?).
Re: Kernel panic on current amd64 running under Ubuntu KVM (patch included)
> On Mon, May 20, 2013 at 8:42 PM, Mike Larkin wrote: > > On Mon, May 20, 2013 at 05:11:35PM +, Alexey E. Suslikov wrote: > >> Theo de Raadt cvs.openbsd.org> writes: > >> > >> > If these VM's are real VM's the should start emulating the machines > >> > they claim to be emulating correctly, or they should start advertising > >> > that they are something "different", so that we can isolate the bullshit > >> > factor. > >> > >> Ok. I see. > >> > >> Could we trim that down to the following? > >> > >> --- sys/arch/amd64/amd64/identcpu.c.orig Mon May 20 19:58:06 2013 > >> +++ sys/arch/amd64/amd64/identcpu.c Mon May 20 20:01:08 2013 > >> @@ -127,6 +127,7 @@ > >> { CPUIDECX_AVX, "AVX" }, > >> { CPUIDECX_F16C,"F16C" }, > >> { CPUIDECX_RDRAND, "RDRAND" }, > >> + { CPUIDECX_HV, "HV" }, > >> }, cpu_ecpuid_ecxfeatures[] = { > >> { CPUIDECX_LAHF,"LAHF" }, > >> { CPUIDECX_CMPLEG, "CMPLEG" }, > >> --- sys/arch/amd64/include/specialreg.h.orig Mon May 20 20:01:56 2013 > >> +++ sys/arch/amd64/include/specialreg.h Mon May 20 20:06:09 2013 > >> @@ -158,6 +158,7 @@ > >> #define CPUIDECX_AVX0x1000 /* Advanced Vector > >> Extensions */ > >> #define CPUIDECX_F16C 0x2000 /* 16bit fp conversion */ > >> #define CPUIDECX_RDRAND 0x4000 /* RDRAND instruction */ > >> +#define CPUIDECX_HV 0x8000 /* Hypervisor > >> presence */ > >> > >> /* > >> * "Structured Extended Feature Flags Parameters" (CPUID function 0x7, > >> leaf 0) > >> > > > > That's certainly less objectionable but I'm not sure what useful information > > this diff provides. > > Seen in dmesg, HV flag will indicate operating system is run under hypervisor > and weird things are possible while running kernel code which depends on CPU > features. > > After all, it is kinda documented by AMD on page 570 of > http://support.amd.com/us/Processor_TechDocs/24594_APM_v3.pdf > > (AMD named it RAZ, but I put meaningful name like in FreeBSD - should we > put a reference to above mentioned document near the define?). Your statements here are trying to convince us of something which is false. AMD says "RAZ. Reserved for use by hypervisor to indicate guest status." That sentence does not translate to "The hypervisor is heavily broken and fails to completely emulate the machine otherwise advertised by the rest of the CPUID fields". It does not indicate that weird things are possible. If those weird things are possible, it is not because the hardware is broken, but because the *emulation of the hardware* by the VM is broken!
Re: Kernel panic on current amd64 running under Ubuntu KVM (patch included)
On Mon, 20 May 2013, Alexey Suslikov wrote: > Seen in dmesg, HV flag will indicate operating system is run under > hypervisor and weird things are possible while running kernel code which > depends on CPU features. > > After all, it is kinda documented by AMD on page 570 of > http://support.amd.com/us/Processor_TechDocs/24594_APM_v3.pdf > > (AMD named it RAZ, but I put meaningful name like in FreeBSD - should we > put a reference to above mentioned document near the define?). AMD did not name it "RAZ". "RAZ" stands for "read as zero", a condition that is true off many bits in the flags register, etc. Calling it RAZ would be like saying that the street I live on is named "One Way Street". The fact that neither AMD or Intel give a real definition for this makes it sound like KVM squatted on the bit without actually getting a real reservation for it. (In general, when Intel and AMD disagree about the name of a CPUID bit, we prefer the name given by Intel.) Philip
Re: Kernel panic on current amd64 running under Ubuntu KVM (patch included)
On Mon, May 20, 2013 at 8:59 PM, Theo de Raadt wrote: >> On Mon, May 20, 2013 at 8:42 PM, Mike Larkin wrote: >> > On Mon, May 20, 2013 at 05:11:35PM +, Alexey E. Suslikov wrote: >> >> Theo de Raadt cvs.openbsd.org> writes: >> >> >> >> > If these VM's are real VM's the should start emulating the machines >> >> > they claim to be emulating correctly, or they should start advertising >> >> > that they are something "different", so that we can isolate the bullshit >> >> > factor. >> >> >> >> Ok. I see. >> >> >> >> Could we trim that down to the following? >> >> >> >> --- sys/arch/amd64/amd64/identcpu.c.orig Mon May 20 19:58:06 2013 >> >> +++ sys/arch/amd64/amd64/identcpu.c Mon May 20 20:01:08 2013 >> >> @@ -127,6 +127,7 @@ >> >> { CPUIDECX_AVX, "AVX" }, >> >> { CPUIDECX_F16C,"F16C" }, >> >> { CPUIDECX_RDRAND, "RDRAND" }, >> >> + { CPUIDECX_HV, "HV" }, >> >> }, cpu_ecpuid_ecxfeatures[] = { >> >> { CPUIDECX_LAHF,"LAHF" }, >> >> { CPUIDECX_CMPLEG, "CMPLEG" }, >> >> --- sys/arch/amd64/include/specialreg.h.orig Mon May 20 20:01:56 2013 >> >> +++ sys/arch/amd64/include/specialreg.h Mon May 20 20:06:09 2013 >> >> @@ -158,6 +158,7 @@ >> >> #define CPUIDECX_AVX0x1000 /* Advanced Vector >> >> Extensions */ >> >> #define CPUIDECX_F16C 0x2000 /* 16bit fp conversion */ >> >> #define CPUIDECX_RDRAND 0x4000 /* RDRAND instruction */ >> >> +#define CPUIDECX_HV 0x8000 /* Hypervisor >> >> presence */ >> >> >> >> /* >> >> * "Structured Extended Feature Flags Parameters" (CPUID function 0x7, >> >> leaf 0) >> >> >> > >> > That's certainly less objectionable but I'm not sure what useful >> > information >> > this diff provides. >> >> Seen in dmesg, HV flag will indicate operating system is run under hypervisor >> and weird things are possible while running kernel code which depends on CPU >> features. >> >> After all, it is kinda documented by AMD on page 570 of >> http://support.amd.com/us/Processor_TechDocs/24594_APM_v3.pdf >> >> (AMD named it RAZ, but I put meaningful name like in FreeBSD - should we >> put a reference to above mentioned document near the define?). > > Your statements here are trying to convince us of something which is > false. > > AMD says "RAZ. Reserved for use by hypervisor to indicate guest status." > > That sentence does not translate to "The hypervisor is heavily broken > and fails to completely emulate the machine otherwise advertised by > the rest of the CPUID fields". > > It does not indicate that weird things are possible. If those weird > things are possible, it is not because the hardware is broken, but > because the *emulation of the hardware* by the VM is broken! You misunderstood my point. I'm not trying to convince, but avoid useless work/talk in the future: * if one will see a *similar* bug report with a dmesg containing RAZ flag, there will be no need for something to explore - it's a 99% broken VM.
Re: Kernel panic on current amd64 running under Ubuntu KVM (patch included)
> On Mon, May 20, 2013 at 8:59 PM, Theo de Raadt > wrote: > >> On Mon, May 20, 2013 at 8:42 PM, Mike Larkin wrote: > >> > On Mon, May 20, 2013 at 05:11:35PM +, Alexey E. Suslikov wrote: > >> >> Theo de Raadt cvs.openbsd.org> writes: > >> >> > >> >> > If these VM's are real VM's the should start emulating the machines > >> >> > they claim to be emulating correctly, or they should start advertising > >> >> > that they are something "different", so that we can isolate the > >> >> > bullshit > >> >> > factor. > >> >> > >> >> Ok. I see. > >> >> > >> >> Could we trim that down to the following? > >> >> > >> >> --- sys/arch/amd64/amd64/identcpu.c.orig Mon May 20 19:58:06 2013 > >> >> +++ sys/arch/amd64/amd64/identcpu.c Mon May 20 20:01:08 2013 > >> >> @@ -127,6 +127,7 @@ > >> >> { CPUIDECX_AVX, "AVX" }, > >> >> { CPUIDECX_F16C,"F16C" }, > >> >> { CPUIDECX_RDRAND, "RDRAND" }, > >> >> + { CPUIDECX_HV, "HV" }, > >> >> }, cpu_ecpuid_ecxfeatures[] = { > >> >> { CPUIDECX_LAHF,"LAHF" }, > >> >> { CPUIDECX_CMPLEG, "CMPLEG" }, > >> >> --- sys/arch/amd64/include/specialreg.h.orig Mon May 20 20:01:56 2013 > >> >> +++ sys/arch/amd64/include/specialreg.h Mon May 20 20:06:09 2013 > >> >> @@ -158,6 +158,7 @@ > >> >> #define CPUIDECX_AVX0x1000 /* Advanced Vector > >> >> Extensions */ > >> >> #define CPUIDECX_F16C 0x2000 /* 16bit fp conversion */ > >> >> #define CPUIDECX_RDRAND 0x4000 /* RDRAND instruction */ > >> >> +#define CPUIDECX_HV 0x8000 /* Hypervisor > >> >> presence */ > >> >> > >> >> /* > >> >> * "Structured Extended Feature Flags Parameters" (CPUID function 0x7, > >> >> leaf 0) > >> >> > >> > > >> > That's certainly less objectionable but I'm not sure what useful > >> > information > >> > this diff provides. > >> > >> Seen in dmesg, HV flag will indicate operating system is run under > >> hypervisor > >> and weird things are possible while running kernel code which depends on > >> CPU > >> features. > >> > >> After all, it is kinda documented by AMD on page 570 of > >> http://support.amd.com/us/Processor_TechDocs/24594_APM_v3.pdf > >> > >> (AMD named it RAZ, but I put meaningful name like in FreeBSD - should we > >> put a reference to above mentioned document near the define?). > > > > Your statements here are trying to convince us of something which is > > false. > > > > AMD says "RAZ. Reserved for use by hypervisor to indicate guest status." > > > > That sentence does not translate to "The hypervisor is heavily broken > > and fails to completely emulate the machine otherwise advertised by > > the rest of the CPUID fields". > > > > It does not indicate that weird things are possible. If those weird > > things are possible, it is not because the hardware is broken, but > > because the *emulation of the hardware* by the VM is broken! > > You misunderstood my point. > > I'm not trying to convince, but avoid useless work/talk in the future: Yes you are. You have an agenda. You want to make us work around a bug, rather than talk to the originators of the problem.
Re: Kernel panic on current amd64 running under Ubuntu KVM (patch included)
On Mon, May 20, 2013 at 9:11 PM, Theo de Raadt wrote: >> I'm not trying to convince, but avoid useless work/talk in the future: > > Yes you are. You have an agenda. > > You want to make us work around a bug, rather than talk to the > originators of the problem. While I see practical use, someone don't. I call this disagreement. There is no problem for me if somebody disagree with a plan I have. It's normal. Btw, Intel's doc I have found at http://www.intel.com/content/dam/www/public/us/en/documents/application-notes/processor-identification-cpuid-instruction-note.pdf says "31 Not Used Always returns 0".
Re: Kernel panic on current amd64 running under Ubuntu KVM (patch included)
On Mon, May 20, 2013 at 21:28, Alexey Suslikov wrote: > > While I see practical use, someone don't. I call this disagreement. There is > no problem for me if somebody disagree with a plan I have. It's normal. > > Btw, Intel's doc I have found at > http://www.intel.com/content/dam/www/public/us/en/documents/application-notes/processor-identification-cpuid-instruction-note.pdf > > > says "31 Not Used Always returns 0". In that case, there's no sense testing for it, because it's always 0. If it isn't 0, then it's not an amd64 computer and we don't support it. Trying to identify all the infinite machines we don't support is fruitless, imo, and perhaps not a path we should start down, because then people will expect us to detect why we don't run on *their* not supported computer.
Re: Kernel panic on current amd64 running under Ubuntu KVM (patch included)
On Mon, May 20, 2013 at 11:58 PM, Ted Unangst wrote: > On Mon, May 20, 2013 at 21:28, Alexey Suslikov wrote: >> >> While I see practical use, someone don't. I call this disagreement. There is >> no problem for me if somebody disagree with a plan I have. It's normal. >> >> Btw, Intel's doc I have found at >> http://www.intel.com/content/dam/www/public/us/en/documents/application-notes/processor-identification-cpuid-instruction-note.pdf >> >> >> says "31 Not Used Always returns 0". > > In that case, there's no sense testing for it, because it's always 0. > > If it isn't 0, then it's not an amd64 computer and we don't support > it. Trying to identify all the infinite machines we don't support is > fruitless, imo, and perhaps not a path we should start down, because > then people will expect us to detect why we don't run on *their* not > supported computer. In practice, it is not zero. This is why my point was opposite: * if I see Hypervisor flag in dmesg, my (virtual) hardware is not guaranteed to operate properly (which is not theoretically, but practically true, because of crash we have).
Re: Kernel panic on current amd64 running under Ubuntu KVM (patch included)
> On Mon, May 20, 2013 at 11:58 PM, Ted Unangst wrote: > > On Mon, May 20, 2013 at 21:28, Alexey Suslikov wrote: > >> > >> While I see practical use, someone don't. I call this disagreement. There > >> is > >> no problem for me if somebody disagree with a plan I have. It's normal. > >> > >> Btw, Intel's doc I have found at > >> http://www.intel.com/content/dam/www/public/us/en/documents/application-notes/processor-identification-cpuid-instruction-note.pdf > >> > >> > >> says "31 Not Used Always returns 0". > > > > In that case, there's no sense testing for it, because it's always 0. > > > > If it isn't 0, then it's not an amd64 computer and we don't support > > it. Trying to identify all the infinite machines we don't support is > > fruitless, imo, and perhaps not a path we should start down, because > > then people will expect us to detect why we don't run on *their* not > > supported computer. > > In practice, it is not zero. This is why my point was opposite: > > * if I see Hypervisor flag in dmesg, my (virtual) hardware is not guaranteed > to operate properly (which is not theoretically, but practically true, because > of crash we have). Well, gee, it sure sounds like KVM is violating Intel's specification. We should not fix this.
Re: Kernel panic on current amd64 running under Ubuntu KVM (patch included)
On Tue, May 21, 2013 at 12:21 AM, Theo de Raadt wrote: >> On Mon, May 20, 2013 at 11:58 PM, Ted Unangst wrote: >> > On Mon, May 20, 2013 at 21:28, Alexey Suslikov wrote: >> >> >> >> While I see practical use, someone don't. I call this disagreement. There >> >> is >> >> no problem for me if somebody disagree with a plan I have. It's normal. >> >> >> >> Btw, Intel's doc I have found at >> >> http://www.intel.com/content/dam/www/public/us/en/documents/application-notes/processor-identification-cpuid-instruction-note.pdf >> >> >> >> >> >> says "31 Not Used Always returns 0". >> > >> > In that case, there's no sense testing for it, because it's always 0. >> > >> > If it isn't 0, then it's not an amd64 computer and we don't support >> > it. Trying to identify all the infinite machines we don't support is >> > fruitless, imo, and perhaps not a path we should start down, because >> > then people will expect us to detect why we don't run on *their* not >> > supported computer. >> >> In practice, it is not zero. This is why my point was opposite: >> >> * if I see Hypervisor flag in dmesg, my (virtual) hardware is not guaranteed >> to operate properly (which is not theoretically, but practically true, >> because >> of crash we have). > > Well, gee, it sure sounds like KVM is violating Intel's specification. > > We should not fix this. $ grep -R "bugs@" /usr/src/sys/* /usr/src/sys/dev/usb/ubsa.c:"Please send your dmesg to , thanks.\n", /usr/src/sys/dev/usb/umsm.c:"Please send your dmesg to , thanks.\n", Sometimes we warn users, sometimes not (but we aware of the problem). What is the criteria?
Re: Kernel panic on current amd64 running under Ubuntu KVM (patch included)
On Tue, May 21, 2013 at 00:32, Alexey Suslikov wrote: > Sometimes we warn users, sometimes not (but we aware of the problem). > > What is the criteria? Is it a device we claim to support? (Or are interested in supporting?)