Re: VM crash on 7.2#4

2023-01-01 Thread Dave Voutila 


Mischa  writes:

> Hi,
>
> Just noticed one of the VMs greeted me with a ddb> prompt.
> The host is running 7.2#4 as well as the VM, dmesg of the host below.
>
> I managed to get the following data from the VM:
>
> ddb> show panic
> *cpu0: kernel diagnostic assertion "m != NULL" failed: file
>  "/usr/src/sys/dev/p
> v/if_vio.c", line 1006
> ddb> trace
> db_enter() at db_enter+0x10
> panic(81f17485) at panic+0xb8
> __assert(81f891d8,81f89d08,3ee,81f90540) at
> __assert+0x
> 25
> vio_rxeof(8003a000) at vio_rxeof+0x23f
> vio_rx_intr(8003a050) at vio_rx_intr+0x38
> virtio_check_vqs(80039400) at virtio_check_vqs+0xfe
> virtio_pci_legacy_intr(80039400) at virtio_pci_legacy_intr+0x61
> intr_handler(80002250c100,80049e80) at intr_handler+0x38
> Xintr_legacy5_untramp() at Xintr_legacy5_untramp+0x1a3
> cpu_idle_cycle() at cpu_idle_cycle+0x1f
> end trace frame: 0x0, count: -10

Since the host is running 7.2, I wouldn't be surprised if this is
related to the previous approach vmd(8) used for updating virtqueues,
i.e. copy from guest -> mutate -> overwrite in the guest. We also didn't
have memory barriers/compiler hints between virtqueue update and
updating the used index.

Given this is a rx interrupt handler, and vmd uses the "device" thread
for pulling packets off the tap(4) and writing them into the virtqueue,
I wouldn't be too surprised if the approach used in 7.2 and earlier
created the conditions that caused this panic. It's one of the only real
"async" portions of vmd device emulation at the moment.

Any idea if this can be reproduced?

>
> root@r2:~ # syspatch -l
> 001_x509
> 002_asn1
> 003_ukbd
> 004_expat
> 005_pixman
> 006_vmm
> 007_unwind
> 008_pfsync
> 009_xserver
> 010_vmd
> 011_gpuinv
> 012_acme
>
> root@r2:~ # dmesg
> OpenBSD 7.2 (GENERIC.MP) #4: Mon Dec 12 06:06:42 MST 2022
> 
> r...@syspatch-72-amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> real mem = 412202078208 (393106MB)
> avail mem = 399692173312 (381176MB)
> random: good seed from bootblocks
> mpath0 at root
> scsibus0 at mpath0: 256 targets
> mainbus0 at root
> bios0 at mainbus0: SMBIOS rev. 2.8 @ 0x7a32f000 (76 entries)
> bios0: vendor Dell Inc. version "2.16.0" date 07/20/2022
> bios0: Dell Inc. PowerEdge R630
> acpi0 at bios0: ACPI 4.0
> acpi0: sleep states S0 S5
> acpi0: tables DSDT FACP MCEJ WD__ SLIC HPET APIC MCFG MSCT SLIT SRAT
> SSDT SSDT SSDT PRAD DMAR HEST BERT ERST EINJ
> acpi0: wakeup devices PCI0(S4) BR1A(S4) BR1B(S4) BR2A(S4) BR2B(S4)
> BR2C(S4) BR2D(S4) BR3A(S4) BR3B(S4) BR3C(S4) BR3D(S4) XHC_(S0)
> RP02(S4) RP03(S4) RP05(S4) RP08(S4) [...]
> acpitimer0 at acpi0: 3579545 Hz, 24 bits
> acpihpet0 at acpi0: 14318179 Hz
> acpimadt0 at acpi0 addr 0xfee0: PC-AT compat
> cpu0 at mainbus0: apid 0 (boot processor)
> cpu0: Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz, 3200.03 MHz, 06-3f-02
> cpu0:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,PQM,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB
> 64b/line 8-way L2 cache, 20MB 64b/line 20-way L3 cache
> cpu0: smt 0, core 0, package 0
> mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
> cpu0: apic clock running at 99MHz
> cpu0: mwait min=64, max=64, C-substates=0.2.1.2, IBE
> cpu1 at mainbus0: apid 16 (application processor)
> cpu1: Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz, 3398.59 MHz, 06-3f-02
> cpu1:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,PQM,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> cpu1: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB
> 64b/line 8-way L2 cache, 20MB 64b/line 20-way L3 cache
> cpu1: smt 0, core 0, package 1
> cpu2 at mainbus0: apid 2 (application processor)
> cpu2: Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz, 3399.01 MHz, 06-3f-02
> cpu2:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,PQM,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> cpu2: 3

VM crash on 7.2#4

2023-01-01 Thread Mischa 

Hi,

Just noticed one of the VMs greeted me with a ddb> prompt.
The host is running 7.2#4 as well as the VM, dmesg of the host below.

I managed to get the following data from the VM:

ddb> show panic
*cpu0: kernel diagnostic assertion "m != NULL" failed: file 
"/usr/src/sys/dev/p

v/if_vio.c", line 1006
ddb> trace
db_enter() at db_enter+0x10
panic(81f17485) at panic+0xb8
__assert(81f891d8,81f89d08,3ee,81f90540) at 
__assert+0x

25
vio_rxeof(8003a000) at vio_rxeof+0x23f
vio_rx_intr(8003a050) at vio_rx_intr+0x38
virtio_check_vqs(80039400) at virtio_check_vqs+0xfe
virtio_pci_legacy_intr(80039400) at virtio_pci_legacy_intr+0x61
intr_handler(80002250c100,80049e80) at intr_handler+0x38
Xintr_legacy5_untramp() at Xintr_legacy5_untramp+0x1a3
cpu_idle_cycle() at cpu_idle_cycle+0x1f
end trace frame: 0x0, count: -10

root@r2:~ # syspatch -l
001_x509
002_asn1
003_ukbd
004_expat
005_pixman
006_vmm
007_unwind
008_pfsync
009_xserver
010_vmd
011_gpuinv
012_acme

root@r2:~ # dmesg
OpenBSD 7.2 (GENERIC.MP) #4: Mon Dec 12 06:06:42 MST 2022

r...@syspatch-72-amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP

real mem = 412202078208 (393106MB)
avail mem = 399692173312 (381176MB)
random: good seed from bootblocks
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.8 @ 0x7a32f000 (76 entries)
bios0: vendor Dell Inc. version "2.16.0" date 07/20/2022
bios0: Dell Inc. PowerEdge R630
acpi0 at bios0: ACPI 4.0
acpi0: sleep states S0 S5
acpi0: tables DSDT FACP MCEJ WD__ SLIC HPET APIC MCFG MSCT SLIT SRAT 
SSDT SSDT SSDT PRAD DMAR HEST BERT ERST EINJ
acpi0: wakeup devices PCI0(S4) BR1A(S4) BR1B(S4) BR2A(S4) BR2B(S4) 
BR2C(S4) BR2D(S4) BR3A(S4) BR3B(S4) BR3C(S4) BR3D(S4) XHC_(S0) RP02(S4) 
RP03(S4) RP05(S4) RP08(S4) [...]

acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpihpet0 at acpi0: 14318179 Hz
acpimadt0 at acpi0 addr 0xfee0: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz, 3200.03 MHz, 06-3f-02
cpu0: 
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,PQM,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB 
64b/line 8-way L2 cache, 20MB 64b/line 20-way L3 cache

cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
cpu0: apic clock running at 99MHz
cpu0: mwait min=64, max=64, C-substates=0.2.1.2, IBE
cpu1 at mainbus0: apid 16 (application processor)
cpu1: Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz, 3398.59 MHz, 06-3f-02
cpu1: 
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,PQM,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu1: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB 
64b/line 8-way L2 cache, 20MB 64b/line 20-way L3 cache

cpu1: smt 0, core 0, package 1
cpu2 at mainbus0: apid 2 (application processor)
cpu2: Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz, 3399.01 MHz, 06-3f-02
cpu2: 
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,PQM,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu2: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB 
64b/line 8-way L2 cache, 20MB 64b/line 20-way L3 cache

cpu2: smt 0, core 1, package 0
cpu3 at mainbus0: apid 18 (application processor)
cpu3: Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz, 3400.00 MHz, 06-3f-02
cpu3: 
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,PQM,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu3: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 256KB 
64b/line 8-way L2 cache, 20MB 64b/line 20-way L3 cache