I am in the process of building a new OpenBSD i386 5.0-release Intel Atom D510-based fw/router. I was editing some config files on the box in emacs when the process threw a core dump. Thinking perhaps it was just emacs, I went to do something else, 'sudo pkg_add -v mutt', and received a coredump again.
I went looking for stress testing apps, thinking I might have a bad CPU or RAM module and came upon 'stress'. After several iterations of stress seeming to cause kernel panics, and then upgrading to a 5.0 snapshot from November 13, 2011[1], I was still seeing panics. I provide the below detail to help those more knowledgeable in debugging. Thanks in advance, Jeff [1] http://openbsd.mirrors.tds.net/pub/OpenBSD/snapshots/i386/ Full stress command line: # stress --cpu 8 --io 4 --vm 2 -m 5 --vm-bytes 128M --timeout 30s -v panic: rw_enter: vmmaplk locking against myself Stopped at Debugger+0x4: popl %ebp RUN AT LEAST 'trace' AND 'ps' AND INCLUDE OUTPUT WHEN REPORTING THIS PANIC! IF RUNNING SMP, USE 'mach ddbcpu <#>' AND 'trace' ON OTHER PROCESSORS, TOO. DO NOT EVEN BOTHER REPORTING THIS WITHOUT INCLUDING THAT INFORMATION! ddb{2}> trace Debugger(d08d8366,dea05ba4,d08b5738,dea05ba4,d0521ecb) at Debugger+0x4 panic(d08b5738,d08cf0ac,0,dea05c3c,d0522be6) at panic+0x5d rw_enter(d8d4eb48,1,d0d10c20,1,dea05c1c) at rw_enter+0x231 rw_enter_read(d8d4eb48,d0d10bdc,1000001,dea05c74,ffffffff) at rw_enter_read+0x2 1 uvmfault_lookup(dea05d44,0,5f4aa000,3,1) at uvmfault_lookup+0x98 uvm_fault(d8d4eb44,400000,0,1,d2725e3c) at uvm_fault+0x59 trap() at trap+0x468 --- trap (number 0) --- Bad frame pointer: 0xd8acf754 0: ddb{2}> ps PID PPID PGRP UID S FLAGS WAIT COMMAND 32600 22207 22207 0 2 0 stress 9130 22207 22207 0 2 0 stress 12362 22207 22207 0 2 0 stress 14411 22207 22207 0 2 0 stress 9246 22207 22207 0 2 0 stress * 8781 22207 22207 0 7 0 stress 28867 22207 22207 0 7 0 stress 22593 22207 22207 0 2 0 stress 9847 22207 22207 0 7 0 stress 17346 22207 22207 0 2 0 stress 2098 22207 22207 0 2 0 stress 27969 22207 22207 0 2 0 stress 13618 22207 22207 0 2 0 stress 13017 22207 22207 0 2 0 stress 23837 22207 22207 0 2 0 stress 24834 22207 22207 0 2 0 stress 22390 22207 22207 0 7 0 stress 22207 21299 22207 0 3 0x80 wait stress 21299 12082 21299 0 3 0x88 pause ksh 18262 28817 28817 95 3 0x80 kqread smtpd 11710 28817 28817 95 3 0x80 kqread smtpd 2941 28817 28817 95 3 0x80 kqread smtpd 3949 28817 28817 95 3 0x80 kqread smtpd 972 28817 28817 95 3 0x80 kqread smtpd 4951 28817 28817 95 3 0x80 kqread smtpd 19759 28817 28817 95 3 0x80 kqread smtpd 28438 28817 28817 95 3 0x80 kqread smtpd 28817 1 28817 0 3 0x80 kqread smtpd 12082 22273 12082 1000 3 0x88 pause ksh 22273 32054 32054 1000 3 0x80 select sshd 32054 22661 32054 0 3 0x80 poll sshd 8577 1 8577 0 3 0x80 ttyin getty 31323 1 31323 0 3 0x80 ttyin getty 27262 1 27262 0 3 0x80 ttyin getty 870 1 870 0 3 0x80 ttyin getty 24785 1 24785 0 3 0x80 ttyin getty 20842 1 20842 0 3 0x80 ttyin getty 23798 1 23798 0 3 0x80 select cron 12148 1 12148 0 3 0x80 select inetd 22661 1 22661 0 3 0x80 select sshd 24417 27000 13320 83 3 0x80 poll ntpd 27000 13320 13320 83 3 0x80 poll ntpd 13320 1 13320 0 3 0x80 poll ntpd 14780 19383 19383 74 3 0x80 bpf pflogd 19383 1 19383 0 3 0x80 netio pflogd 14790 23994 23994 73 3 0x80 poll syslogd 23994 1 23994 0 3 0x80 netio syslogd 18 0 0 0 3 0x100200 aiodoned aiodoned 17 0 0 0 3 0x100200 syncer update 16 0 0 0 3 0x100200 cleaner cleaner 15 0 0 0 3 0x100200 reaper reaper 14 0 0 0 3 0x100200 pgdaemon pagedaemon 13 0 0 0 3 0x100200 bored crypto 12 0 0 0 3 0x100200 pftm pfpurge 11 0 0 0 3 0x100200 usbtsk usbtask 10 0 0 0 3 0x100200 usbatsk usbatsk 9 0 0 0 3 0x100200 bored intelrel 8 0 0 0 3 0x100200 acpi0 acpi0 7 0 0 0 3 0x40100200 idle3 6 0 0 0 3 0x40100200 idle2 5 0 0 0 3 0x40100200 idle1 4 0 0 0 3 0x100200 bored syswq 3 0 0 0 3 0x40100200 idle0 2 0 0 0 3 0x100200 kmalloc kmthread 1 0 1 0 3 0x80 wait init 0 -1 0 0 3 0x200 scheduler swapper ddb{2}> mach ddbcpu 0 Stopped at Debugger+0x4: popl %ebp RUN AT LEAST 'trace' AND 'ps' AND INCLUDE OUTPUT WHEN REPORTING THIS PANIC! IF RUNNING SMP, USE 'mach ddbcpu <#>' AND 'trace' ON OTHER PROCESSORS, TOO. DO NOT EVEN BOTHER REPORTING THIS WITHOUT INCLUDING THAT INFORMATION! ddb{0}> trace Debugger(d0ae1aa0,0,d09bda40,292,d0202236) at Debugger+0x4 i386_ipi_handler(30,de9e0020,d0200000,10,de9e0010) at i386_ipi_handler+0x5f Xintripi() at Xintripi+0x49 --- interrupt --- __mp_lock(d0a26844,d0202613,de9e4fa0,d020230c,5b) at __mp_lock+0x52 i386_softintlock(5b,63,cfbc0033,33,0) at i386_softintlock+0x12 Bad frame pointer: 0xde9e4fa0 ddb{0}> mach ddbcpu 1 Stopped at Debugger+0x4: popl %ebp RUN AT LEAST 'trace' AND 'ps' AND INCLUDE OUTPUT WHEN REPORTING THIS PANIC! IF RUNNING SMP, USE 'mach ddbcpu <#>' AND 'trace' ON OTHER PROCESSORS, TOO. DO NOT EVEN BOTHER REPORTING THIS WITHOUT INCLUDING THAT INFORMATION! ddb{1}> trace Debugger(d2e29000,dea02ea8,d0775161,246,0) at Debugger+0x4 i386_ipi_handler(0,d2fa0020,0,d0520010,d0a40010) at i386_ipi_handler+0x5f Xintripi() at Xintripi+0x49 --- interrupt --- __mp_lock(d0a26844,286,dea02f1c,d03d7132,d0a26844) at __mp_lock+0x50 _kernel_lock(d8cef574,dea02f64,dea02f84,d03d7132,2) at _kernel_lock+0x22 syscall() at syscall+0x115 --- syscall (number 24000) --- 0x2: ddb{1}> mach ddbcpu 2 Stopped at Debugger+0x4: popl %ebp RUN AT LEAST 'trace' AND 'ps' AND INCLUDE OUTPUT WHEN REPORTING THIS PANIC! IF RUNNING SMP, USE 'mach ddbcpu <#>' AND 'trace' ON OTHER PROCESSORS, TOO. DO NOT EVEN BOTHER REPORTING THIS WITHOUT INCLUDING THAT INFORMATION! ddb{2}> trace Debugger(d08d8366,dea05ba4,d08b5738,dea05ba4,d0521ecb) at Debugger+0x4 panic(d08b5738,d08cf0ac,0,dea05c3c,d0522be6) at panic+0x5d rw_enter(d8d4eb48,1,d0d10c20,1,dea05c1c) at rw_enter+0x231 rw_enter_read(d8d4eb48,d0d10bdc,1000001,dea05c74,ffffffff) at rw_enter_read+0x2 1 uvmfault_lookup(dea05d44,0,5f4aa000,3,1) at uvmfault_lookup+0x98 uvm_fault(d8d4eb44,400000,0,1,d2725e3c) at uvm_fault+0x59 trap() at trap+0x468 --- trap (number 0) --- Bad frame pointer: 0xd8acf754 0: ddb{2}> mach ddbcpu 3 Stopped at Debugger+0x4: popl %ebp RUN AT LEAST 'trace' AND 'ps' AND INCLUDE OUTPUT WHEN REPORTING THIS PANIC! IF RUNNING SMP, USE 'mach ddbcpu <#>' AND 'trace' ON OTHER PROCESSORS, TOO. DO NOT EVEN BOTHER REPORTING THIS WITHOUT INCLUDING THAT INFORMATION! ddb{3}> trace Debugger(d2f48800,de9fcec8,d0775161,246,d8cef2c4) at Debugger+0x4 i386_ipi_handler(0,20,0,10,10) at i386_ipi_handler+0x5f Xintripi() at Xintripi+0x49 --- interrupt --- __mp_lock(d0a26844,282,de9fcf3c,d03d7132,d0a26844) at __mp_lock+0x52 _kernel_lock(d8cef2c4,8d93e000,0,3,de9fcf94) at _kernel_lock+0x22 trap() at trap+0x3e4 --- trap (number 73048064) --- 0x6: dmesg porn: OpenBSD 5.0-current (GENERIC.MP) #114: Wed Nov 2 22:36:15 MDT 2011 dera...@i386.openbsd.org:/usr/src/sys/arch/i386/compile/GENERIC.MP cpu0: Intel(R) Atom(TM) CPU D510 @ 1.66GHz ("GenuineIntel" 686-class) 1.67 GHz cpu0: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,NXE,LONG,SSE3,MWAIT,DS-CPL,TM2,SSSE3,CX16,xTPR,PDCM,MOVBE,LAHF real mem = 2137169920 (2038MB) avail mem = 2092105728 (1995MB) mainbus0 at root bios0 at mainbus0: AT/286+ BIOS, date 11/23/10, BIOS32 rev. 0 @ 0xf0010, SMBIOS rev. 2.6 @ 0xfc120 (24 entries) bios0: vendor American Megatrends Inc. version "080015" date 11/23/2010 acpi0 at bios0: rev 0 acpi0: sleep states S0 S1 S4 S5 acpi0: tables DSDT FACP APIC MCFG OEMB HPET GSCI acpi0: wakeup devices P0P1(S4) PS2K(S4) PS2M(S4) USB0(S4) USB1(S4) USB2(S4) USB3(S4) EUSB(S4) P0P4(S4) P0P5(S4) P0P6(S4) P0P7(S4) P0P8(S4) P0P9(S4) HDAC(S4) USB4(S4) USB5(S4) USBE(S4) GBEC(S4) acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: apic clock running at 166MHz cpu1 at mainbus0: apid 2 (application processor) cpu1: Intel(R) Atom(TM) CPU D510 @ 1.66GHz ("GenuineIntel" 686-class) 1.67 GHz cpu1: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,NXE,LONG,SSE3,MWAIT,DS-CPL,TM2,SSSE3,CX16,xTPR,PDCM,MOVBE,LAHF cpu2 at mainbus0: apid 1 (application processor) cpu2: Intel(R) Atom(TM) CPU D510 @ 1.66GHz ("GenuineIntel" 686-class) 1.67 GHz cpu2: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,NXE,LONG,SSE3,MWAIT,DS-CPL,TM2,SSSE3,CX16,xTPR,PDCM,MOVBE,LAHF cpu3 at mainbus0: apid 3 (application processor) cpu3: Intel(R) Atom(TM) CPU D510 @ 1.66GHz ("GenuineIntel" 686-class) 1.67 GHz cpu3: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,NXE,LONG,SSE3,MWAIT,DS-CPL,TM2,SSSE3,CX16,xTPR,PDCM,MOVBE,LAHF ioapic0 at mainbus0: apid 4 pa 0xfec00000, version 20, 24 pins ioapic0: misconfigured as apic 1, remapped to apid 4 acpimcfg0 at acpi0 addr 0xe0000000, bus 0-255 acpihpet0 at acpi0: 14318179 Hz acpiprt0 at acpi0: bus 0 (PCI0) acpiprt1 at acpi0: bus 7 (P0P1) acpiprt2 at acpi0: bus 1 (P0P4) acpiprt3 at acpi0: bus 2 (P0P5) acpiprt4 at acpi0: bus 3 (P0P6) acpiprt5 at acpi0: bus 4 (P0P7) acpiprt6 at acpi0: bus 5 (P0P8) acpiprt7 at acpi0: bus 6 (P0P9) acpicpu0 at acpi0 acpicpu1 at acpi0 acpicpu2 at acpi0 acpicpu3 at acpi0 acpibtn0 at acpi0: PWRB bios0: ROM list: 0xc0000/0xda00! 0xce000/0x1000 0xcf000/0x1000 pci0 at mainbus0 bus 0: configuration mode 1 (bios) pchb0 at pci0 dev 0 function 0 "Intel Pineview DMI" rev 0x02 vga1 at pci0 dev 2 function 0 "Intel Pineview Video" rev 0x02 wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) wsdisplay0: screen 1-5 added (80x25, vt100 emulation) intagp0 at vga1 agp0 at intagp0: aperture at 0xd0000000, size 0x10000000 inteldrm0 at vga1: apic 4 int 16 drm0 at inteldrm0 "Intel Pineview Video" rev 0x02 at pci0 dev 2 function 1 not configured ppb0 at pci0 dev 28 function 0 "Intel 82801H PCIE" rev 0x03: apic 4 int 22 pci1 at ppb0 bus 1 em0 at pci1 dev 0 function 0 "Intel PRO/1000 MT (82574L)" rev 0x00: msi, address 00:90:0b:1f:95:ba ppb1 at pci0 dev 28 function 1 "Intel 82801H PCIE" rev 0x03: apic 4 int 23 pci2 at ppb1 bus 2 em1 at pci2 dev 0 function 0 "Intel PRO/1000 (82583V)" rev 0x00: msi, address 00:90:0b:1f:95:bb ppb2 at pci0 dev 28 function 2 "Intel 82801H PCIE" rev 0x03: apic 4 int 20 pci3 at ppb2 bus 3 em2 at pci3 dev 0 function 0 "Intel PRO/1000 (82583V)" rev 0x00: msi, address 00:90:0b:1f:95:bc ppb3 at pci0 dev 28 function 3 "Intel 82801H PCIE" rev 0x03: apic 4 int 21 pci4 at ppb3 bus 4 em3 at pci4 dev 0 function 0 "Intel PRO/1000 (82583V)" rev 0x00: msi, address 00:90:0b:1f:95:bd ppb4 at pci0 dev 28 function 4 "Intel 82801H PCIE" rev 0x03: apic 4 int 22 pci5 at ppb4 bus 5 em4 at pci5 dev 0 function 0 "Intel PRO/1000 (82583V)" rev 0x00: msi, address 00:90:0b:1f:95:be ppb5 at pci0 dev 28 function 5 "Intel 82801H PCIE" rev 0x03: apic 4 int 23 pci6 at ppb5 bus 6 em5 at pci6 dev 0 function 0 "Intel PRO/1000 (82583V)" rev 0x00: msi, address 00:90:0b:1f:95:bf uhci0 at pci0 dev 29 function 0 "Intel 82801H USB" rev 0x03: apic 4 int 23 ehci0 at pci0 dev 29 function 7 "Intel 82801H USB" rev 0x03: apic 4 int 23 usb0 at ehci0: USB revision 2.0 uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1 ppb6 at pci0 dev 30 function 0 "Intel 82801BAM Hub-to-PCI" rev 0xf3 pci7 at ppb6 bus 7 ichpcib0 at pci0 dev 31 function 0 "Intel 82801HBM LPC" rev 0x03: PM disabled pciide0 at pci0 dev 31 function 1 "Intel 82801HBM IDE" rev 0x03: DMA, channel 0 configured to compatibility, channel 1 configured to compatibility wd0 at pciide0 channel 0 drive 0: <SanDisk SDCFH-002G> wd0: 1-sector PIO, LBA, 1918MB, 3928176 sectors wd0(pciide0:0:0): using PIO mode 4, DMA mode 2 pciide0: channel 1 ignored (disabled) ahci0 at pci0 dev 31 function 2 "Intel 82801HBM AHCI" rev 0x03: msi, AHCI 1.1 scsibus0 at ahci0: 32 targets sd0 at scsibus0 targ 0 lun 0: <ATA, WDC WD3200BPVT-0, 01.0> SCSI3 0/direct fixed naa.50014ee601895a40 sd0: 305245MB, 512 bytes/sector, 625142448 sectors ichiic0 at pci0 dev 31 function 3 "Intel 82801H SMBus" rev 0x03: apic 4 int 17 iic0 at ichiic0 spdmem0 at iic0 addr 0x50: 2GB DDR2 SDRAM non-parity PC2-6400CL5 SO-DIMM usb1 at uhci0: USB revision 1.0 uhub1 at usb1 "Intel UHCI root hub" rev 1.00/1.00 addr 1 isa0 at ichpcib0 isadma0 at isa0 com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo com0: console pckbc0 at isa0 port 0x60/5 pckbd0 at pckbc0 (kbd slot) pckbc0: using irq 1 for kbd slot wskbd0 at pckbd0: console keyboard, using wsdisplay0 pcppi0 at isa0 port 0x61 spkr0 at pcppi0 wbsio0 at isa0 port 0x2e/2: W83627THF rev 0x85 lm1 at wbsio0 port 0xa00/8: W83627THF npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16 mtrr: Pentium Pro MTRR support vscsi0 at root scsibus1 at vscsi0: 256 targets softraid0 at root scsibus2 at softraid0: 256 targets root on sd0a (b501426b8bd6beca.a) swap on sd0b dump on sd0b WARNING: / was not properly unmounted
