Hey folks, I've had a helluva week - my colocated server has crashed at least four times, and I'd like a little sanity check from people that know a lot more than I do. Sorry for the length of this, trying to include all the data I'm aware of that might be relevant and helpful.
For the two crashes that I've been able to capture some output from (one from an IP KVM, one from /var/log/messages after setting ddb.panic=0), I've seen: uvm_fault(0xffffffff81cf2b20, 0xffff800000cef000, 0, 2) -> e kernel: page fault trap, code=0 Stopped at memmove+0x16: repe movsq (%rsi),%es:(%rdi) and reboot after panic: trap type 8, code=0, pc=ffffffff81292dff Because kernel panics are so rare in OpenBSD, I don't have much experience debugging them. Following crash(8), I fired up gdb and took a look at this morning's crash and auto-reboot: gdb GNU gdb 6.3 Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-unknown-openbsd5.4". (gdb) file /var/crash/bsd.0 Reading symbols from /var/crash/bsd.0...(no debugging symbols found)...done. (gdb) target kvm /var/crash/bsd.0.core #0 0xffffffff8130a194 in dumpsys () (gdb) where #0 0xffffffff8130a194 in dumpsys () #1 0xffffffff8130a2e5 in boot () #2 0xffffffff811a2d76 in panic () #3 0xffffffff81313d51 in trap () #4 0xffffffff81315766 in alltraps () #5 0x0000000000000000 in ?? () I don't *think* it was resource starvation: vmstat -N /var/crash/bsd.0 -M /var/crash/bsd.0.core -m Memory statistics by bucket size Size In Use Free Requests HighWater Couldfree 16 46085 47867 10903348 1280 2417 32 3653 571 1604650 640 0 64 4215 1289 2687011 320 18492 128 5405 1411 925024 160 930 256 2066 286 629177 80 74 512 1774 338 462020 40 9397 1024 1539 685 578108 20 141600 2048 287 45 78486 10 21570 4096 83 528 144485 5 101528 8192 20 8 18105 5 7483 16384 1 0 366 5 0 32768 8 0 102 5 0 65536 2 0 1909341 5 0 524288 2 0 2 5 0 Memory usage type by bucket size Size Type(s) 16 devbuf, pcb, routetbl, sem, dirhash, ACPI, exec, UVM amap, UVM aobj, USB, USB device, temp 32 devbuf, pcb, routetbl, ifaddr, sysctl, vnodes, sem, dirhash, ACPI, in_multi, exec, UVM amap, USB, temp 64 devbuf, routetbl, ifaddr, vnodes, UFS mount, dirhash, ACPI, proc, VFS cluster, in_multi, ether_multi, VM swap, UVM amap, USB, USB device, NDP, temp 128 devbuf, pcb, routetbl, sysctl, UFS mount, sem, dirhash, ACPI, NFS srvsock, ttys, pfkey data, inodedep, VM swap, UVM amap, USB, USB device, USB HC, NDP, temp 256 devbuf, routetbl, ifaddr, ioctlops, vnodes, UFS mount, shm, VM map, sem, dirhash, ACPI, exec, xform_data, UVM amap, USB, USB device, temp 512 devbuf, routetbl, ifaddr, ioctlops, sem, dirhash, ACPI, file desc, NFS daemon, ttys, xform_data, newblk, UVM amap, USB, temp 1024 devbuf, pcb, sysctl, ioctlops, mount, UFS mount, shm, dirhash, ACPI, file desc, proc, ttys, exec, UVM amap, crypto data, temp 2048 devbuf, ioctlops, UFS mount, sem, dirhash, ACPI, file desc, VM swap, UVM amap, UVM aobj, temp 4096 devbuf, ifaddr, ioctlops, UFS mount, shm, dirhash, file desc, proc, UVM amap, memdesc, temp 8192 devbuf, file, ttys, pagedep, UVM amap, USB, temp 16384 devbuf, MSDOSFS mount, indirdep, temp 32768 devbuf, UFS quota, UFS mount, ISOFS mount, inodedep, indirdep, NTFS hash 65536 devbuf, temp 524288 VM swap Memory statistics by type Type Kern Type InUse MemUse HighUse Limit Requests Limit Limit Size(s) devbuf 733 495K 2597K 78644K 23287 0 0 16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536 pcb 218 34K 42K 78644K 40723 0 0 16,32,128,1024 routetbl 78 9K 10K 78644K 4198 0 0 16,32,64,128,256,512 ifaddr 56 16K 16K 78644K 58 0 0 32,64,256,512,4096 sysctl 3 2K 2K 78644K 3 0 0 32,128,1024 ioctlops 0 0K 4K 78644K 4632 0 0 256,512,1024,2048,4096 mount 13 13K 13K 78644K 13 0 0 1024 vnodes 37 3K 72K 78644K 3344 0 0 32,64,256 UFS quota 1 32K 32K 78644K 1 0 0 32768 UFS mount 53 111K 111K 78644K 53 0 0 64,128,256,1024,2048,4096,32768 shm 20 74K 86K 78644K 19131 0 0 256,1024,4096 VM map 2 1K 1K 78644K 2 0 0 256 sem 14 5K 5K 78644K 16 0 0 16,32,128,256,512,2048 dirhash 1612 503K 670K 78644K 49359 0 0 16,32,64,128,256,512,1024,2048,4096 ACPI 2449 292K 305K 78644K 8406 0 0 16,32,64,128,256,512,1024,2048 file 0 0K 8K 78644K 87 0 0 8192 file desc 63 73K 112K 78644K 10748 0 0 512,1024,2048,4096 proc 22 11K 11K 78644K 22 0 0 64,1024,4096 VFS cluster 0 0K 1K 78644K 91814 0 0 64 NFS srvsock 1 1K 1K 78644K 1 0 0 128 NFS daemon 1 1K 1K 78644K 1 0 0 512 in_multi 24 2K 2K 78644K 24 0 0 32,64 ether_multi 4 1K 1K 78644K 4 0 0 64 ISOFS mount 1 32K 32K 78644K 1 0 0 32768 MSDOSFS mount 1 16K 16K 78644K 1 0 0 16384 ttys 420 308K 308K 78644K 420 0 0 128,512,1024,8192 exec 0 0K 4K 78644K 79942 0 0 16,32,256,1024 pfkey data 1 1K 1K 78644K 2 0 0 128 xform_data 0 0K 4K 78644K 744 0 0 256,512 pagedep 1 8K 8K 78644K 1 0 0 8192 inodedep 1 32K 121K 78644K 2779 0 0 128,32768 newblk 1 1K 1K 78644K 1 0 0 512 indirdep 0 0K 64K 78644K 399 0 0 16384,32768 VM swap 7 723K 723K 78644K 7 0 0 64,128,2048,524288 UVM amap 59056 3913K 4819K 78644K 15666014 0 0 16,32,64,128,256,512,1024,2048,4096,8192 UVM aobj 3 5K 5K 78644K 3 0 0 16,2048 USB 135 40K 40K 78644K 136 0 0 16,32,64,128,256,512,8192 USB device 31 3K 3K 78644K 31 0 0 16,64,128,256 USB HC 1 1K 1K 78644K 1 0 0 128 memdesc 1 4K 4K 78644K 1 0 0 4096 crypto data 1 1K 1K 78644K 1 0 0 1024 NDP 7 1K 1K 78644K 8 0 0 64,128 temp 67 58K 250K 78644K 3933805 0 0 16,32,64,128,256,512,1024,2048,4096,8192,16384,65536 NTFS hash 1 32K 32K 78644K 1 0 0 32768 Memory Totals: In Use Free Requests 6842K 4215K 19940225 Memory resource pool statistics Name Size Requests Fail InUse Pgreq Pgrel Npage Hiwat Minpg Maxpg Idle inpcbpl 360 40934 0 49 9 0 9 9 0 8 3 plimitpl 152 5302 0 47 3 0 3 3 0 8 1 synpl 248 20216 0 0 2 0 2 2 0 8 2 sackhlpl 24 5337 0 0 1 0 1 1 0 8 1 tcpqepl 32 18281 0 0 1 0 1 1 0 8 1 tcpcbpl 560 27685 0 42 13 0 13 13 0 8 5 rttmrpl 72 314 0 0 1 0 1 1 0 8 1 rtentpl 200 247 0 34 3 0 3 3 0 8 1 strpolpl 48 372 0 0 1 0 1 1 0 8 1 strprocpl 2456 744 0 0 26 18 8 10 0 8 8 pfosfp 40 840 0 420 5 0 5 5 0 8 0 pfosfpen 112 1420 0 710 34 13 21 21 0 8 0 pfrke_plain 160 48 0 48 2 0 2 2 0 8 0 pfrktable 1344 4 0 2 2 0 2 2 0 8 1 pfstateitempl 24 26514 0 52 2 0 2 2 0 8 1 pfstatekeypl 104 26514 0 52 7 0 7 7 0 8 5 pfstatepl 312 26514 0 52 21 11 10 21 0 8 6 pfsnitempl 16 445 0 2 1 0 1 1 0 8 0 pfsrctrpl 152 25 0 1 1 0 1 1 0 8 0 pfrulepl 1328 36 0 25 12 0 12 12 0 8 3 semupl 112 7095 0 2 1 0 1 1 0 8 0 semapl 112 12 0 12 1 0 1 1 0 8 0 shmpl 112 1 0 1 1 0 1 1 0 8 0 dirhash 1024 399977 0 1691 44652 44227 425 497 0 8 2 newdirblkpl 32 3633 0 0 3 0 3 3 0 8 3 dirrempl 64 196471 0 4 2432 2424 8 345 0 8 7 mkdirpl 56 11676 0 2 74 66 8 39 0 8 7 diraddpl 56 216316 0 5 1808 1800 8 388 0 8 7 freefilepl 48 14733 0 3 23 15 8 16 0 8 7 freeblkspl 192 15081 0 4 250 242 8 80 0 8 7 freefragpl 64 3195 0 1 1 0 1 1 0 8 0 allocindirpl 104 116476 0 0 2758 2750 8 346 0 8 8 indirdeppl 56 630 0 0 1 0 1 1 0 8 1 allocdirectpl 128 30713 0 5 139 131 8 78 0 8 7 bmsafemappl 64 5468 0 3 1 0 1 1 0 8 0 newblkpl 64 147189 0 0 1 0 1 1 0 8 1 inodedeppl 152 396484 0 9 9916 9908 8 1313 0 8 7 pagedeppl 128 13863 0 5 125 117 8 55 0 8 7 dino1pl 128 10011177 0 40593 1310 0 1310 1310 0 8 0 ffsino 216 10011177 0 40593 2260 4 2256 2256 0 8 0 nchpl 144 1421690 0 5920 220 0 220 220 0 8 0 vnodes 264 40610 0 40610 2708 0 2708 2708 0 8 0 namei 1024 16704896 0 0 2 0 2 2 0 8 2 scxspl 192 4523636 0 2 2 0 2 2 0 8 1 ifaddritempl 64 19 0 19 1 0 1 1 0 8 0 wqtasks 40 38688 0 11 1 0 1 1 0 8 0 sigapl 432 81820 0 108 16 0 16 16 0 8 3 knotepl 112 59123 0 224 10 0 10 10 0 8 3 kqueuepl 320 5077 0 25 4 0 4 4 0 8 1 pipepl 120 112374 0 189 8 0 8 8 0 8 2 fdescpl 440 81821 0 109 16 0 16 16 0 8 3 filepl 120 2726118 0 800 31 0 31 31 0 8 5 lockfpl 88 35919 0 6 1 0 1 1 0 8 0 pcredpl 24 81833 0 122 1 0 1 1 0 8 0 sessionpl 64 20349 0 44 1 0 1 1 0 8 0 pgrppl 40 29027 0 46 1 0 1 1 0 8 0 ucredpl 80 36657 0 122 3 0 3 3 0 8 0 zombiepl 144 81714 0 3 1 0 1 1 0 8 0 processpl 640 81833 0 122 26 0 26 26 0 8 5 procpl 616 83493 0 128 27 0 27 27 0 8 5 sockpl 472 81644 0 256 45 7 38 41 0 8 4 mcl2k 2048 1569909 0 3 134 0 134 134 4 3072 132 mbpl 256 10056711 0 117 64 0 64 64 1 384 52 bufpl 304 7009052 0 33823 51326 48719 2607 3124 0 8 0 anonpl 16 37902106 0 142101 801 15 786 786 0 253 137 amappl 72 15383835 0 58157 3565 2253 1312 1942 0 75 75 dma8192 8192 1 0 0 1 0 1 1 0 8 1 dma4096 4096 1 0 0 1 0 1 1 0 8 1 dma2048 2048 19589 0 0 1 0 1 1 0 8 1 dma1024 1024 19588 0 0 1 0 1 1 0 8 1 dma512 512 10859 0 1 1 0 1 1 0 8 0 dma256 256 9 0 0 1 0 1 1 0 8 1 dma64 64 17 0 0 1 0 1 1 0 8 1 dma32 32 8 0 0 1 0 1 1 0 8 1 dma16 16 3 0 1 1 0 1 1 0 8 0 aobjpl 64 2 0 2 1 0 1 1 0 8 0 uaddrrnd 40 81821 0 109 2 0 2 2 0 8 0 uaddrbestfit 32 2 0 2 1 0 1 1 0 8 0 uaddr 24 81821 0 109 1 0 1 1 0 8 0 vmmpekpl 168 891340 0 86 6 0 6 6 0 8 1 vmmpepl 168 34504561 0 97037 10173 5441 4732 6331 0 357 357 vmsppl 224 81820 0 108 9 0 9 9 0 8 2 pdppl 4096 81820 0 108 3616 3503 113 141 0 8 5 pvpl 32 96993649 0 309758 3570 675 2895 2912 0 265 261 pmappl 152 81820 0 108 6 0 6 6 0 8 1 extentpl 40 114 0 51 1 0 1 1 0 8 0 phpool 104 102773 0 6288 181 0 181 181 0 8 0 In use 70500K, total allocated 80816K; utilization 87.2% The reason I am suspecting hardware is because I have run -CURRENT snapshots on this system forever, and this is suddenly rearing its ugly head. I was running a snapshot from February when it crashed the first time a week from yesterday, so I went ahead and updated to "OpenBSD 5.4-current (GENERIC.MP) #54: Tue Sep 10 17:21:35 MDT 2013". It has since crashed another three times, which is why I doubt it's a software issue. Could someone that has some more knowledge about kernel panics take a look and make a suggestion if possible? Would you suspect RAM? CPU? Some other hardware bit? I didn't really see anything obvious to a kernel novice other than 'memmove', so I would appreciate any and all help you could give me to help track this down (and perhaps point the hosting company towards a failing component if that's what it turns out to be). And the dmesg: OpenBSD 5.4-current (GENERIC.MP) #54: Tue Sep 10 17:21:35 MDT 2013 dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 4242931712 (4046MB) avail mem = 4121878528 (3930MB) mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.5 @ 0xcfedf000 (39 entries) bios0: vendor Phoenix Technologies LTD version "1.2a" date 12/19/2008 bios0: Supermicro X7SBL acpi0 at bios0: rev 2 acpi0: sleep states S0 S1 S4 S5 acpi0: tables DSDT FACP _MAR TCPA MCFG HPET APIC BOOT SPCR ERST HEST BERT EINJ SLIC SSDT SSDT SSDT SSDT SSDT SSDT SSDT SSDT SSDT acpi0: wakeup devices PEG_(S5) PEX_(S5) LAN_(S5) USB4(S5) USB5(S5) USB7(S5) ESB2(S5) EXP1(S5) EXP5(S5) EXP6(S5) USB1(S5) USB2(S5) USB3(S5) USB6(S5) ESB1(S5) PCIB(S5) [...] acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimcfg0 at acpi0 addr 0xe0000000, bus 0-16 acpihpet0 at acpi0: 14318179 Hz acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Pentium(R) Dual-Core CPU E5400 @ 2.70GHz, 2693.65 MHz cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,XSAVE,NXE,LONG,LAHF,PERF cpu0: 2MB 64b/line 8-way L2 cache cpu0: apic clock running at 199MHz cpu1 at mainbus0: apid 1 (application processor) cpu1: Pentium(R) Dual-Core CPU E5400 @ 2.70GHz, 2693.25 MHz cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,XSAVE,NXE,LONG,LAHF,PERF cpu1: 2MB 64b/line 8-way L2 cache ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 20, 24 pins acpiprt0 at acpi0: bus 0 (PCI0) acpiprt1 at acpi0: bus 1 (PEG_) acpiprt2 at acpi0: bus -1 (PEX_) acpiprt3 at acpi0: bus 5 (EXP1) acpiprt4 at acpi0: bus 13 (EXP5) acpiprt5 at acpi0: bus 15 (EXP6) acpiprt6 at acpi0: bus 17 (PCIB) acpicpu0 at acpi0: C1, PSS acpicpu1 at acpi0: C1, PSS acpibtn0 at acpi0: PWRB acpivideo0 at acpi0: IGD0 ipmi at mainbus0 not configured cpu0: Enhanced SpeedStep 2693 MHz: speeds: 2700, 2400, 2000, 1600, 1200 MHz pci0 at mainbus0 bus 0 pchb0 at pci0 dev 0 function 0 "Intel 3200/3210 Host" rev 0x01 ppb0 at pci0 dev 1 function 0 "Intel 3200/3210 PCIE" rev 0x01: msi pci1 at ppb0 bus 1 mfi0 at pci1 dev 0 function 0 "Symbios Logic MegaRAID SAS2008" rev 0x03: apic 2 int 16 mfi0: "LSI MegaRAID SAS 9240-4i", firmware 20.10.1-0107 scsibus0 at mfi0: 16 targets sd0 at scsibus0 targ 0 lun 0: <LSI, MR9240-4i, 2.13> SCSI3 0/direct fixed naa.600605b004be9ab017e570f461b13ae3 sd0: 475883MB, 512 bytes/sector, 974608384 sectors scsibus1 at mfi0: 256 targets uhci0 at pci0 dev 26 function 0 "Intel 82801I USB" rev 0x02: apic 2 int 16 uhci1 at pci0 dev 26 function 1 "Intel 82801I USB" rev 0x02: apic 2 int 17 uhci2 at pci0 dev 26 function 2 "Intel 82801I USB" rev 0x02: apic 2 int 18 ehci0 at pci0 dev 26 function 7 "Intel 82801I USB" rev 0x02: apic 2 int 18 usb0 at ehci0: USB revision 2.0 uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1 ppb1 at pci0 dev 28 function 0 "Intel 82801I PCIE" rev 0x02: msi pci2 at ppb1 bus 5 ppb2 at pci0 dev 28 function 4 "Intel 82801I PCIE" rev 0x02: msi pci3 at ppb2 bus 13 em0 at pci3 dev 0 function 0 "Intel 82573E" rev 0x03: msi, address 00:30:48:fb:cf:9e ppb3 at pci0 dev 28 function 5 "Intel 82801I PCIE" rev 0x02: msi pci4 at ppb3 bus 15 em1 at pci4 dev 0 function 0 "Intel 82573L" rev 0x00: msi, address 00:30:48:fb:cf:9f uhci3 at pci0 dev 29 function 0 "Intel 82801I USB" rev 0x02: apic 2 int 23 uhci4 at pci0 dev 29 function 1 "Intel 82801I USB" rev 0x02: apic 2 int 22 uhci5 at pci0 dev 29 function 2 "Intel 82801I USB" rev 0x02: apic 2 int 18 ehci1 at pci0 dev 29 function 7 "Intel 82801I USB" rev 0x02: apic 2 int 23 usb1 at ehci1: USB revision 2.0 uhub1 at usb1 "Intel EHCI root hub" rev 2.00/1.00 addr 1 ppb4 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0x92 pci5 at ppb4 bus 17 vga1 at pci5 dev 4 function 0 "XGI Technology Volari Z9s/Z9m" rev 0x00 wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) wsdisplay0: screen 1-5 added (80x25, vt100 emulation) pcib0 at pci0 dev 31 function 0 "Intel 82801IR LPC" rev 0x02 ahci0 at pci0 dev 31 function 2 "Intel 82801I AHCI" rev 0x02: msi, AHCI 1.2 scsibus2 at ahci0: 32 targets ichiic0 at pci0 dev 31 function 3 "Intel 82801I SMBus" rev 0x02: apic 2 int 17 iic0 at ichiic0 lm1 at iic0 addr 0x2d: W83627HF wbng0 at iic0 addr 0x2f: w83793g spdmem0 at iic0 addr 0x50: 2GB DDR2 SDRAM non-parity PC2-5300CL5 spdmem1 at iic0 addr 0x52: 2GB DDR2 SDRAM non-parity PC2-5300CL5 "Intel 82801I Thermal" rev 0x02 at pci0 dev 31 function 6 not configured usb2 at uhci0: USB revision 1.0 uhub2 at usb2 "Intel UHCI root hub" rev 1.00/1.00 addr 1 usb3 at uhci1: USB revision 1.0 uhub3 at usb3 "Intel UHCI root hub" rev 1.00/1.00 addr 1 usb4 at uhci2: USB revision 1.0 uhub4 at usb4 "Intel UHCI root hub" rev 1.00/1.00 addr 1 usb5 at uhci3: USB revision 1.0 uhub5 at usb5 "Intel UHCI root hub" rev 1.00/1.00 addr 1 usb6 at uhci4: USB revision 1.0 uhub6 at usb6 "Intel UHCI root hub" rev 1.00/1.00 addr 1 usb7 at uhci5: USB revision 1.0 uhub7 at usb7 "Intel UHCI root hub" rev 1.00/1.00 addr 1 isa0 at pcib0 isadma0 at isa0 com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo pckbc0 at isa0 port 0x60/5 pckbd0 at pckbc0 (kbd slot) pckbc0: using irq 1 for kbd slot wskbd0 at pckbd0: console keyboard, using wsdisplay0 pms0 at pckbc0 (aux slot) pckbc0: using irq 12 for aux slot wsmouse0 at pms0 mux 0 pcppi0 at isa0 port 0x61 Thanks much! Benny -- "No matter how tempted I am with the prospect of unlimited power, I will not consume any energy field bigger than my head." -- #22 on Peter Anspach's Evil Overlord list