Box running pf/ospf/bgp/relayd entered ddb with "Stopped at cpu_idle_cycle+0xe: hlt".
Various of these seen in 'sh all pools':- mcl9k: pool(0xd0a2df78:mcl9k): page inconsistency: page 0xd1cc6000; item ordinal 0; addr 0xd1cc6018 (p 0xd1cc4000) mcl9k: pool(0xd0a2df78:mcl9k): page inconsistency: page 0xd1cc2000; item ordinal 0; addr 0xd1cc2014 (p 0xd1cc0000) Various pieces of ddb output below followed by a dmesg. Running i386 GENERIC from 9 Feb. Anyone have ideas or requests for more info to collect if it recurs? Thanks. [-- MARK -- Tue Feb 21 07:00:00 2012 .. Tue Feb 21 18:00:00 2012 -- MARK --] [-- Console down -- Tue Feb 21 18:05:52 2012] [-- Console up -- Tue Feb 21 18:05:52 2012] [-- MARK -- Tue Feb 21 19:00:00 2012 .. Tue Feb 21 21:00:00 2012 -- MARK --] Stopped at cpu_idle_cycle+0xe: hlt ddb> [read-only -- use ^E c ? for help] [read-only -- use ^E c ? for help] [read-only -- use ^E c ? for help] [attach to reopen] [bumped sthen@localhost] ddb> show panic the kernel did not panic ddb> tr cpu_idle_cycle(d0ae19a0) at cpu_idle_cycle+0xe Bad frame pointer: 0xd0b97e28 ddb> ps PID PPID PGRP UID S FLAGS WAIT COMMAND 11086 11860 11860 0 3 0x80 nanosleep newsyslog 11860 25649 11860 0 3 0x88 pause sh 25649 1142 1142 0 3 0x80 piperd cron 18671 1 18671 77 3 0x80 poll dhcpd 30544 1 30544 0 3 0x80 ttyin getty 14942 1 14942 0 3 0x80 ttyin getty 16786 1 16786 0 3 0x80 ttyin getty 16944 1 16944 0 3 0x80 ttyin getty 8025 1 8025 0 3 0x80 ttyin getty 26943 1 26943 0 3 0x80 ttyin getty 1142 1 1142 0 3 0x80 select cron 5690 1 5690 0 3 0x80 select sendmail 24136 1 24136 0 3 0x80 nanosleep sensorsd 3775 1 3775 535 3 0x80 nanosleep symon 5698 1 5698 99 3 0x80 poll sndiod 2672 1 2672 0 3 0x80 select inetd 5804 1 5804 71 3 0x80 kqread ftp-proxy 11338 3713 3713 89 3 0x80 kqread relayd 2489 3713 3713 89 3 0x80 kqread relayd 29211 3713 3713 89 3 0x80 kqread relayd 19896 3713 3713 89 3 0x80 kqread relayd 3713 8594 3713 89 3 0x80 kqread relayd 20059 8594 20059 89 3 0x80 kqread relayd 5952 8594 5952 89 3 0x80 kqread relayd 8594 1 8594 0 3 0x80 kqread relayd 30614 7638 7638 75 3 0x80 poll bgpd 28779 7638 7638 75 3 0x80 poll bgpd 7638 1 7638 0 3 0x80 poll bgpd 23530 12413 12413 85 3 0x80 kqread ospfd 4844 12413 12413 85 3 0x80 kqread ospfd 12413 1 12413 0 3 0x80 kqread ospfd 19399 21681 21681 91 3 0x80 kqread snmpd 21681 1 21681 0 3 0x80 kqread snmpd 29693 1 29693 0 3 0x80 select sshd 17309 11222 13566 83 3 0x80 poll ntpd 11222 13566 13566 83 3 0x80 poll ntpd 13566 1 13566 0 3 0x80 poll ntpd 7424 24405 24405 74 3 0x80 bpf pflogd 24405 1 24405 0 3 0x80 netio pflogd 5067 30685 30685 73 3 0x80 poll syslogd 30685 1 30685 0 3 0x80 netio syslogd 14 0 0 0 3 0x100200 aiodoned aiodoned 13 0 0 0 3 0x100200 syncer update 12 0 0 0 3 0x100200 cleaner cleaner 11 0 0 0 3 0x100200 reaper reaper 10 0 0 0 3 0x100200 pgdaemon pagedaemon 9 0 0 0 3 0x100200 bored crypto 8 0 0 0 3 0x100200 pftm pfpurge 7 0 0 0 3 0x100200 usbtsk usbtask 6 0 0 0 3 0x100200 usbatsk usbatsk 5 0 0 0 3 0x100200 acpi0 acpi0 4 0 0 0 3 0x100200 bored syswq * 3 0 0 0 7 0x40100200 idle0 2 0 0 0 3 0x100200 kmalloc kmthread 1 0 1 0 3 0x80 wait init 0 -1 0 0 3 0x200 scheduler swapper ddb> sh reg ds 0x10 es 0x10 fs 0x20 gs 0 edi 0xd0ae19d4 cpu_info_primary+0x34 esi 0xd6d8b298 end+0x62a6c94 ebp 0xdc1b1f8c end+0xb6cd988 ebx 0 edx 0x1 ecx 0 eax 0 eip 0xd02008d2 cpu_idle_cycle+0xe cs 0x8 eflags 0x246 esp 0xdc1b1f60 end+0xb6cd95c ss 0x10 cpu_idle_cycle+0xe: hlt ddb> sh uvmexp Current UVM status: pagesize=4096 (0x1000), pagemask=0xfff, pageshift=12 255281 VM pages: 4744 active, 692 inactive, 0 wired, 232168 free min 10% (25) anon, 10% (25) vnode, 5% (12) vtext pages 0 anon, 0 vnode, 0 vtext freemin=8509, free-target=11345, inactive-target=0, wired-max=85093 faults=76492848, traps=60500530, intrs=1089622887, ctxswitch=28674543 fpuswit ch=793866 softint=1675263079, syscalls=215393823, swapins=0, swapouts=0, kmapent=50 fault counts: noram=0, noanon=0, pgwait=0, pgrele=0 ok relocks(total)=27567(27567), anget(retries)=53422799(0), amapcopy=164175 45 neighbor anon/obj pg=2908271/47825150, gets(lock/unlock)=14004865/27567 cases: anon=47464331, anoncow=5958468, obj=12274522, prcopy=1730343, przero =9065184 daemon and swap counts: woke=0, revs=0, scans=0, obscans=0, anscans=0 busy=0, freed=0, reactivate=0, deactivate=0 pageouts=0, pending=0, nswget=0 nswapdev=1, nanon=0, nanonneeded=0 nfreeanon=0 swpages=66267, swpginuse=0, swpgonly=0 paging=0 kernel pointers: objs(kern)=0xd0a04760 ddb> sh all pools Name Size Requests Fail Releases Pgreq Pgrel Npage Hiwat Minpg Maxpg Idle pfsync 40 85503 0 85503 1 0 1 1 0 8 1 inpcbpl 232 1120366 0 1120217 12 0 12 12 0 8 2 plimitpl 148 37635 0 37606 2 0 2 2 0 8 0 synpl 192 19651 0 19651 1 0 1 1 0 8 1 sackhlpl 20 679 0 679 1 0 1 1 0 8 1 tcpqepl 16 401617 0 401617 1 0 1 1 0 8 1 tcpcbpl 408 1081254 0 1081213 9 0 9 9 0 8 4 rtentpl 116 46189 0 45751 20 0 20 20 0 8 6 pffrag 80 4823 0 4823 1 0 1 1 0 20 1 pffrent 24 9730 0 9730 1 0 1 1 0 8 1 pfosfp 28 6150 0 5740 3 0 3 3 0 8 0 pfosfpen 108 10500 0 9800 173 154 19 19 0 8 0 pfrke_plain 92 5549 0 5065 41 20 21 26 0 8 8 pfrktable 1252 583 0 539 32 0 32 32 0 8 5 pfruleitempl 12 7679482 0 7678671 11 0 11 11 0 8 3 pfstateitempl 12 37705433 0 37703660 32 6 26 27 0 8 5 pfstatekeypl 72 38142525 0 38140752 32822 32726 96 163 0 8 8 pfstatepl 220 36635527 0 36633826 535680 535442 238 546 0 8 8 pfsnitempl 12 12898 0 12897 1 0 1 1 0 8 0 pfsrctrpl 124 6402 0 6401 1 0 1 1 0 8 0 pfrulepl 1192 10053 0 9002 3028 2637 391 726 0 8 8 dirhash 1024 9664 0 9445 98 0 98 98 0 128 42 dino1pl 128 1685640 0 1679725 191 0 191 191 0 8 0 ffsino 184 1685640 0 1679725 282 0 282 282 0 8 0 nchpl 88 1851502 0 1846704 107 0 107 107 0 8 0 vnodes 156 5926 0 0 238 0 238 238 0 8 0 namei 1024 5759590 0 5759590 3 0 3 3 0 8 3 scxspl 124 905304 0 905304 4 0 4 4 0 8 4 wdcspl 96 9 0 9 1 0 1 1 0 8 1 pfiaddrpl 100 11 0 10 1 0 1 1 0 8 0 ifaddritempl 32 191 0 4 2 0 2 2 0 8 0 wqtasks 20 194060 0 194060 1 0 1 1 0 8 1 sigapl 296 302227 0 302185 6 0 6 6 0 8 2 knotepl 64 5258371 0 5258262 3 0 3 3 0 8 1 kqueuepl 192 16 0 2 1 0 1 1 0 8 0 pipepl 72 380964 0 380961 1 0 1 1 0 8 0 fdescpl 300 302228 0 302185 6 0 6 6 0 8 2 filepl 88 3045329 0 3045064 9 0 9 9 0 8 2 lockfpl 56 3444 0 3442 1 0 1 1 0 8 0 pcredpl 20 302240 0 302185 1 0 1 1 0 8 0 sessionpl 48 52197 0 52173 1 0 1 1 0 8 0 pgrppl 24 52678 0 52651 1 0 1 1 0 8 0 ucredpl 80 245140 0 245106 2 0 2 2 0 8 1 zombiepl 72 302185 0 302185 1 0 1 1 0 8 1 processpl 88 302240 0 302185 2 0 2 2 0 8 0 procpl 332 302240 0 302185 8 0 8 8 0 8 3 sockpl 272 1260026 0 1259801 19 0 19 19 0 8 3 mcl9k 9216 51 0 17 42 0 42 42 8 6144 8 mcl9k: pool(0xd0a2df78:mcl9k): page inconsistency: page 0xd1cc6000; item ordinal 0; addr 0xd1cc6018 (p 0xd1cc4000) mcl9k: pool(0xd0a2df78:mcl9k): page inconsistency: page 0xd1cc2000; item ordinal 0; addr 0xd1cc2014 (p 0xd1cc0000) mcl9k: pool(0xd0a2df78:mcl9k): page inconsistency: page 0xd1cbe000; item ordinal 0; addr 0xd1cbe010 (p 0xd1cbc000) mcl9k: pool(0xd0a2df78:mcl9k): page inconsistency: page 0xd1cba000; item ordinal 0; addr 0xd1cba00c (p 0xd1cb8000) mcl9k: pool(0xd0a2df78:mcl9k): page inconsistency: page 0xd1cb6000; item ordinal 0; addr 0xd1cb6008 (p 0xd1cb4000) mcl9k: pool(0xd0a2df78:mcl9k): page inconsistency: page 0xd1cb2000; item ordinal 0; addr 0xd1cb2004 (p 0xd1cb0000) mcl2k 2048 1520482091 0 1520481971 158 0 158 158 4 3072 93 mbpl 256 62209509 0 62209271 37 0 37 37 1 384 21 bufpl 172 258607 0 255568 140 0 140 140 0 8 1 anonpl 12 16753995 0 16750498 18 0 18 18 0 48 3 amappl 44 13008470 0 13006867 37 0 37 37 0 46 14 dma512 512 3 0 3 1 0 1 1 0 8 1 dma256 256 8 0 8 1 0 1 1 0 8 1 dma64 64 88 0 88 1 0 1 1 0 8 1 dma16 16 8 0 7 1 0 1 1 0 8 0 aobjpl 40 1 0 0 1 0 1 1 0 8 0 vmmpekpl 88 1084818 0 1084735 4 0 4 4 0 8 2 vmmpepl 88 25488195 0 25485799 107 0 107 107 0 183 47 vmsppl 180 302227 0 302185 4 0 4 4 0 8 1 pmappl 72 302227 0 302185 2 0 2 2 0 8 1 extentpl 20 232 0 202 1 0 1 1 0 8 0 phpool 52 3488 0 2637 15 0 15 15 0 8 0 ddb> sh malloc Type InUse MemUse HighUse Limit Requests Type Lim Kern Lim devbuf 5504 3150K 3279K 39321K 201416 0 0 pcb 91 8K 10K 39321K 139677 0 0 routetbl 831 48K 70K 39321K 896657 0 0 ifaddr 403 82K 82K 39321K 434 0 0 sysctl 2 0K 0K 39321K 2 0 0 ioctlops 0 0K 4K 39321K 318794 0 0 iov 0 0K 0K 39321K 112936 0 0 mount 14 7K 7K 39321K 14 0 0 vnodes 1515 94K 94K 39321K 21009 0 0 UFS quota 1 16K 16K 39321K 1 0 0 UFS mount 25 79K 79K 39321K 25 0 0 shm 2 0K 0K 39321K 2 0 0 VM map 2 0K 0K 39321K 2 0 0 sem 2 0K 0K 39321K 2 0 0 dirhash 138 26K 43K 39321K 6267 0 0 ACPI 1425 85K 89K 39321K 5659 0 0 file 0 0K 2K 39321K 14 0 0 file desc 10 3K 3K 39321K 12 0 0 proc 20 5K 5K 39321K 20 0 0 VFS cluster 0 0K 0K 39321K 4357 0 0 NFS srvsock 1 0K 0K 39321K 1 0 0 NFS daemon 1 0K 0K 39321K 1 0 0 ip_moptions 23 1K 1K 39321K 27 0 0 in_multi 522 18K 18K 39321K 539 0 0 ether_multi 358 11K 11K 39321K 365 0 0 mrt 1 0K 0K 39321K 1 0 0 ISOFS mount 1 16K 16K 39321K 1 0 0 MSDOSFS mount 1 8K 8K 39321K 1 0 0 ttys 414 281K 281K 39321K 414 0 0 exec 0 0K 1K 39321K 296137 0 0 pfkey data 2 0K 0K 39321K 23 0 0 tdb 7 4K 5K 39321K 11 0 0 xform_data 4 0K 0K 39321K 4 0 0 pagedep 1 4K 4K 39321K 1 0 0 inodedep 1 16K 16K 39321K 1 0 0 newblk 1 0K 0K 39321K 1 0 0 VM swap 7 26K 26K 39321K 7 0 0 UVM amap 1642 139K 263K 39321K 13219569 0 0 UVM aobj 2 1K 1K 39321K 2 0 0 USB 12 1K 1K 39321K 12 0 0 USB device 3 0K 0K 39321K 3 0 0 memdesc 1 4K 4K 39321K 1 0 0 crypto data 1 1K 1K 39321K 1 0 0 packet tags 0 0K 0K 39321K 2802 0 0 ip6_options 0 0K 0K 39321K 2 0 0 NDP 90 8K 9K 39321K 114 0 0 temp 427 54K 66K 39321K 2663946 0 0 NTFS hash 1 16K 16K 39321K 1 0 0 OpenBSD 5.1 (GENERIC) #156: Thu Feb 9 13:28:45 MST 2012 dera...@i386.openbsd.org:/usr/src/sys/arch/i386/compile/GENERIC cpu0: Intel(R) Xeon(TM) CPU 2.80GHz ("GenuineIntel" 686-class) 2.79 GHz cpu0: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36, CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,CNXT-ID,xTPR real mem = 1073115136 (1023MB) avail mem = 1045463040 (997MB) mainbus0 at root bios0 at mainbus0: AT/286+ BIOS, date 04/14/04, BIOS32 rev. 0 @ 0xffe90, SMBIO S rev. 2.3 @ 0xfb320 (56 entries) bios0: vendor Dell Computer Corporation version "A08" date 04/14/2004 bios0: Dell Computer Corporation PowerEdge 1750 acpi0 at bios0: rev 0 acpi0: sleep states S0 S4 S5 acpi0: tables DSDT FACP APIC SPCR acpi0: wakeup devices RTC_(S5) PCI0(S5) PCI3(S5) PCI2(S5) PCI1(S5) acpitimer0 at acpi0: 3579545 Hz, 32 bits acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: apic clock running at 132MHz cpu at mainbus0: not configured ioapic0 at mainbus0: apid 8 pa 0xfec00000, version 11, 16 pins ioapic0: misconfigured as apic 0, remapped to apid 8 ioapic1 at mainbus0: apid 9 pa 0xfec01000, version 11, 16 pins ioapic1: misconfigured as apic 0, remapped to apid 9 ioapic2 at mainbus0: apid 10 pa 0xfec02000, version 11, 16 pins ioapic2: misconfigured as apic 0, remapped to apid 10 acpiprt0 at acpi0: bus 0 (PCI0) acpiprt1 at acpi0: bus 4 (PCI4) acpiprt2 at acpi0: bus 3 (PCI3) acpiprt3 at acpi0: bus 2 (PCI2) acpiprt4 at acpi0: bus 1 (PCI1) acpicpu0 at acpi0 bios0: ROM list: 0xc0000/0x8000 0xc8000/0x1000 0xc9000/0x2200 0xcb800/0x1800 0 xcd000/0x1800 0xec000/0x4000! ipmi at mainbus0 not configured pci0 at mainbus0 bus 0: configuration mode 1 (bios) pchb0 at pci0 dev 0 function 0 "ServerWorks CNB20-HE Host (GC-LE)" rev 0x33 pchb1 at pci0 dev 0 function 1 "ServerWorks CNB20-HE Host (GC-LE)" rev 0x00 pci1 at pchb1 bus 1 pchb2 at pci0 dev 0 function 2 "ServerWorks CNB20-HE Host (GC-LE)" rev 0x00 pci2 at pchb2 bus 3 em0 at pci2 dev 6 function 0 "Intel PRO/1000MT (82546EB)" rev 0x01: apic 9 int 8, address 00:11:0a:62:92:68 em1 at pci2 dev 6 function 1 "Intel PRO/1000MT (82546EB)" rev 0x01: apic 9 int 9, address 00:11:0a:62:92:69 vga1 at pci0 dev 14 function 0 "ATI Rage XL" rev 0x27 wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) wsdisplay0: screen 1-5 added (80x25, vt100 emulation) piixpm0 at pci0 dev 15 function 0 "ServerWorks CSB5" rev 0x93: SMBus disabled pciide0 at pci0 dev 15 function 1 "ServerWorks CSB5 IDE" rev 0x93: DMA atapiscsi0 at pciide0 channel 1 drive 0 scsibus0 at atapiscsi0: 2 targets cd0 at scsibus0 targ 0 lun 0: <TEAC, CD-224E, K.9A> ATAPI 5/cdrom removable cd0(pciide0:1:0): using PIO mode 4, DMA mode 2, Ultra-DMA mode 2 ohci0 at pci0 dev 15 function 2 "ServerWorks OSB4/CSB5 USB" rev 0x05: apic 8 i nt 11, version 1.0, legacy support pcib0 at pci0 dev 15 function 3 "ServerWorks CSB5 LPC" rev 0x00 pchb3 at pci0 dev 16 function 0 "ServerWorks CIOB-E" rev 0x12 pchb4 at pci0 dev 16 function 2 "ServerWorks CIOB-E" rev 0x12 pci3 at pchb4 bus 2 bge0 at pci3 dev 0 function 0 "Broadcom BCM5704C" rev 0x02, BCM5704 A2 (0x2002 ): apic 9 int 0, address 00:0f:1f:67:54:a0 brgphy0 at bge0 phy 1: BCM5704 10/100/1000baseT PHY, rev. 0 bge1 at pci3 dev 0 function 1 "Broadcom BCM5704C" rev 0x02, BCM5704 A2 (0x2002 ): apic 9 int 1, address 00:0f:1f:67:54:a2 brgphy1 at bge1 phy 1: BCM5704 10/100/1000baseT PHY, rev. 0 pchb5 at pci0 dev 17 function 0 "ServerWorks CIOB-X2 PCIX" rev 0x05 pchb6 at pci0 dev 17 function 2 "ServerWorks CIOB-X2 PCIX" rev 0x05 pci4 at pchb6 bus 4 ami0 at pci4 dev 3 function 0 "Dell PERC 4/Di Verde" rev 0x02: apic 9 int 2 ami0: Dell 14a, 32b, FW 412W, BIOS vH406, 128MB RAM ami0: 2 channels, 0 FC loops, 1 logical drives scsibus1 at ami0: 40 targets sd0 at scsibus1 targ 0 lun 0: <AMI, Host drive #00, > SCSI2 0/direct fixed sd0: 17280MB, 512 bytes/sector, 35389440 sectors scsibus2 at ami0: 16 targets safte0 at scsibus2 targ 6 lun 0: <PE/PV, 1x3 SCSI BP, 1.1> SCSI2 3/processor f ixed scsibus3 at ami0: 16 targets usb0 at ohci0: USB revision 1.0 uhub0 at usb0 "ServerWorks OHCI root hub" rev 1.00/1.00 addr 1 isa0 at pcib0 isadma0 at isa0 com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo com0: console pckbc0 at isa0 port 0x60/5 pckbd0 at pckbc0 (kbd slot) pckbc0: using irq 1 for kbd slot wskbd0 at pckbd0: console keyboard, using wsdisplay0 pcppi0 at isa0 port 0x61 spkr0 at pcppi0 npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16 fdc0 at isa0 port 0x3f0/6 irq 6 drq 2 fd0 at fdc0 drive 0: 1.44MB 80 cyl, 2 head, 18 sec mtrr: Pentium Pro MTRR support vscsi0 at root scsibus4 at vscsi0: 256 targets softraid0 at root scsibus5 at softraid0: 256 targets root on sd0a (7232af9e27547fba.a) swap on sd0b dump on sd0b Automatic boot in progress: starting file system checks.