Hello,
This panic occurs when one of our border router with BGP is panicking, and
when the traffic (and bgp routes) failover to this router, it panics. The
router kernel and userland are sync from source repository, build from
source last night.
I haven't had the opportunity to follow the source code debugging
procedure. These are the data which I able to pull by taking pictures of
the panic.
DMESG
-----
OpenBSD 5.0 (GENERIC.MP) #5: Sat Aug 6 00:19:38 WIT 2011
r...@border-lf.mygreenlinks.net:/usr/src/sys/arch/i386/compile/GENERIC.MP
RTC BIOS diagnostic error e<fixed_disk,invalid_time>
cpu0: Intel(R) Xeon(R) CPU X3220 @ 2.40GHz ("GenuineIntel" 686-class) 2.41
GHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,S
SE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM
real mem = 2142711808 (2043MB)
avail mem = 2097582080 (2000MB)
mainbus0 at root
bios0 at mainbus0: AT/286+ BIOS, date 03/26/07, SMBIOS rev. 2.4 @
0x7fbe4000 (43 entries)
bios0: vendor Intel Corporation version
"S3000.86B.02.00.0054.061120091710" date 06/11/2009
bios0: Intel S3000AH
acpi0 at bios0: rev 2
acpi0: sleep states S0 S1 S4 S5
acpi0: tables DSDT SLIC FACP APIC WDDT HPET MCFG ASF! SSDT SSDT SSDT SSDT
SSDT HEST BERT ERST EINJ
acpi0: wakeup devices SLPB(S4) P32_(S4) UAR1(S1) PEX4(S4) PEX5(S4)
UHC1(S1) UHC2(S1) UHC3(S1) UHC4(S1) EHCI(S1) AC9M(S4) AZAL(
S4)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: apic clock running at 266MHz
cpu1 at mainbus0: apid 2 (application processor)
cpu1: Intel(R) Xeon(R) CPU X3220 @ 2.40GHz ("GenuineIntel" 686-class) 2.41
GHz
cpu1:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,S
SE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM
cpu2 at mainbus0: apid 1 (application processor)
cpu2: Intel(R) Xeon(R) CPU X3220 @ 2.40GHz ("GenuineIntel" 686-class) 2.41
GHz
cpu2:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,S
SE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM
cpu3 at mainbus0: apid 3 (application processor)
cpu3: Intel(R) Xeon(R) CPU X3220 @ 2.40GHz ("GenuineIntel" 686-class) 2.41
GHz
cpu3:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,S
SE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM
ioapic0 at mainbus0: apid 5 pa 0xfec00000, version 20, 24 pins
ioapic0: misconfigured as apic 0, remapped to apid 5
acpihpet0 at acpi0: 14318179 Hz
acpimcfg0 at acpi0 addr 0xf0000000, bus 0-127
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 4 (P32_)
acpiprt2 at acpi0: bus 1 (PEX0)
acpiprt3 at acpi0: bus -1 (PEX1)
acpiprt4 at acpi0: bus -1 (PEX2)
acpiprt5 at acpi0: bus -1 (PEX3)
acpiprt6 at acpi0: bus 2 (PEX4)
acpiprt7 at acpi0: bus 3 (PEX5)
acpicpu0 at acpi0: PSS
acpicpu1 at acpi0: PSS
acpicpu2 at acpi0: PSS
acpicpu3 at acpi0: PSS
acpibtn0 at acpi0: SLPB
bios0: ROM list: 0xc0000/0x9000 0xc9000/0x1000 0xca000/0x1000
cpu0: Enhanced SpeedStep 2401 MHz: speeds: 2394, 1596 MHz
pci0 at mainbus0 bus 0: configuration mode 1 (no bios)
pchb0 at pci0 dev 0 function 0 "Intel E7230 Host" rev 0x00
ppb0 at pci0 dev 28 function 0 "Intel 82801GB PCIE" rev 0x01: apic 5 int 17
pci1 at ppb0 bus 1
ppb1 at pci0 dev 28 function 4 "Intel 82801G PCIE" rev 0x01: apic 5 int 17
pci2 at ppb1 bus 2
em0 at pci2 dev 0 function 0 "Intel PRO/1000 PT (82571EB)" rev 0x06: msi,
address 00:15:17:86:54:08
em1 at pci2 dev 0 function 1 "Intel PRO/1000 PT (82571EB)" rev 0x06: msi,
address 00:15:17:86:54:09
ppb2 at pci0 dev 28 function 5 "Intel 82801G PCIE" rev 0x01: apic 5 int 16
pci3 at ppb2 bus 3
em2 at pci3 dev 0 function 0 "Intel PRO/1000MT (82573E)" rev 0x03: msi,
address 00:15:17:25:0a:9d
"Intel 82573E Serial" rev 0x03 at pci3 dev 0 function 3 not configured
"Intel 82573E KCS" rev 0x03 at pci3 dev 0 function 4 not configured
uhci0 at pci0 dev 29 function 0 "Intel 82801GB USB" rev 0x01: apic 5 int 23
uhci1 at pci0 dev 29 function 1 "Intel 82801GB USB" rev 0x01: apic 5 int 19
uhci2 at pci0 dev 29 function 2 "Intel 82801GB USB" rev 0x01: apic 5 int 18
uhci3 at pci0 dev 29 function 3 "Intel 82801GB USB" rev 0x01: apic 5 int 16
ehci0 at pci0 dev 29 function 7 "Intel 82801GB USB" rev 0x01: apic 5 int 23
ehci0: timed out waiting for BIOS
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb3 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0xe1
pci4 at ppb3 bus 4
skc0 at pci4 dev 0 function 0 "D-Link DGE-530T B1" rev 0x11, Yukon Lite
(0x9): apic 5 int 21
sk0 at skc0 port A: address 00:1b:11:10:07:26
eephy0 at sk0 phy 0: 88E1011 Gigabit PHY, rev. 5
skc1 at pci4 dev 1 function 0 "D-Link DGE-530T B1" rev 0x11, Yukon Lite
(0x9): apic 5 int 22
sk1 at skc1 port A: address 00:1c:f0:d1:a3:77
eephy1 at sk1 phy 0: 88E1011 Gigabit PHY, rev. 5
vga1 at pci4 dev 4 function 0 "ATI ES1000" rev 0x02
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
radeondrm0 at vga1: apic 5 int 18
drm0 at radeondrm0
em3 at pci4 dev 5 function 0 "Intel PRO/1000MT (82541GI)" rev 0x05: apic 5
int 17, address 00:15:17:25:0a:9e
ichpcib0 at pci0 dev 31 function 0 "Intel 82801GB LPC" rev 0x01: PM
disabled
pciide0 at pci0 dev 31 function 1 "Intel 82801GB IDE" rev 0x01: DMA,
channel 0 configured to compatibility, channel 1 configured to
compatibility
pciide0: channel 0 disabled (no drives)
pciide0: channel 1 disabled (no drives)
pciide1 at pci0 dev 31 function 2 "Intel 82801GB SATA" rev 0x01: DMA,
channel 0 configured to native-PCI, channel 1 configured to native-PCI
pciide1: using apic 5 int 19 for native-PCI interrupt
wd0 at pciide1 channel 0 drive 0: <ST380215AS>
wd0: 16-sector PIO, LBA48, 76319MB, 156301488 sectors
wd0(pciide1:0:0): using PIO mode 4, Ultra-DMA mode 6
ichiic0 at pci0 dev 31 function 3 "Intel 82801GB SMBus" rev 0x01: apic 5
int 19
iic0 at ichiic0
adt0 at iic0 addr 0x2e: sch5027 rev 0x69
spdmem0 at iic0 addr 0x50: 1GB DDR2 SDRAM ECC PC2-5300CL5
spdmem1 at iic0 addr 0x52: 1GB DDR2 SDRAM ECC PC2-5300CL5
usb1 at uhci0: USB revision 1.0
uhub1 at usb1 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb2 at uhci1: USB revision 1.0
uhub2 at usb2 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb3 at uhci2: USB revision 1.0
uhub3 at usb3 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb4 at uhci3: USB revision 1.0
uhub4 at usb4 "Intel UHCI root hub" rev 1.00/1.00 addr 1
isa0 at ichpcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
mtrr: Pentium Pro MTRR support
vscsi0 at root
scsibus0 at vscsi0: 256 targets
softraid0 at root
scsibus1 at softraid0: 256 targets
root on wd0a swap on wd0b dump on wd0b
WARNING: / was not properly unmounted
uvm_fault(0xd0a20000, 0xcfc00000, 0, 3) -> e
kernel: page fault trap, code=0
stopped at pmap_kenter_pa+0x68: xchgl %ebx,0(%eax)
ddb{0}> trace
pmap_kenter_pa(0,2c687000, 3, 1, 0) at pmap_kenter_pa+0x68
km_alloc(1000,d08c9d50, d08c9d5c, dea47ab4, 2000000) at km_alloc+0x193
pool_page_alloc(d0a17ce0, a, dea47af4, d8e4b534, 0) at pool_page_alloc+0x50
pool_do_get(d0a17ce0,a,0,0,0) at pool_do_get+0x4c
pool_get(d0a17ce0,a,0,2,e974001c) at pool_get+0x3d
pf_test_rule(dea47ca8,dea47ca4,2, d3473000, d8e8e200) at pf_test_rule+0xbdc
pf-test(2,2,d3474400,dea47dec,0) at pf_test+0xc64
ip_output(d8e8e200,0,d0a2da44,1,0) at ip_output+0x776
ip_forward(d8e8e200,0,d300fc00,dea47edc,0) at ip_forward+0x181
ipv4_input(d8e8e200,6,dea47ef4,d0445b55,d0203776) at ipv4_input+373
ipintr(d0203776,d2f6aa40,dea47f14,d05752df,0) at ipintr+0x73
netintr(0,282,0,0,d020226e) at netintr+0xc5
softintr_dispatch(1) at softintr_dispatch+0x4f
Xsoftnet() at Xsoftnet+0x17
cpu_idle_cycle(d0adb7e0) at cpu_idle_cycle+0xf
Bad frame pointer:0xd0b91e48
ddb{0}> ps
PID PPID PGRP UID S FLAGS WAIT
COMMAND
2480 1 2480 0 3 0x80 nanosleep
symon
30043 1 30043 0 3 0x80 select
sendmail
23600 1 23600 0 3 0x80 ttyin
getty
16714 1 16714 0 3 0x80 ttyin
getty
29264 1 29264 0 3 0x80 ttyin
getty
1853 1 1853 0 3 0x80 ttyin
getty
32331 1 32331 0 3 0x80 ttyin
getty
15906 1 15906 0 3 0x80 select
cron
5807 1 5807 0 3 0x80 select
inetd
17083 8181 8181 75 3 0x80 poll
bgpd
11555 8181 8181 75 3 0x80 poll
bgpd
8181 1 8181 0 3 0x80 poll
bgpd
16464 28391 28391 85 3 0x80 kqread
ospfd
30572 28391 28391 85 3 0x80 kqread
ospfd
28391 1 28391 0 3 0x80 kqread
ospfd
27689 15013 15013 91 3 0x80 kqread
snmpd
15013 1 15013 0 3 0x80 kqread
snmpd
26364 1 26364 0 3 0x80 select
sshd
737 29 5255 83 3 0x80 poll
ntpd
29 5255 5255 83 3 0x80 poll
ntpd
5255 1 5255 0 3 0x80 poll
ntpd
12647 880 880 74 3 0x80 bpf
pflogd
880 1 880 0 3 0x80 netio
pflogd
17218 1340 1340 73 7 0x80
syslogd
1340 1 1340 0 3 0x80 netio
syslogd
17 0 0 0 3 0x100200 aiodoned aiodoned
16 0 0 0 3 0x100200 syncer
update
15 0 0 0 3 0x100200 cleaner
cleaner
14 0 0 0 3 0x100200 reaper
reaper
13 0 0 0 3 0x100200 pgdaemon
pagedaemon
12 0 0 0 3 0x100200 bored
crypto
11 0 0 0 3 0x100200 pftm
pfpruge
10 0 0 0 3 0x100200 usbtsk
usbtask
9 0 0 0 3 0x100200 usbtask
usbtask
8 0 0 0 3 0x100200 acpi0
acpi0
7 0 0 0 7 0x40100200
idle3
6 0 0 0 7 0x40100200
idle2
5 0 0 0 3 0x40100200
idle1
4 0 0 0 3 0x100200 bored
syswq
*3 0 0 0 7 0x40100200
idle0
2 0 0 0 3 0x100200 kmalloc
kmthread
1 0 1 0 3 0x80 wait
init
0 -1 0 0 3 0x200 scheduler
swapper
Thank You,
Insan Praja SW
--
Using Opera's revolutionary email client: http://www.opera.com/mail/
