Hello,
I got a kernel panic and dropped to ddb as I was gdb'ing an nginx
worker. System got reinstalled (hdd replacement) 3 days ago from latest
snapshot. gdb itself was sudo'ed.
The diagnostic message talk about a mp_lock_held assertion failed and the
worker I was debugging contains the proxy module, which uses shm and locks. I
kept a snapshot of it so if you need some commands to be run on it, just ask.
Cheers,
Vincent
OpenBSD 5.4-current (GENERIC.MP) #239: Thu Jan 2 21:57:41 MST 2014
t...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 2120597504 (2022MB)
avail mem = 2056015872 (1960MB)
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xf71d0 (59 entries)
bios0: vendor Dell Inc. version "A08" date 07/28/2008
bios0: Dell Inc. Latitude D430
acpi0 at bios0: rev 0
acpi0: sleep states S0 S3 S4 S5
acpi0: tables DSDT FACP HPET APIC ASF! MCFG SLIC SSDT
acpi0: wakeup devices LID_(S3) PBTN(S4) PCI0(S5) USB0(S0) USB1(S0) USB2(S0)
USB3(S0) EHCI(S0) AZAL(S3) PCIE(S4) RP01(S3) RP02(S4) NIC_(S5) RP04(S3)
RP05(S3) RP06(S3)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpihpet0 at acpi0: 14318179 Hz
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Core(TM)2 Duo CPU U7700 @ 1.33GHz, 1330.24 MHz
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,NXE,LONG,LAHF,PERF
cpu0: 2MB 64b/line 8-way L2 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 133MHz
cpu0: mwait min=64, max=64, C-substates=0.2.2.2.2, IBE
cpu1 at mainbus0: apid 1 (application processor)
cpu1: Intel(R) Core(TM)2 Duo CPU U7700 @ 1.33GHz, 1330.00 MHz
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,NXE,LONG,LAHF,PERF
cpu1: 2MB 64b/line 8-way L2 cache
cpu1: smt 0, core 1, package 0
ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 20, 24 pins
ioapic0: misconfigured as apic 0, remapped to apid 2
acpimcfg0 at acpi0 addr 0xf0000000, bus 0-63
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 2 (PCIE)
acpiprt2 at acpi0: bus 11 (RP01)
acpiprt3 at acpi0: bus 12 (RP02)
acpiprt4 at acpi0: bus 9 (PXP0)
acpiprt5 at acpi0: bus -1 (RP04)
acpiprt6 at acpi0: bus -1 (RP05)
acpiprt7 at acpi0: bus -1 (RP06)
acpicpu0 at acpi0: C3, C2, C1, PSS
acpicpu1 at acpi0: C3, C2, C1, PSS
acpitz0 at acpi0: critical temperature is 99 degC
acpiac0 at acpi0: AC unit online
acpibat0 at acpi0: BAT0 model "DELL FG4516" serial 787 type LION oem "Sanyo"
acpibtn0 at acpi0: LID_
acpibtn1 at acpi0: PBTN
acpibtn2 at acpi0: SBTN
acpidock0 at acpi0: GDCK not docked (0)
acpivideo0 at acpi0: VID_
acpivideo1 at acpi0: VID2
cpu0: Enhanced SpeedStep 1330 MHz: speeds: 1333, 1067, 800 MHz
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 82945GM Host" rev 0x03
vga1 at pci0 dev 2 function 0 "Intel 82945GM Video" rev 0x03
intagp0 at vga1
agp0 at intagp0: aperture at 0xd0000000, size 0x10000000
inteldrm0 at vga1
drm0 at inteldrm0
inteldrm0: 1280x800
wsdisplay0 at vga1 mux 1: console (std, vt100 emulation)
wsdisplay0: screen 1-5 added (std, vt100 emulation)
"Intel 82945GM Video" rev 0x03 at pci0 dev 2 function 1 not configured
azalia0 at pci0 dev 27 function 0 "Intel 82801GB HD Audio" rev 0x01: msi
azalia0: codecs: Sigmatel STAC9200, Conexant/0x2c06, using Sigmatel STAC9200
audio0 at azalia0
ppb0 at pci0 dev 28 function 0 "Intel 82801GB PCIE" rev 0x01: msi
pci1 at ppb0 bus 11
ppb1 at pci0 dev 28 function 1 "Intel 82801GB PCIE" rev 0x01: msi
pci2 at ppb1 bus 12
wpi0 at pci2 dev 0 function 0 "Intel PRO/Wireless 3945ABG" rev 0x02: msi, MoW2,
address 00:1f:3c:b6:bd:c5
ppb2 at pci0 dev 28 function 2 "Intel 82801GB PCIE" rev 0x01: msi
pci3 at ppb2 bus 9
bge0 at pci3 dev 0 function 0 "Broadcom BCM5752" rev 0x02, BCM5752 A2 (0x6002):
msi, address 00:21:70:d8:44:e1
brgphy0 at bge0 phy 1: BCM5752 10/100/1000baseT PHY, rev. 0
uhci0 at pci0 dev 29 function 0 "Intel 82801GB USB" rev 0x01: apic 2 int 20
uhci1 at pci0 dev 29 function 1 "Intel 82801GB USB" rev 0x01: apic 2 int 21
uhci2 at pci0 dev 29 function 2 "Intel 82801GB USB" rev 0x01: apic 2 int 22
uhci3 at pci0 dev 29 function 3 "Intel 82801GB USB" rev 0x01: apic 2 int 23
ehci0 at pci0 dev 29 function 7 "Intel 82801GB USB" rev 0x01: apic 2 int 20
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb3 at pci0 dev 30 function 0 "Intel 82801BAM Hub-to-PCI" rev 0xe1
pci4 at ppb3 bus 2
cbb0 at pci4 dev 1 function 0 "Ricoh 5C476 CardBus" rev 0xb4: apic 2 int 19
"Ricoh 5C552 Firewire" rev 0x09 at pci4 dev 1 function 1 not configured
sdhc0 at pci4 dev 1 function 2 "Ricoh 5C822 SD/MMC" rev 0x18: apic 2 int 18
sdmmc0 at sdhc0
cardslot0 at cbb0 slot 0 flags 0
cardbus0 at cardslot0: bus 3 device 0 cacheline 0x0, lattimer 0x20
pcmcia0 at cardslot0
pcib0 at pci0 dev 31 function 0 "Intel 82801GBM LPC" rev 0x01
pciide0 at pci0 dev 31 function 1 "Intel 82801GB IDE" rev 0x01: DMA, channel 0
configured to compatibility, channel 1 configured to compatibility
wd0 at pciide0 channel 0 drive 0: <TOSHIBA MK6028GAL>
wd0: 16-sector PIO, LBA48, 57231MB, 117210240 sectors
wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 5
pciide0: channel 1 ignored (disabled)
ichiic0 at pci0 dev 31 function 3 "Intel 82801GB SMBus" rev 0x01: SMI
iic0 at ichiic0
spdmem0 at iic0 addr 0x51: 1GB DDR2 SDRAM non-parity PC2-6400CL5 SO-DIMM
usb1 at uhci0: USB revision 1.0
uhub1 at usb1 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb2 at uhci1: USB revision 1.0
uhub2 at usb2 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb3 at uhci2: USB revision 1.0
uhub3 at usb3 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb4 at uhci3: USB revision 1.0
uhub4 at usb4 "Intel UHCI root hub" rev 1.00/1.00 addr 1
isa0 at pcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
pckbc0: using irq 12 for aux slot
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
sdmmc0: can't enable card
umass0 at uhub0 port 6 configuration 1 interface 0 "Generic Mass Storage" rev
2.00/1.03 addr 2
umass0: using SCSI over Bulk-Only
scsibus0 at umass0: 2 targets, initiator 0
sd0 at scsibus0 targ 1 lun 0: <Generic, Flash Disk, 8.00> SCSI2 0/direct
removable
sd0: 64000MB, 512 bytes/sector, 131072000 sectors
uhub5 at uhub1 port 2 "Dell product 0xa005" rev 2.00/50.18 addr 2
uhub6 at uhub5 port 3 "O2 Micro OZ776 Hub" rev 1.10/1.10 addr 3
ugen0 at uhub6 port 2 "O2 O2Micro CCID SC Reader" rev 1.10/1.10 addr 4
ugen1 at uhub5 port 4 "Dell product 0x8140" rev 2.00/48.70 addr 5
vscsi0 at root
scsibus1 at vscsi0: 256 targets
softraid0 at root
scsibus2 at softraid0: 256 targets
softraid0: sd1 was not shutdown properly
sd1 at scsibus2 targ 1 lun 0: <OPENBSD, SR CRYPTO, 005> SCSI2 0/direct fixed
sd1: 55301MB, 512 bytes/sector, 113257648 sectors
root on sd1a (160c5ffa87a3fba9.a) swap on sd1b dump on sd1b
WARNING: / was not properly unmounted
panic: kernel diagnostic assertion "__mp_lock_held(&sched_lock) == 0" failed:
file "../../../../kern/kern_lock.c, line 126
Stopped at Debugger+0x5: leave
RUN AT LEAST 'trace' AND 'ps' AND INCLUDE OUTPUT WHEN REPORTING THIS PANIC!
IF RUNNING SMP, USE 'mach ddbcpu <#>' AND 'trace' ON OTHER PROCESSORS, TOO.
DO NOT EVEN BOTHER REPORTING THIS WITHOUT INCLUDING THAT INFORMATION.
ddb{0}> trace
Debugger() at Debugger+0x5
panic() at panic+0xee
__assert() at __assert+0x21
_kernel_lock_init() at _kernel_lock_init
issignal() at issignal+0x205
sleep_setup_signal() at sleep_setup_signal+0x94
tsleep() at tsleep+0x92
sys_sigsuspend() at sys_sigsuspend+0x46
syscall() at syscall+0x249
--- syscall (number 111) ---
end of kernel
end trace frame: 0x7f7fffff6280, count: -9
0x4e70535f5ac
ddb{0}> mach ddbcpu 1
Stopped at Debugger+0x5: leave
RUN AT LEAST 'trace' AND 'ps' AND INCLUDE OUTPUT WHEN REPORTING THIS PANIC!
IF RUNNING SMP, USE 'mach ddbcpu <#>' AND 'trace' ON OTHER PROCESSORS, TOO.
DO NOT EVEN BOTHER REPORTING THIS WITHOUT INCLUDING THAT INFORMATION.
ddb{1}> trace
Debugger() at Debugger+0x5
x86_ipi_handler() at x86_ipi_handler+0x64
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x1b
--- interrupt ---
Bad frame pointer: 0xffff800031925ed8
end trace frame: 0xffff800031925ed8, count: -3
cpu_idle_mwait_cycle+0x52:
ddb{1}> ps
PID PPID PGRP UID S FLAGS WAIT COMMAND
43 6523 3161 1000 3 0x82 nanosleep sleep
10554 1263 10554 0 3 0x83 wait gdb
8741 12802 12802 32767 3 0x90 kqread nginx
19688 12802 12802 32767 3 0x90 kqread nginx
12802 10554 12802 0 7 0x8000688 pause nginx
18923 10505 10505 67 3 0x90 kqread nginx
10505 1 10505 0 3 0x88 pause nginx
7887 9336 7887 1000 3 0x83 poll mutt
30967 12459 30967 1000 3 0x83 ttyin ksh
12459 17590 12459 1000 3 0xb2 select xterm
24518 29085 29085 1000 3 0x4100082 thrsleep chrome
32492 29085 29085 1000 3 0x4100082 thrsleep chrome
18027 29085 29085 1000 3 0x4100082 thrsleep chrome
17527 29085 29085 1000 3 0x4100082 thrsleep chrome
28109 29085 29085 1000 3 0x4100082 thrsleep chrome
1058 29085 29085 1000 3 0x4100082 thrsleep chrome
3516 29085 29085 1000 3 0x4100082 kqread chrome
4055 29085 29085 1000 3 0x82 thrsleep chrome
17395 28431 15484 1000 3 0x4100083 poll vlc
10245 28431 15484 1000 3 0x4100083 thrsleep vlc
15484 28431 15484 1000 3 0x83 sigwait vlc
28431 17265 28431 1000 3 0x8b pause ksh
17265 17590 17265 1000 3 0xb2 select xterm
2191 17590 29085 1000 3 0x4100082 thrsleep chrome
19060 17590 29085 1000 3 0x4100082 thrsleep chrome
16606 17590 29085 1000 3 0x4100082 thrsleep chrome
30674 17590 29085 1000 3 0x4100082 thrsleep chrome
14942 17590 29085 1000 3 0x4100082 thrsleep chrome
26625 29085 29085 1000 3 0x4100082 thrsleep chrome
21582 29085 29085 1000 3 0x4100082 thrsleep chrome
27482 29085 29085 1000 3 0x4100082 thrsleep chrome
3929 29085 29085 1000 3 0x4100082 thrsleep chrome
21808 29085 29085 1000 3 0x4100082 thrsleep chrome
3807 29085 29085 1000 3 0x4100082 thrsleep chrome
12001 29085 29085 1000 3 0x4100082 thrsleep chrome
28767 17590 29085 1000 3 0x4100082 thrsleep chrome
27932 29085 29085 1000 3 0x4100082 thrsleep chrome
30593 29085 29085 1000 3 0x4100082 thrsleep chrome
969 29085 29085 1000 3 0x4100082 thrsleep chrome
28864 29085 29085 1000 3 0x4100082 thrsleep chrome
28039 29085 29085 1000 3 0x4100082 thrsleep chrome
27653 29085 29085 1000 3 0x4100082 thrsleep chrome
3065 29085 29085 1000 3 0x4100082 thrsleep chrome
9204 29085 29085 1000 3 0x4100082 thrsleep chrome
19185 29085 29085 1000 3 0x4100082 thrsleep chrome
24574 29085 29085 1000 3 0x4100082 thrsleep chrome
11965 29085 29085 1000 3 0x4100082 thrsleep chrome
21781 29085 29085 1000 3 0x4100082 thrsleep chrome
7278 29085 29085 1000 3 0x4100082 thrsleep chrome
22960 29085 29085 1000 3 0x4100082 thrsleep chrome
6829 29085 29085 1000 3 0x4100082 thrsleep chrome
32566 29085 29085 1000 3 0x4100082 thrsleep chrome
6456 29085 29085 1000 3 0x4100082 kqread chrome
20952 29085 29085 1000 3 0x4100082 thrsleep chrome
26702 29085 29085 1000 3 0x4100082 kqread chrome
18238 29085 29085 1000 3 0x82 thrsleep chrome
23678 29085 29085 1000 3 0x82 thrsleep chrome
10583 29085 29085 1000 3 0x4100082 thrsleep chrome
25595 29085 29085 1000 3 0x4100082 kqread chrome
1151 29085 29085 1000 3 0x82 thrsleep chrome
1938 29085 29085 1000 3 0x4100082 thrsleep chrome
9803 29085 29085 1000 3 0x4100082 kqread chrome
31168 29085 29085 1000 3 0x82 thrsleep chrome
4904 29085 29085 1000 3 0x4100082 thrsleep chrome
27189 17590 29085 1000 3 0x4100082 thrsleep chrome
3740 29085 29085 1000 3 0x4100082 kqread chrome
186 17590 29085 1000 3 0x4100082 thrsleep chrome
5105 29085 29085 1000 3 0x82 thrsleep chrome
31618 17590 29085 1000 3 0x4100082 thrsleep chrome
1232 17590 29085 1000 3 0x4100082 thrsleep chrome
22890 17590 29085 1000 3 0x4100082 thrsleep chrome
22522 17590 29085 1000 3 0x4100082 thrsleep chrome
19520 17590 29085 1000 3 0x4100082 thrsleep chrome
23007 17590 29085 1000 3 0x4100082 thrsleep chrome
4982 17590 29085 1000 3 0x4100082 thrsleep chrome
3255 17590 29085 1000 3 0x4100082 kqread chrome
17075 17590 29085 1000 3 0x4100082 kqread chrome
2194 17590 29085 1000 3 0x4100082 thrsleep chrome
17887 17590 29085 1000 3 0x4100082 thrsleep chrome
22698 17590 29085 1000 3 0x4100082 kqread chrome
26401 17590 29085 1000 3 0x4100082 thrsleep chrome
13521 17590 29085 1000 3 0x4100082 piperd chrome
18320 17590 29085 1000 3 0x4100082 thrsleep chrome
29085 17590 29085 1000 3 0x82 poll chrome
16502 31070 16502 1000 3 0x83 select ssh
19760 23287 19760 1000 3 0x83 ttyin ksh
23287 17590 23287 1000 3 0xb2 select xterm
8575 31765 8575 1000 3 0x83 ttyin ksh
31765 17590 31765 1000 3 0xb2 select xterm
1263 21785 1263 1000 2 0x8b pause ksh
21785 17590 21785 1000 3 0xb2 select xterm
21270 1 21270 1000 3 0x80 poll dbus-daemon
836 1 13386 1000 3 0x80 select dbus-launch
22994 1 22994 0 3 0x83 ttyin getty
31070 28436 31070 1000 3 0x8b pause ksh
28436 17590 28436 1000 3 0xb2 select xterm
9336 32143 9336 1000 3 0x8b pause ksh
32143 17590 32143 1000 3 0xb2 select xterm
6523 17590 3161 1000 3 0x8a pause sh
17590 15936 3161 1000 3 0x82 select spectrwm
15936 3161 3161 1000 3 0x8a pause sh
20418 1 20418 1000 3 0xb0 select ssh-agent
3161 11307 3161 1000 3 0x8a pause sh
21971 1 29462 35 3 0x90 poll xconsole
2183 1 29462 0 2 0x80 netio xonsole
11307 7456 11307 0 3 0x80 wait xdm
15664 1 15664 0 3 0xb0 select sendmail
7439 1725 1725 0 3 0xb0 netio Xorg
25668 1 25668 0 3 0x83 ttyin getty
19248 1 19248 0 3 0x83 ttyin getty
4038 1 2038 0 3 0x83 ttyin getty
8220 1 8220 0 3 0x83 ttyin getty
1725 7456 1725 35 3 0xb2 select Xorg
7456 1 7456 0 3 0x88 pause xdm
5046 1 5046 0 3 0x80 select cron
18868 1 18868 99 3 0x90 poll sndiod
3876 10525 10525 74 3 0x90 bpf pflogd
10525 1 10525 0 3 0x80 netio pflogd
4300 19508 19508 73 3 0x90 poll syslogd
19508 1 19508 0 3 0x80 netio syslogd
4777 1 4777 77 3 0x90 poll dhclient
17185 1 17185 0 3 0x80 poll dhclient
29723 0 0 0 3 0x100200 aiodoned aiodoned
28759 0 0 0 3 0x100200 syncer update
24423 0 0 0 3 0x100200 cleaner cleaner
31149 0 0 0 3 0x100200 reaper reaper
1382 0 0 0 3 0x100200 pgdaemon pagedaemon
19103 0 0 0 3 0x100200 bored srdis
27949 0 0 0 3 0x100200 bored crypto
12347 0 0 0 3 0x100200 pftm pfpurge
5787 0 0 0 3 0x100200 mmctsk sdmmc0
28718 0 0 0 3 0x100200 usbtsk usbtask
4516 0 0 0 3 0x100200 usbatsk usbatsk
31358 0 0 0 3 0x100200 bored intelrel
5349 0 0 0 3 0x40100200 acpi0 acpi0
*29648 0 0 0 7 0x40100200 idle1
22859 0 0 0 3 0x100200 bored sensors
15003 0 0 0 3 0x100200 bored systq
20069 0 0 0 3 0x100200 bored syswq
8838 0 0 0 3 0x40100200 idle0
1 0 1 0 3 0x82 wait init
0 -1 0 0 3 0x200 scheduler swapper
[demime 1.01d removed an attachment of type application/pgp-signature]
