I have an AMD based system that I frequently find at the ddb prompt
with messages about tcp_usrreq and nfsrv_recv (see trace info
below). I haven't discerned a specific pattern or time between
occurrences. It ranges from happening twice in the same day to going
for 3+ days without incident.
When I review the dmesg I usually see a message like:
<6>arp info overwritten for 10.10.15.106 by 00:26:bb:72:c0:e5 on sk0
The server's ip is 10.10.15.10. I've wondered whether they're
related. The server exports user home directories to clients (see
below) but all the systems that access the server have specified IP
addresses in the .30 - .50 range. Everything above .99 is reserved for
ephemeral clients.
Does anyone have any ideas or pointers to resources or articles that
might help resolve this?
This may be an unrelated problem, but I almost always get a kernel
panic when I type `boot <command>` at the ddb prompt. I've included it
at the bottom of the message.
Attached are the initial trace, ps, show registers, dmesg and the
results from `boot dump` (in this case) followed by the kernel panic.
Cheers and thanks for any help,
--Aaron
----------------------------------------------------------------------
Hardware:
AMD Athlon(tm) 64 X2 Dual Core Processor 3800+
1GB RAM
1xPATA boot drive
4xSATA drives using stacked softraid to create two mirrored drives
with encryption
OS Version:
OpenBSD 4.8 i386 (no patches applied yet)
Services:
nfsd
dovecot
fetchmail
procmail
sendmail
nfsd exports home directories to Mac OS X 10.6 clients over IPv4.
dhcp and dns are handled by another OpenBSD 4.8 server.
----------------------------------------------------------------------
ddb> trace
tcp_usrreq(d6b40014,8,0,80,0) at tcp_usrreq+0xbd
soreceive(d6b40014,dc466e20,dc466dfc,dc466e24,0) at soreceive+0x624
nfsrv_rcv(d6b40014,d1f35cc0,1,dc466e54,d0202fe5) at nfsrv_rcv+0x138
soisdisconnected(d6b40014,0,d6b40014,d6b417dc,d048be50) at
soisdisconnected+0x66
tcp_close(d6b417dc,3c,0,0,d0ac1ad4) at tcp_close+0x97
tcp_timer_rexmt(d6b417dc,dc466f00,d03cbe36,dc466ef4,d1d949e0) at
tcp_timer_rexmt+0xa1
softclock(0,dc466f10,d07672d1,0,d0201fc6) at softclock+0x225
softintr_dispatch(0) at softintr_dispatch+0x4f
Xsoftclock() at Xsoftclock+0x12
--- interrupt ---
cpu_idle_cycle(d0ac1aa0) at cpu_idle_cycle+0xf
Bad frame pointer: 0xd0b75e48
ddb> ps
PID PPID PGRP UID S FLAGS WAIT COMMAND
13697 20622 20622 518 3 0x4180 kqread imap-login
7751 20622 20622 1000 3 0x44180 kqread imap
26824 20622 20622 1000 3 0x44180 kqread imap
19716 20622 20622 2000 3 0x44180 kqread imap
14259 21940 14259 1000 3 0x4080 ttyin ksh
21940 8247 8247 1000 3 0x180 select sshd
30780 20622 20622 2000 3 0x44180 kqread imap
13379 20622 20622 518 3 0x4180 kqread imap-login
16574 20622 20622 518 3 0x4180 kqread imap-login
8247 28378 8247 0 3 0x4180 netio sshd
22687 20622 20622 1000 3 0x44180 kqread imap
18020 20622 20622 1000 3 0x44180 kqread imap
29759 1 29759 0 3 0x4080 ttyin getty
21313 20622 20622 518 3 0x4180 kqread pop3-login
18205 20622 20622 518 3 0x4180 kqread pop3-login
146 20622 20622 518 3 0x4180 kqread pop3-login
5104 1 5104 0 3 0x40180 select sendmail
21349 1 21349 0 3 0x4080 ttyin getty
18697 1 18697 0 3 0x4080 ttyin getty
11800 1 11800 0 3 0x4080 ttyin getty
24887 1 24887 0 3 0x4080 ttyin getty
7814 1 7814 0 3 0x4080 ttyin getty
21068 1 21068 0 3 0x80 select cron
14352 20622 20622 0 3 0x4180 kqread dovecot-auth
20622 1 20622 0 3 0x80 kqread dovecot
29601 1 29601 0 3 0x180 select inetd
28378 1 28378 0 3 0x80 select sshd
19159 26558 27956 83 3 0x180 poll ntpd
26558 27956 27956 83 3 0x180 poll ntpd
27956 1 27956 0 3 0x80 poll ntpd
17993 10933 10933 0 3 0 nfsndlck nfsd
16904 10933 10933 0 3 0 nfsndlck nfsd
19532 10933 10933 0 2 0x80 nfsd
18407 10933 10933 0 3 0 nfsndlck nfsd
10933 1 10933 0 3 0x80 netcon nfsd
12841 1 12841 0 3 0x80 select mountd
22181 1 22181 28 3 0x180 poll portmap
2998 12777 12777 74 3 0x180 bpf pflogd
12777 1 12777 0 3 0x80 netio pflogd
32090 16237 16237 73 2 0x180 syslogd
16237 1 16237 0 3 0x88 netio syslogd
22583 1 22583 77 3 0x180 poll dhclient
3071 1 20000 0 3 0x80 poll dhclient
15 0 0 0 3 0x100200 bored crypto
14 0 0 0 3 0x100200 aiodoned aiodoned
13 0 0 0 3 0x100200 syncer update
12 0 0 0 3 0x100200 cleaner cleaner
11 0 0 0 3 0x100200 reaper reaper
10 0 0 0 3 0x100200 pgdaemon pagedaemon
9 0 0 0 3 0x100200 pftm pfpurge
8 0 0 0 3 0x100200 usbevt usb1
7 0 0 0 3 0x100200 usbtsk usbtask
6 0 0 0 3 0x100200 usbevt usb0
5 0 0 0 3 0x100200 acpi0 acpi0
4 0 0 0 3 0x100200 bored syswq
* 3 0 0 0 7 0x40100200 idle0
2 0 0 0 3 0x100200 kmalloc kmthread
1 0 1 0 3 0x4080 wait init
0 -1 0 0 3 0x80200 scheduler swapper
ddb> show registers
ds 0x10
es 0x10
fs 0x58
gs 0x10
edi 0xd6d66278 end+0x62a1b74
esi 0x80
ebp 0xdc466d64 end+0xb9a2660
ebx 0
edx 0
ecx 0xd6b40014 end+0x607b910
eax 0x40
eip 0xd048ce4d tcp_usrreq+0xbd
cs 0x50
eflags 0x10283
esp 0xdc466d1c end+0xb9a2618
ss 0xdc460010 end+0xb99b90c
tcp_usrreq+0xbd: movzwl 0x68(%edx),%ecx
ddb> dmesg
OpenBSD 4.8 (GENERIC) #136: Mon Aug 16 09:06:23 MDT 2010
dera...@i386.openbsd.org:/usr/src/sys/arch/i386/compile/GENERIC
cpu0: AMD Athlon(tm) 64 X2 Dual Core Processor 3800+ ("AuthenticAMD"
686-class, 512KB L2 cache) 2.01 GHz
cpu0:FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT
,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,CX16
real mem = 1072193536 (1022MB)
avail mem = 1044692992 (996MB)
mainbus0 at root
bios0 at mainbus0: AT/286+ BIOS, date 11/29/06, BIOS32 rev. 0 @
0xf1fc0, SMBIOS rev. 2.4 @ 0xf0000 (67 entries)
bios0: vendor Phoenix Technologies, LTD version "ASUS M2NPV-VM ACPI
BIOS Revision 0603" date 11/29/2006
bios0: ASUSTek Computer INC. M2NPV-VM
acpi0 at bios0: rev 2
acpi0: sleep states S0 S1 S3 S4 S5
acpi0: tables DSDT FACP HPET MCFG APIC
acpi0: wakeup devices HUB0(S5) XVRA(S5) XVRB(S5) XVRC(S5) USB0(S4)
USB2(S4) AZA
D(S5) MMAC(S5) MMCI(S5) UAR1(S5) UAR2(S5) PS2M(S4) PS2K(S4)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpihpet0 at acpi0: 25000000 Hz
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: apic clock running at 200MHz
cpu at mainbus0: not configured
ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 11, 24 pins
ioapic0: misconfigured as apic 0, remapped to apid 2
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 4 (HUB0)
acpicpu0 at acpi0
acpitz0 at acpi0: critical temperature 75 degC
acpibtn0 at acpi0: PWRB
aibs0 at acpi0
aibs0: FSIF: misformed package: 3/5, assume 5
bios0: ROM list: 0xc0000/0x10000 0xd0000/0x4000! 0xd4000/0x4800
pci0 at mainbus0 bus 0: configuration mode 1 (bios)
"NVIDIA C51 Host" rev 0xa2 at pci0 dev 0 function 0 not configured
"NVIDIA C51 Memory" rev 0xa2 at pci0 dev 0 function 1 not configured
"NVIDIA C51 Memory" rev 0xa2 at pci0 dev 0 function 2 not configured
"NVIDIA C51 Memory" rev 0xa2 at pci0 dev 0 function 3 not configured
"NVIDIA C51 Memory" rev 0xa2 at pci0 dev 0 function 4 not configured
"NVIDIA C51 Memory" rev 0xa2 at pci0 dev 0 function 5 not configured
"NVIDIA C51 Memory" rev 0xa2 at pci0 dev 0 function 6 not configured
"NVIDIA C51 Memory" rev 0xa2 at pci0 dev 0 function 7 not configured
ppb0 at pci0 dev 2 function 0 "NVIDIA C51 PCIE" rev 0xa1
pci1 at ppb0 bus 1
ppb1 at pci0 dev 3 function 0 "NVIDIA C51 PCIE" rev 0xa1
pci2 at ppb1 bus 2
ppb2 at pci0 dev 4 function 0 "NVIDIA C51 PCIE" rev 0xa1
pci3 at ppb2 bus 3
vga1 at pci3 dev 0 function 0 "ATI Radeon X300" rev 0x00
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
radeondrm0 at vga1: apic 2 int 16 (irq 11)
drm0 at radeondrm0
"ATI Radeon X300 Sec" rev 0x00 at pci3 dev 0 function 1 not configured
"NVIDIA MCP51 Host" rev 0xa2 at pci0 dev 9 function 0 not configured
pcib0 at pci0 dev 10 function 0 "NVIDIA MCP51 ISA" rev 0xa3
nviic0 at pci0 dev 10 function 1 "NVIDIA MCP51 SMBus" rev 0xa3
iic0 at nviic0
spdmem0 at iic0 addr 0x50: 512MB DDR2 SDRAM non-parity PC2-6400CL3
spdmem1 at iic0 addr 0x51: 512MB DDR2 SDRAM non-parity PC2-6400CL3
iic1 at nviic0
"NVIDIA MCP51 Memory" rev 0xa3 at pci0 dev 10 function 2 not
configured
ohci0 at pci0 dev 11 function 0 "NVIDIA MCP51 USB" rev 0xa3: apic 2
int 20 (irq 5), version 1.0, legacy support
ehci0 at pci0 dev 11 function 1 "NVIDIA MCP51 USB" rev 0xa3: apic 2
int 20 (irq 10)
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "NVIDIA EHCI root hub" rev 2.00/1.00 addr 1
pciide0 at pci0 dev 13 function 0 "NVIDIA MCP51 IDE" rev 0xa1: DMA,
channel 0 configured to compatibility, channel 1 configured to
compatibility
wd0 at pciide0 channel 0 drive 0: <Maxtor 6L250R0>
wd0: 16-sector PIO, LBA48, 239372MB, 490234752 sectors
atapiscsi0 at pciide0 channel 0 drive 1
scsibus0 at atapiscsi0: 2 targets
cd0 at scsibus0 targ 0 lun 0: <SONY, DVD RW DRU-510A, 1.0a> ATAPI
5/cdrom removable
wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 6
cd0(pciide0:0:1): using PIO mode 4, Ultra-DMA mode 2
pciide0: channel 1 disabled (no drives)
pciide1 at pci0 dev 14 function 0 "NVIDIA MCP51 SATA" rev 0xa1: DMA
pciide1: using apic 2 int 20 (irq 5) for native-PCI interrupt
wd1 at pciide1 channel 0 drive 0: <WDC WD10EACS-00ZJB0>
wd1: 16-sector PIO, LBA48, 953869MB, 1953525168 sectors
wd1(pciide1:0:0): using PIO mode 4, Ultra-DMA mode 6
wd2 at pciide1 channel 1 drive 0: <SAMSUNG HD103UJ>
wd2: 16-sector PIO, LBA48, 953869MB, 1953525168 sectors
wd2(pciide1:1:0): using PIO mode 4, Ultra-DMA mode 6
pciide2 at pci0 dev 15 function 0 "NVIDIA MCP51 SATA" rev 0xa1: DMA
pciide2: using apic 2 int 20 (irq 5) for native-PCI interrupt
wd3 at pciide2 channel 0 drive 0: <ST3500641AS>
wd3: 16-sector PIO, LBA48, 476938MB, 976771055 sectors
wd3(pciide2:0:0): using PIO mode 4, Ultra-DMA mode 6
wd4 at pciide2 channel 1 drive 0: <ST3500641AS>
wd4: 16-sector PIO, LBA48, 476940MB, 976773168 sectors
wd4(pciide2:1:0): using PIO mode 4, Ultra-DMA mode 6
ppb3 at pci0 dev 16 function 0 "NVIDIA MCP51 PCI-PCI" rev 0xa2
pci4 at ppb3 bus 4
skc0 at pci4 dev 8 function 0 "D-Link DGE-530T B1" rev 0x11, Yukon
Lite (0x9): apic 2 int 16 (irq 7)
sk0 at skc0 port A: address 00:17:9a:84:62:9c
eephy0 at sk0 phy 0: 88E1011 Gigabit PHY, rev. 5
pciide3 at pci4 dev 9 function 0 "CMD Technology SiI3512 SATA" rev
0x01: DMA
pciide3: using apic 2 int 17 (irq 5) for native-PCI interrupt
pciide3: port 1: device present, speed: 1.5Gb/s
wd5 at pciide3 channel 1 drive 0: <SAMSUNG HD103UJ>
wd5: 16-sector PIO, LBA48, 953869MB, 1953525168 sectors
wd5(pciide3:1:0): using BIOS timings, Ultra-DMA mode 7
pchb0 at pci0 dev 24 function 0 "AMD AMD64 0Fh HyperTransport" rev
0x00
pchb1 at pci0 dev 24 function 1 "AMD AMD64 0Fh Address Map" rev 0x00
pchb2 at pci0 dev 24 function 2 "AMD AMD64 0Fh DRAM Cfg" rev 0x00
kate0 at pci0 dev 24 function 3 "AMD AMD64 0Fh Misc Cfg" rev 0x00:
core rev BH-F2
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com0: console
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
it0 at isa0 port 0x2e/2: IT8716F rev 0, EC port 0x290
npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16
usb1 at ohci0: USB revision 1.0
uhub1 at usb1 "NVIDIA OHCI root hub" rev 1.00/1.00 addr 1
mtrr: Pentium Pro MTRR support
umass0 at uhub0 port 7 configuration 1 interface 0 "Atech Flash
PRO-Gear XM-4U" rev 2.00/1.00 addr 2
umass0: using SCSI over Bulk-Only
scsibus1 at umass0: 2 targets, initiator 0
sd0 at scsibus1 targ 1 lun 0: <USB2.0, CardReader CF, 0100> SCSI0
0/direct removable
sd0: drive offline
sd1 at scsibus1 targ 1 lun 1: <USB2.0, CardReader SM XD, 0100> SCSI0
0/direct removable
sd1: drive offline
sd2 at scsibus1 targ 1 lun 2: <USB2.0, CardReader MS, 0100> SCSI0
0/direct removable
sd2: drive offline
sd3 at scsibus1 targ 1 lun 3: <USB2.0, CardReader SD, 0100> SCSI0
0/direct removable
sd3: drive offline
uhub2 at uhub1 port 5 "BTC Multimedia USB Keyboard" rev 1.10/0.01 addr
2
uhidev0 at uhub2 port 1 configuration 1 interface 0 "BTC Multimedia
USB Keyboard" rev 1.10/0.01 addr 3
uhidev0: iclass 3/1
ukbd0 at uhidev0: 8 modifier keys, 6 key codes
wskbd1 at ukbd0 mux 1
wskbd1: connecting to wsdisplay0
softraid0 at root
root on wd0a swap on wd0b dump on wd0b
softraid0: sd4 was not shutdown properly
softraid0: sd4 was not shutdown properly
scsibus2 at softraid0: 1 targets
sd4 at scsibus2 targ 0 lun 0: <OPENBSD, SR CRYPTO, 004> SCSI2 0/direct
fixed
sd4: 953868MB, 512 bytes/sec, 1953523649 sec total
softraid0: sd5 was not shutdown properly
softraid0: sd5 was not shutdown properly
scsibus3 at softraid0: 1 targets
sd5 at scsibus3 targ 0 lun 0: <OPENBSD, SR CRYPTO, 004> SCSI2 0/direct
fixed
sd5: 953868MB, 512 bytes/sec, 1953523649 sec total
softraid0: trying to bring up sd6 degraded
scsibus4 at softraid0: 1 targets
sd6 at scsibus4 targ 0 lun 0: <OPENBSD, SR RAID 1, 004> SCSI2 0/direct
fixed
sd6: 953859MB, 512 bytes/sec, 1953503474 sec total
softraid0: roaming device -> sd5d
<6>arp info overwritten for 10.10.15.106 by 00:26:bb:72:c0:e5 on sk0
<6>arp info overwritten for 10.10.15.106 by 00:26:bb:72:c0:e5 on sk0
<6>arp info overwritten for 10.10.15.106 by 00:16:cb:a7:ca:3c on sk0
uvm_fault(0xd0a05960, 0x0, 0, 1) -> e
kernel: page fault trap, code=0
Stopped at tcp_usrreq+0xbd: movzwl 0x68(%edx),%ecx
ddb> tcp_usrreq(d6b40014,8,0,80,0) at tcp_usrreq+0xbd
soreceive(d6b40014,dc466e20,dc466dfc,dc466e24,0) at soreceive+0x624
nfsrv_rcv(d6b40014,d1f35cc0,1,dc466e54,d0202fe5) at nfsrv_rcv+0x138
soisdisconnected(d6b40014,0,d6b40014,d6b417dc,d048be50) at
soisdisconnected+0x66
tcp_close(d6b417dc,3c,0,0,d0ac1ad4) at tcp_close+0x97
tcp_timer_rexmt(d6b417dc,dc466f00,d03cbe36,dc466ef4,d1d949e0) at
tcp_timer_rexmt+0xa1
softclock(0,dc466f10,d07672d1,0,d0201fc6) at softclock+0x225
softintr_dispatch(0) at softintr_dispatch+0x4f
Xsoftclock() at Xsoftclock+0x12
ddb> show panic
the kernel did not panic
ddb> boot dump
syncing disks... panic: cpu_switch1 0xd6d66278
Stopped at Debugger+0x4: popl %ebp
RUN AT LEAST 'trace' AND 'ps' AND INCLUDE OUTPUT WHEN REPORTING THIS
PANIC!
DO NOT EVEN BOTHER REPORTING THIS WITHOUT INCLUDING THAT INFORMATION!
ddb> show panic
cpu_switch1 0xd6d66278
ddb>
