I have an AMD based system that I frequently find at the ddb prompt with messages about tcp_usrreq and nfsrv_recv (see trace info below). I haven't discerned a specific pattern or time between occurrences. It ranges from happening twice in the same day to going for 3+ days without incident.
When I review the dmesg I usually see a message like: <6>arp info overwritten for 10.10.15.106 by 00:26:bb:72:c0:e5 on sk0 The server's ip is 10.10.15.10. I've wondered whether they're related. The server exports user home directories to clients (see below) but all the systems that access the server have specified IP addresses in the .30 - .50 range. Everything above .99 is reserved for ephemeral clients. Does anyone have any ideas or pointers to resources or articles that might help resolve this? This may be an unrelated problem, but I almost always get a kernel panic when I type `boot <command>` at the ddb prompt. I've included it at the bottom of the message. Attached are the initial trace, ps, show registers, dmesg and the results from `boot dump` (in this case) followed by the kernel panic. Cheers and thanks for any help, --Aaron ---------------------------------------------------------------------- Hardware: AMD Athlon(tm) 64 X2 Dual Core Processor 3800+ 1GB RAM 1xPATA boot drive 4xSATA drives using stacked softraid to create two mirrored drives with encryption OS Version: OpenBSD 4.8 i386 (no patches applied yet) Services: nfsd dovecot fetchmail procmail sendmail nfsd exports home directories to Mac OS X 10.6 clients over IPv4. dhcp and dns are handled by another OpenBSD 4.8 server. ---------------------------------------------------------------------- ddb> trace tcp_usrreq(d6b40014,8,0,80,0) at tcp_usrreq+0xbd soreceive(d6b40014,dc466e20,dc466dfc,dc466e24,0) at soreceive+0x624 nfsrv_rcv(d6b40014,d1f35cc0,1,dc466e54,d0202fe5) at nfsrv_rcv+0x138 soisdisconnected(d6b40014,0,d6b40014,d6b417dc,d048be50) at soisdisconnected+0x66 tcp_close(d6b417dc,3c,0,0,d0ac1ad4) at tcp_close+0x97 tcp_timer_rexmt(d6b417dc,dc466f00,d03cbe36,dc466ef4,d1d949e0) at tcp_timer_rexmt+0xa1 softclock(0,dc466f10,d07672d1,0,d0201fc6) at softclock+0x225 softintr_dispatch(0) at softintr_dispatch+0x4f Xsoftclock() at Xsoftclock+0x12 --- interrupt --- cpu_idle_cycle(d0ac1aa0) at cpu_idle_cycle+0xf Bad frame pointer: 0xd0b75e48 ddb> ps PID PPID PGRP UID S FLAGS WAIT COMMAND 13697 20622 20622 518 3 0x4180 kqread imap-login 7751 20622 20622 1000 3 0x44180 kqread imap 26824 20622 20622 1000 3 0x44180 kqread imap 19716 20622 20622 2000 3 0x44180 kqread imap 14259 21940 14259 1000 3 0x4080 ttyin ksh 21940 8247 8247 1000 3 0x180 select sshd 30780 20622 20622 2000 3 0x44180 kqread imap 13379 20622 20622 518 3 0x4180 kqread imap-login 16574 20622 20622 518 3 0x4180 kqread imap-login 8247 28378 8247 0 3 0x4180 netio sshd 22687 20622 20622 1000 3 0x44180 kqread imap 18020 20622 20622 1000 3 0x44180 kqread imap 29759 1 29759 0 3 0x4080 ttyin getty 21313 20622 20622 518 3 0x4180 kqread pop3-login 18205 20622 20622 518 3 0x4180 kqread pop3-login 146 20622 20622 518 3 0x4180 kqread pop3-login 5104 1 5104 0 3 0x40180 select sendmail 21349 1 21349 0 3 0x4080 ttyin getty 18697 1 18697 0 3 0x4080 ttyin getty 11800 1 11800 0 3 0x4080 ttyin getty 24887 1 24887 0 3 0x4080 ttyin getty 7814 1 7814 0 3 0x4080 ttyin getty 21068 1 21068 0 3 0x80 select cron 14352 20622 20622 0 3 0x4180 kqread dovecot-auth 20622 1 20622 0 3 0x80 kqread dovecot 29601 1 29601 0 3 0x180 select inetd 28378 1 28378 0 3 0x80 select sshd 19159 26558 27956 83 3 0x180 poll ntpd 26558 27956 27956 83 3 0x180 poll ntpd 27956 1 27956 0 3 0x80 poll ntpd 17993 10933 10933 0 3 0 nfsndlck nfsd 16904 10933 10933 0 3 0 nfsndlck nfsd 19532 10933 10933 0 2 0x80 nfsd 18407 10933 10933 0 3 0 nfsndlck nfsd 10933 1 10933 0 3 0x80 netcon nfsd 12841 1 12841 0 3 0x80 select mountd 22181 1 22181 28 3 0x180 poll portmap 2998 12777 12777 74 3 0x180 bpf pflogd 12777 1 12777 0 3 0x80 netio pflogd 32090 16237 16237 73 2 0x180 syslogd 16237 1 16237 0 3 0x88 netio syslogd 22583 1 22583 77 3 0x180 poll dhclient 3071 1 20000 0 3 0x80 poll dhclient 15 0 0 0 3 0x100200 bored crypto 14 0 0 0 3 0x100200 aiodoned aiodoned 13 0 0 0 3 0x100200 syncer update 12 0 0 0 3 0x100200 cleaner cleaner 11 0 0 0 3 0x100200 reaper reaper 10 0 0 0 3 0x100200 pgdaemon pagedaemon 9 0 0 0 3 0x100200 pftm pfpurge 8 0 0 0 3 0x100200 usbevt usb1 7 0 0 0 3 0x100200 usbtsk usbtask 6 0 0 0 3 0x100200 usbevt usb0 5 0 0 0 3 0x100200 acpi0 acpi0 4 0 0 0 3 0x100200 bored syswq * 3 0 0 0 7 0x40100200 idle0 2 0 0 0 3 0x100200 kmalloc kmthread 1 0 1 0 3 0x4080 wait init 0 -1 0 0 3 0x80200 scheduler swapper ddb> show registers ds 0x10 es 0x10 fs 0x58 gs 0x10 edi 0xd6d66278 end+0x62a1b74 esi 0x80 ebp 0xdc466d64 end+0xb9a2660 ebx 0 edx 0 ecx 0xd6b40014 end+0x607b910 eax 0x40 eip 0xd048ce4d tcp_usrreq+0xbd cs 0x50 eflags 0x10283 esp 0xdc466d1c end+0xb9a2618 ss 0xdc460010 end+0xb99b90c tcp_usrreq+0xbd: movzwl 0x68(%edx),%ecx ddb> dmesg OpenBSD 4.8 (GENERIC) #136: Mon Aug 16 09:06:23 MDT 2010 dera...@i386.openbsd.org:/usr/src/sys/arch/i386/compile/GENERIC cpu0: AMD Athlon(tm) 64 X2 Dual Core Processor 3800+ ("AuthenticAMD" 686-class, 512KB L2 cache) 2.01 GHz cpu0:FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT ,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,CX16 real mem = 1072193536 (1022MB) avail mem = 1044692992 (996MB) mainbus0 at root bios0 at mainbus0: AT/286+ BIOS, date 11/29/06, BIOS32 rev. 0 @ 0xf1fc0, SMBIOS rev. 2.4 @ 0xf0000 (67 entries) bios0: vendor Phoenix Technologies, LTD version "ASUS M2NPV-VM ACPI BIOS Revision 0603" date 11/29/2006 bios0: ASUSTek Computer INC. M2NPV-VM acpi0 at bios0: rev 2 acpi0: sleep states S0 S1 S3 S4 S5 acpi0: tables DSDT FACP HPET MCFG APIC acpi0: wakeup devices HUB0(S5) XVRA(S5) XVRB(S5) XVRC(S5) USB0(S4) USB2(S4) AZA D(S5) MMAC(S5) MMCI(S5) UAR1(S5) UAR2(S5) PS2M(S4) PS2K(S4) acpitimer0 at acpi0: 3579545 Hz, 24 bits acpihpet0 at acpi0: 25000000 Hz acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: apic clock running at 200MHz cpu at mainbus0: not configured ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 11, 24 pins ioapic0: misconfigured as apic 0, remapped to apid 2 acpiprt0 at acpi0: bus 0 (PCI0) acpiprt1 at acpi0: bus 4 (HUB0) acpicpu0 at acpi0 acpitz0 at acpi0: critical temperature 75 degC acpibtn0 at acpi0: PWRB aibs0 at acpi0 aibs0: FSIF: misformed package: 3/5, assume 5 bios0: ROM list: 0xc0000/0x10000 0xd0000/0x4000! 0xd4000/0x4800 pci0 at mainbus0 bus 0: configuration mode 1 (bios) "NVIDIA C51 Host" rev 0xa2 at pci0 dev 0 function 0 not configured "NVIDIA C51 Memory" rev 0xa2 at pci0 dev 0 function 1 not configured "NVIDIA C51 Memory" rev 0xa2 at pci0 dev 0 function 2 not configured "NVIDIA C51 Memory" rev 0xa2 at pci0 dev 0 function 3 not configured "NVIDIA C51 Memory" rev 0xa2 at pci0 dev 0 function 4 not configured "NVIDIA C51 Memory" rev 0xa2 at pci0 dev 0 function 5 not configured "NVIDIA C51 Memory" rev 0xa2 at pci0 dev 0 function 6 not configured "NVIDIA C51 Memory" rev 0xa2 at pci0 dev 0 function 7 not configured ppb0 at pci0 dev 2 function 0 "NVIDIA C51 PCIE" rev 0xa1 pci1 at ppb0 bus 1 ppb1 at pci0 dev 3 function 0 "NVIDIA C51 PCIE" rev 0xa1 pci2 at ppb1 bus 2 ppb2 at pci0 dev 4 function 0 "NVIDIA C51 PCIE" rev 0xa1 pci3 at ppb2 bus 3 vga1 at pci3 dev 0 function 0 "ATI Radeon X300" rev 0x00 wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) wsdisplay0: screen 1-5 added (80x25, vt100 emulation) radeondrm0 at vga1: apic 2 int 16 (irq 11) drm0 at radeondrm0 "ATI Radeon X300 Sec" rev 0x00 at pci3 dev 0 function 1 not configured "NVIDIA MCP51 Host" rev 0xa2 at pci0 dev 9 function 0 not configured pcib0 at pci0 dev 10 function 0 "NVIDIA MCP51 ISA" rev 0xa3 nviic0 at pci0 dev 10 function 1 "NVIDIA MCP51 SMBus" rev 0xa3 iic0 at nviic0 spdmem0 at iic0 addr 0x50: 512MB DDR2 SDRAM non-parity PC2-6400CL3 spdmem1 at iic0 addr 0x51: 512MB DDR2 SDRAM non-parity PC2-6400CL3 iic1 at nviic0 "NVIDIA MCP51 Memory" rev 0xa3 at pci0 dev 10 function 2 not configured ohci0 at pci0 dev 11 function 0 "NVIDIA MCP51 USB" rev 0xa3: apic 2 int 20 (irq 5), version 1.0, legacy support ehci0 at pci0 dev 11 function 1 "NVIDIA MCP51 USB" rev 0xa3: apic 2 int 20 (irq 10) usb0 at ehci0: USB revision 2.0 uhub0 at usb0 "NVIDIA EHCI root hub" rev 2.00/1.00 addr 1 pciide0 at pci0 dev 13 function 0 "NVIDIA MCP51 IDE" rev 0xa1: DMA, channel 0 configured to compatibility, channel 1 configured to compatibility wd0 at pciide0 channel 0 drive 0: <Maxtor 6L250R0> wd0: 16-sector PIO, LBA48, 239372MB, 490234752 sectors atapiscsi0 at pciide0 channel 0 drive 1 scsibus0 at atapiscsi0: 2 targets cd0 at scsibus0 targ 0 lun 0: <SONY, DVD RW DRU-510A, 1.0a> ATAPI 5/cdrom removable wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 6 cd0(pciide0:0:1): using PIO mode 4, Ultra-DMA mode 2 pciide0: channel 1 disabled (no drives) pciide1 at pci0 dev 14 function 0 "NVIDIA MCP51 SATA" rev 0xa1: DMA pciide1: using apic 2 int 20 (irq 5) for native-PCI interrupt wd1 at pciide1 channel 0 drive 0: <WDC WD10EACS-00ZJB0> wd1: 16-sector PIO, LBA48, 953869MB, 1953525168 sectors wd1(pciide1:0:0): using PIO mode 4, Ultra-DMA mode 6 wd2 at pciide1 channel 1 drive 0: <SAMSUNG HD103UJ> wd2: 16-sector PIO, LBA48, 953869MB, 1953525168 sectors wd2(pciide1:1:0): using PIO mode 4, Ultra-DMA mode 6 pciide2 at pci0 dev 15 function 0 "NVIDIA MCP51 SATA" rev 0xa1: DMA pciide2: using apic 2 int 20 (irq 5) for native-PCI interrupt wd3 at pciide2 channel 0 drive 0: <ST3500641AS> wd3: 16-sector PIO, LBA48, 476938MB, 976771055 sectors wd3(pciide2:0:0): using PIO mode 4, Ultra-DMA mode 6 wd4 at pciide2 channel 1 drive 0: <ST3500641AS> wd4: 16-sector PIO, LBA48, 476940MB, 976773168 sectors wd4(pciide2:1:0): using PIO mode 4, Ultra-DMA mode 6 ppb3 at pci0 dev 16 function 0 "NVIDIA MCP51 PCI-PCI" rev 0xa2 pci4 at ppb3 bus 4 skc0 at pci4 dev 8 function 0 "D-Link DGE-530T B1" rev 0x11, Yukon Lite (0x9): apic 2 int 16 (irq 7) sk0 at skc0 port A: address 00:17:9a:84:62:9c eephy0 at sk0 phy 0: 88E1011 Gigabit PHY, rev. 5 pciide3 at pci4 dev 9 function 0 "CMD Technology SiI3512 SATA" rev 0x01: DMA pciide3: using apic 2 int 17 (irq 5) for native-PCI interrupt pciide3: port 1: device present, speed: 1.5Gb/s wd5 at pciide3 channel 1 drive 0: <SAMSUNG HD103UJ> wd5: 16-sector PIO, LBA48, 953869MB, 1953525168 sectors wd5(pciide3:1:0): using BIOS timings, Ultra-DMA mode 7 pchb0 at pci0 dev 24 function 0 "AMD AMD64 0Fh HyperTransport" rev 0x00 pchb1 at pci0 dev 24 function 1 "AMD AMD64 0Fh Address Map" rev 0x00 pchb2 at pci0 dev 24 function 2 "AMD AMD64 0Fh DRAM Cfg" rev 0x00 kate0 at pci0 dev 24 function 3 "AMD AMD64 0Fh Misc Cfg" rev 0x00: core rev BH-F2 isa0 at pcib0 isadma0 at isa0 com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo com0: console com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo pckbc0 at isa0 port 0x60/5 pckbd0 at pckbc0 (kbd slot) pckbc0: using irq 1 for kbd slot wskbd0 at pckbd0: console keyboard, using wsdisplay0 pcppi0 at isa0 port 0x61 spkr0 at pcppi0 it0 at isa0 port 0x2e/2: IT8716F rev 0, EC port 0x290 npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16 usb1 at ohci0: USB revision 1.0 uhub1 at usb1 "NVIDIA OHCI root hub" rev 1.00/1.00 addr 1 mtrr: Pentium Pro MTRR support umass0 at uhub0 port 7 configuration 1 interface 0 "Atech Flash PRO-Gear XM-4U" rev 2.00/1.00 addr 2 umass0: using SCSI over Bulk-Only scsibus1 at umass0: 2 targets, initiator 0 sd0 at scsibus1 targ 1 lun 0: <USB2.0, CardReader CF, 0100> SCSI0 0/direct removable sd0: drive offline sd1 at scsibus1 targ 1 lun 1: <USB2.0, CardReader SM XD, 0100> SCSI0 0/direct removable sd1: drive offline sd2 at scsibus1 targ 1 lun 2: <USB2.0, CardReader MS, 0100> SCSI0 0/direct removable sd2: drive offline sd3 at scsibus1 targ 1 lun 3: <USB2.0, CardReader SD, 0100> SCSI0 0/direct removable sd3: drive offline uhub2 at uhub1 port 5 "BTC Multimedia USB Keyboard" rev 1.10/0.01 addr 2 uhidev0 at uhub2 port 1 configuration 1 interface 0 "BTC Multimedia USB Keyboard" rev 1.10/0.01 addr 3 uhidev0: iclass 3/1 ukbd0 at uhidev0: 8 modifier keys, 6 key codes wskbd1 at ukbd0 mux 1 wskbd1: connecting to wsdisplay0 softraid0 at root root on wd0a swap on wd0b dump on wd0b softraid0: sd4 was not shutdown properly softraid0: sd4 was not shutdown properly scsibus2 at softraid0: 1 targets sd4 at scsibus2 targ 0 lun 0: <OPENBSD, SR CRYPTO, 004> SCSI2 0/direct fixed sd4: 953868MB, 512 bytes/sec, 1953523649 sec total softraid0: sd5 was not shutdown properly softraid0: sd5 was not shutdown properly scsibus3 at softraid0: 1 targets sd5 at scsibus3 targ 0 lun 0: <OPENBSD, SR CRYPTO, 004> SCSI2 0/direct fixed sd5: 953868MB, 512 bytes/sec, 1953523649 sec total softraid0: trying to bring up sd6 degraded scsibus4 at softraid0: 1 targets sd6 at scsibus4 targ 0 lun 0: <OPENBSD, SR RAID 1, 004> SCSI2 0/direct fixed sd6: 953859MB, 512 bytes/sec, 1953503474 sec total softraid0: roaming device -> sd5d <6>arp info overwritten for 10.10.15.106 by 00:26:bb:72:c0:e5 on sk0 <6>arp info overwritten for 10.10.15.106 by 00:26:bb:72:c0:e5 on sk0 <6>arp info overwritten for 10.10.15.106 by 00:16:cb:a7:ca:3c on sk0 uvm_fault(0xd0a05960, 0x0, 0, 1) -> e kernel: page fault trap, code=0 Stopped at tcp_usrreq+0xbd: movzwl 0x68(%edx),%ecx ddb> tcp_usrreq(d6b40014,8,0,80,0) at tcp_usrreq+0xbd soreceive(d6b40014,dc466e20,dc466dfc,dc466e24,0) at soreceive+0x624 nfsrv_rcv(d6b40014,d1f35cc0,1,dc466e54,d0202fe5) at nfsrv_rcv+0x138 soisdisconnected(d6b40014,0,d6b40014,d6b417dc,d048be50) at soisdisconnected+0x66 tcp_close(d6b417dc,3c,0,0,d0ac1ad4) at tcp_close+0x97 tcp_timer_rexmt(d6b417dc,dc466f00,d03cbe36,dc466ef4,d1d949e0) at tcp_timer_rexmt+0xa1 softclock(0,dc466f10,d07672d1,0,d0201fc6) at softclock+0x225 softintr_dispatch(0) at softintr_dispatch+0x4f Xsoftclock() at Xsoftclock+0x12 ddb> show panic the kernel did not panic ddb> boot dump syncing disks... panic: cpu_switch1 0xd6d66278 Stopped at Debugger+0x4: popl %ebp RUN AT LEAST 'trace' AND 'ps' AND INCLUDE OUTPUT WHEN REPORTING THIS PANIC! DO NOT EVEN BOTHER REPORTING THIS WITHOUT INCLUDING THAT INFORMATION! ddb> show panic cpu_switch1 0xd6d66278 ddb>