Hi misc@, I have a Dell PowerEdge M600 machine running OpenBSD 5.3 which causes frequent problems - once about every few days vlan interfaces stop working.
Ifconfig reports them being up the whole time, but when trying to ping anything in the given vlan, the ping fails (this also applies to the loopback address on the interface - it doesn't reply as well), as if no packets were transmitted on it. This applies to all packets on the interface, not just ICMP ones. This doesn't seem to be a PF issue, since disabling it doesn't help, and non-vlan interfaces (like trunk0 with trunkproto failover) continue to work. Putting the interface down and up, destroying and creating it or running /etc/netstart doesn't help - a reboot is required, after which the machine works correctly again for some time. I checked the logs and dmesg and found there nothing of any value - no new messages around the time of the crash. As this is a blade machine, most hardware issues (like memory problems) would be reported by the chassis, and it reports none. I have a couple other M600 machines running OpenBSD, with the same network cards, and only once this problem did occur on another machine - a slave for the problematic one, running pfsync, sasync and acting as a carp backup. Could anyone provide me with some tips on how to debug the problem further? Turn on some kernel debug mode, make use of the kernel debugger? I'm at my wits end here. If there is any information that I could provide to help solve the problem, please don't hesitate to ask. The dmesg is attached below: OpenBSD 5.3 (GENERIC.MP) #62: Tue Mar 12 18:21:20 MDT 2013 dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 4292550656 (4093MB) avail mem = 4155789312 (3963MB) mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.5 @ 0xbff9c000 (50 entries) bios0: vendor Dell Inc. version "2.4.0" date 10/29/2010 bios0: Dell Inc. PowerEdge M600 acpi0 at bios0: rev 2 acpi0: sleep states S0 S4 S5 acpi0: tables DSDT FACP APIC SPCR HPET MCFG WD__ SLIC ERST HEST BERT EINJ SSDT SSDT SSDT SSDT SSDT acpi0: wakeup devices PCI0(S5) acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Xeon(R) CPU E5420 @ 2.50GHz, 2494.11 MHz cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,XSAVE,NXE,LONG,LAHF,PERF cpu0: 6MB 64b/line 16-way L2 cache cpu0: apic clock running at 332MHz cpu1 at mainbus0: apid 2 (application processor) cpu1: Intel(R) Xeon(R) CPU E5420 @ 2.50GHz, 2493.75 MHz cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,XSAVE,NXE,LONG,LAHF,PERF cpu1: 6MB 64b/line 16-way L2 cache cpu2 at mainbus0: apid 1 (application processor) cpu2: Intel(R) Xeon(R) CPU E5420 @ 2.50GHz, 2493.75 MHz cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,XSAVE,NXE,LONG,LAHF,PERF cpu2: 6MB 64b/line 16-way L2 cache cpu3 at mainbus0: apid 3 (application processor) cpu3: Intel(R) Xeon(R) CPU E5420 @ 2.50GHz, 2493.75 MHz cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,XSAVE,NXE,LONG,LAHF,PERF cpu3: 6MB 64b/line 16-way L2 cache ioapic0 at mainbus0: apid 4 pa 0xfec00000, version 20, 24 pins ioapic0: misconfigured as apic 0, remapped to apid 4 acpihpet0 at acpi0: 14318179 Hz acpimcfg0 at acpi0 addr 0xe0000000, bus 0-255 acpiprt0 at acpi0: bus 0 (PCI0) acpiprt1 at acpi0: bus 1 (PEX2) acpiprt2 at acpi0: bus 2 (UPST) acpiprt3 at acpi0: bus 3 (DWN1) acpiprt4 at acpi0: bus 5 (DWN2) acpiprt5 at acpi0: bus 8 (PEX3) acpiprt6 at acpi0: bus 9 (PEX4) acpiprt7 at acpi0: bus 11 (PEX5) acpiprt8 at acpi0: bus 13 (PEX6) acpiprt9 at acpi0: bus 14 (PEX7) acpiprt10 at acpi0: bus 15 (COMP) acpicpu0 at acpi0: C3, C2, C1, PSS acpicpu1 at acpi0: C3, C2, C1, PSS acpicpu2 at acpi0: C3, C2, C1, PSS acpicpu3 at acpi0: C3, C2, C1, PSS ipmi at mainbus0 not configured cpu0: Enhanced SpeedStep 2494 MHz: speeds: 2500, 2000 MHz pci0 at mainbus0 bus 0 pchb0 at pci0 dev 0 function 0 "Intel 5000P Host" rev 0x92 ppb0 at pci0 dev 2 function 0 "Intel 5000 PCIE" rev 0x92 pci1 at ppb0 bus 1 ppb1 at pci1 dev 0 function 0 "Intel 6321ESB PCIE" rev 0x01 pci2 at ppb1 bus 2 ppb2 at pci2 dev 0 function 0 "Intel 6321ESB PCIE" rev 0x01 pci3 at ppb2 bus 3 ppb3 at pci3 dev 0 function 0 "ServerWorks PCIE-PCIX" rev 0xc3 pci4 at ppb3 bus 4 bnx0 at pci4 dev 0 function 0 "Broadcom BCM5708S" rev 0x12: apic 4 int 17 ppb4 at pci2 dev 1 function 0 "Intel 6321ESB PCIE" rev 0x01 pci5 at ppb4 bus 5 ppb5 at pci5 dev 0 function 0 "ServerWorks PCIE-PCIX" rev 0xc3 pci6 at ppb5 bus 6 bnx1 at pci6 dev 0 function 0 "Broadcom BCM5708S" rev 0x12: apic 4 int 18 ppb6 at pci1 dev 0 function 3 "Intel 6321ESB PCIE-PCIX" rev 0x01 pci7 at ppb6 bus 7 ppb7 at pci0 dev 3 function 0 "Intel 5000 PCIE" rev 0x92 pci8 at ppb7 bus 8 ppb8 at pci0 dev 4 function 0 "Intel 5000 PCIE" rev 0x92: msi pci9 at ppb8 bus 9 ppb9 at pci9 dev 0 function 0 "ServerWorks PCIE-PCIX" rev 0xc3 pci10 at ppb9 bus 10 bnx2 at pci10 dev 0 function 0 "Broadcom BCM5708S" rev 0x12: apic 4 int 18 ppb10 at pci0 dev 5 function 0 "Intel 5000 PCIE" rev 0x92: msi pci11 at ppb10 bus 11 ppb11 at pci11 dev 0 function 0 "ServerWorks PCIE-PCIX" rev 0xc3 pci12 at ppb11 bus 12 bnx3 at pci12 dev 0 function 0 "Broadcom BCM5708S" rev 0x12: apic 4 int 19 ppb12 at pci0 dev 6 function 0 "Intel 5000 PCIE x8" rev 0x92: msi pci13 at ppb12 bus 13 ppb13 at pci0 dev 7 function 0 "Intel 5000 PCIE" rev 0x92: msi pci14 at ppb13 bus 14 pchb1 at pci0 dev 16 function 0 "Intel 5000 Error Reporting" rev 0x92 pchb2 at pci0 dev 16 function 1 "Intel 5000 Error Reporting" rev 0x92 pchb3 at pci0 dev 16 function 2 "Intel 5000 Error Reporting" rev 0x92 pchb4 at pci0 dev 17 function 0 "Intel 5000 Reserved" rev 0x92 pchb5 at pci0 dev 19 function 0 "Intel 5000 Reserved" rev 0x92 pchb6 at pci0 dev 21 function 0 "Intel 5000 FBD" rev 0x92 pchb7 at pci0 dev 22 function 0 "Intel 5000 FBD" rev 0x92 uhci0 at pci0 dev 29 function 0 "Intel 6321ESB USB" rev 0x09: apic 4 int 21 uhci1 at pci0 dev 29 function 1 "Intel 6321ESB USB" rev 0x09: apic 4 int 20 uhci2 at pci0 dev 29 function 2 "Intel 6321ESB USB" rev 0x09: apic 4 int 21 ehci0 at pci0 dev 29 function 7 "Intel 6321ESB USB" rev 0x09: apic 4 int 21 usb0 at ehci0: USB revision 2.0 uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1 ppb14 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0xd9 pci15 at ppb14 bus 15 vga1 at pci15 dev 4 function 0 "ATI ES1000" rev 0x02 wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) wsdisplay0: screen 1-5 added (80x25, vt100 emulation) radeondrm0 at vga1: apic 4 int 19 drm0 at radeondrm0 pcib0 at pci0 dev 31 function 0 "Intel 6321ESB LPC" rev 0x09 pciide0 at pci0 dev 31 function 2 "Intel 6321ESB SATA" rev 0x09: DMA, channel 0 configured to native-PCI, channel 1 configured to native-PCI pciide0: using apic 4 int 23 for native-PCI interrupt wd0 at pciide0 channel 0 drive 0: <FUJITSU MHZ2160BK G2> wd0: 16-sector PIO, LBA48, 152627MB, 312581808 sectors wd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 5 usb1 at uhci0: USB revision 1.0 uhub1 at usb1 "Intel UHCI root hub" rev 1.00/1.00 addr 1 usb2 at uhci1: USB revision 1.0 uhub2 at usb2 "Intel UHCI root hub" rev 1.00/1.00 addr 1 usb3 at uhci2: USB revision 1.0 uhub3 at usb3 "Intel UHCI root hub" rev 1.00/1.00 addr 1 isa0 at pcib0 isadma0 at isa0 pckbc0 at isa0 port 0x60/5 pckbd0 at pckbc0 (kbd slot) pckbc0: using irq 1 for kbd slot wskbd0 at pckbd0: console keyboard, using wsdisplay0 pcppi0 at isa0 port 0x61 spkr0 at pcppi0 mtrr: Pentium Pro MTRR support umass0 at uhub0 port 2 configuration 1 interface 0 "American Megatrends Inc. Virtual Composite Storage Device" rev 2.00/1.00 addr 2 umass0: using SCSI over Bulk-Only scsibus0 at umass0: 2 targets, initiator 0 cd0 at scsibus0 targ 1 lun 0: <, Virtual CDROM, 1.00> SCSI0 5/cdrom removable umass1 at uhub0 port 2 configuration 1 interface 1 "American Megatrends Inc. Virtual Composite Storage Device" rev 2.00/1.00 addr 2 umass1: using SCSI over Bulk-Only scsibus1 at umass1: 2 targets, initiator 0 sd0 at scsibus1 targ 1 lun 0: <, Virtual Floppy, 1.00> SCSI0 0/direct removable uhidev0 at uhub2 port 1 configuration 1 interface 0 "American Megatrends Inc. Virtual Keyboard and Mouse" rev 1.10/1.00 addr 2 uhidev0: iclass 3/1 ukbd0 at uhidev0: 8 variable keys, 6 key codes wskbd1 at ukbd0 mux 1 wskbd1: connecting to wsdisplay0 uhidev1 at uhub2 port 1 configuration 1 interface 1 "American Megatrends Inc. Virtual Keyboard and Mouse" rev 1.10/1.00 addr 2 uhidev1: iclass 3/1 ums0 at uhidev1: 3 buttons, Z dir wsmouse0 at ums0 mux 0 uhidev2 at uhub2 port 2 configuration 1 interface 0 "Avocent Avocent KVM" rev 1.10/1.00 addr 3 uhidev2: iclass 3/1 ukbd1 at uhidev2: 8 variable keys, 6 key codes wskbd2 at ukbd1 mux 1 wskbd2: connecting to wsdisplay0 uhidev3 at uhub2 port 2 configuration 1 interface 1 "Avocent Avocent KVM" rev 1.10/1.00 addr 3 uhidev3: iclass 3/1, 3 report ids ums1 at uhidev3 reportid 1: 5 buttons, Z dir wsmouse1 at ums1 mux 0 uhid0 at uhidev3 reportid 2: input=2, output=0, feature=0 uhid1 at uhidev3 reportid 3: input=1, output=0, feature=0 vscsi0 at root scsibus2 at vscsi0: 256 targets softraid0 at root scsibus3 at softraid0: 256 targets root on wd0a (c12abee9ad63bcb9.a) swap on wd0b dump on wd0b bnx0: address 00:22:19:87:00:31 brgphy0 at bnx0 phy 2: BCM5708S 1000/2500baseSX PHY, rev. 1 bnx1: address 00:22:19:87:00:33 brgphy1 at bnx1 phy 2: BCM5708S 1000/2500baseSX PHY, rev. 1 bnx2: address 00:1d:09:74:30:50 brgphy2 at bnx2 phy 2: BCM5708S 1000/2500baseSX PHY, rev. 1 bnx3: address 00:1d:09:74:30:52 brgphy3 at bnx3 phy 2: BCM5708S 1000/2500baseSX PHY, rev. 1 -- regards, Wiesław Kielas
