[pfSense Support] em0: Watchdog timeout -- resetting

[Nathan Eisenberg]

Hello,   I am deploying a new set of firewall boxes using PFSense 1.2.1.  These boxes have 4GB of RAM, Intel Quad Core CPUs, and a PCI-E Intel 4 Port 10/100/1000 NIC (em0-3) in addition to the 2 onboard 10/100/1000 NICs (em4-5).   The Intel NIC seems to go to pieces whenever load is passed through it; the two onboard NICs do fine.   Since I'm using a Cisco switch, the port takes about 30 seconds to start forwarding packets after the issue passes.  I've seen this on my home PFSense box, as well (completely different hardware config), but since I'm not using a Cisco switch at home, I don't notice its effects as severely.   The error I am seeing is "em0: Watchdog Timeout -- Resetting", which seems to have several root causes.  I have tried disabling ACPI, both in the BIOS, and in the bootloader.  I have disabled all nonessential devices in the BIOS (except USB), and swapped all cables out (they're all brand new CAT6).  The problem occurs if em0 is connected to a switch, or to my laptop via crossover.  After three hours of Googling, I am stumped.   Help?   dmesg follows (although em0 and em1 have been replaced with em4 and em5 temporarily).   $ dmesg Copyright (c) 1992-2008 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994                 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 7.0-RELEASE-p7 #0: Thu Dec 25 14:39:15 EST 2008     sullr...@freebsd7-releng_1_2_1.pfsense.org:/usr/obj.pfSense/usr/src/sys/pfSense_SMP.7 Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(R) CPU           E5410  @ 2.33GHz (2333.43-MHz 686-class CPU)   Origin = "GenuineIntel"  Id = 0x1067a  Stepping = 10   Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>   Features2=0x40ce3bd<SSE3,RSVD2,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,<b19>,<b26>>   AMD Features=0x20100000<NX,LM>   AMD Features2=0x1<LAHF>   Cores per package: 4 real memory  = 2146467840 (2047 MB) avail memory = 2090897408 (1994 MB) MPTable: <INTEL    Bridge CRB  > FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs  cpu0 (BSP): APIC ID:  0  cpu1 (AP): APIC ID:  1  cpu2 (AP): APIC ID:  2  cpu3 (AP): APIC ID:  3 ioapic0: Assuming intbase of 0 ioapic1: Assuming intbase of 24 ioapic0 <Version 2.0> irqs 0-23 on motherboard ioapic1 <Version 2.0> irqs 24-47 on motherboard wlan: mac acl policy registered kbd1 at kbdmux0 ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413) hptrr: HPT RocketRAID controller driver v1.1 (Dec 25 2008 14:38:55) cryptosoft0: <software crypto> on motherboard cpu0 on motherboard est0: <Enhanced SpeedStep Frequency Control> on cpu0 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 721072106000721 device_attach: est0 attach returned 6 p4tcc0: <CPU Frequency Thermal Control> on cpu0 cpu1 on motherboard est1: <Enhanced SpeedStep Frequency Control> on cpu1 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 721072106000721 device_attach: est1 attach returned 6 p4tcc1: <CPU Frequency Thermal Control> on cpu1 cpu2 on motherboard est2: <Enhanced SpeedStep Frequency Control> on cpu2 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 721072106000721 device_attach: est2 attach returned 6 p4tcc2: <CPU Frequency Thermal Control> on cpu2 cpu3 on motherboard est3: <Enhanced SpeedStep Frequency Control> on cpu3 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 721072106000721 device_attach: est3 attach returned 6 p4tcc3: <CPU Frequency Thermal Control> on cpu3 pcib0: <MPTable Host-PCI bridge> pcibus 0 on motherboard pci0: <PCI bus> on pcib0 pcib1: <PCI-PCI bridge> irq 24 at device 1.0 on pci0 pci1: <PCI bus> on pcib1 pcib2: <PCI-PCI bridge> at device 0.0 on pci1 pci2: <PCI bus> on pcib2 pcib3: <MPTable PCI-PCI bridge> at device 2.0 on pci2 pci3: <PCI bus> on pcib3 pcib3: unable to route slot 0 INTB em0: <Intel(R) PRO/1000 Network Connection Version - 6.7.3> port 0x2000-0x201f mem 0xd8020000-0xd803ffff,0xd8000000-0xd801ffff irq 11 at device 0.0 on pci3 em0: Using MSI interrupt em0: Ethernet address: 00:15:17:90:09:c1 em0: [FILTER] em1: <Intel(R) PRO/1000 Network Connection Version - 6.7.3> port 0x2020-0x203f mem 0xd8060000-0xd807ffff,0xd8040000-0xd805ffff irq 28 at device 0.1 on pci3 em1: Using MSI interrupt em1: Ethernet address: 00:15:17:90:09:c0 em1: [FILTER] pcib4: <MPTable PCI-PCI bridge> at device 4.0 on pci2 pci4: <PCI bus> on pcib4 pcib4: unable to route slot 0 INTB em2: <Intel(R) PRO/1000 Network Connection Version - 6.7.3> port 0x3000-0x301f mem 0xd8120000-0xd813ffff,0xd8100000-0xd811ffff irq 11 at device 0.0 on pci4 em2: Using MSI interrupt em2: Ethernet address: 00:15:17:90:09:c3 em2: [FILTER] em3: <Intel(R) PRO/1000 Network Connection Version - 6.7.3> port 0x3020-0x303f mem 0xd8160000-0xd817ffff,0xd8140000-0xd815ffff irq 28 at device 0.1 on pci4 em3: Using MSI interrupt em3: Ethernet address: 00:15:17:90:09:c2 em3: [FILTER] pcib5: <PCI-PCI bridge> irq 28 at device 5.0 on pci0 pci5: <PCI bus> on pcib5 pcib6: <MPTable PCI-PCI bridge> irq 28 at device 0.0 on pci5 pci6: <PCI bus> on pcib6 pcib6: unable to route slot 0 INTA pcib7: <PCI-PCI bridge> irq 11 at device 0.0 on pci6 pci7: <PCI bus> on pcib7 pcib8: <MPTable PCI-PCI bridge> irq 44 at device 2.0 on pci6 pci8: <PCI bus> on pcib8 em4: <Intel(R) PRO/1000 Network Connection Version - 6.7.3> port 0x4000-0x401f mem 0xd8220000-0xd823ffff,0xd8200000-0xd821ffff irq 44 at device 0.0 on pci8 em4: Using MSI interrupt em4: Ethernet address: 00:30:48:c4:98:4c em4: [FILTER] em5: <Intel(R) PRO/1000 Network Connection Version - 6.7.3> port 0x4020-0x403f mem 0xd8260000-0xd827ffff,0xd8240000-0xd825ffff irq 40 at device 0.1 on pci8 em5: Using MSI interrupt em5: Ethernet address: 00:30:48:c4:98:4d em5: [FILTER] pcib9: <PCI-PCI bridge> at device 0.3 on pci5 pci9: <PCI bus> on pcib9 pcib10: <PCI-PCI bridge> irq 30 at device 7.0 on pci0 pci10: <PCI bus> on pcib10 uhci0: <Intel 631XESB/632XESB/3100 USB controller USB-1> port 0x1800-0x181f irq 20 at device 29.0 on pci0 uhci0: [GIANT-LOCKED] uhci0: [ITHREAD] usb0: <Intel 631XESB/632XESB/3100 USB controller USB-1> on uhci0 usb0: USB revision 1.0 uhub0: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0 uhub0: 2 ports with 2 removable, self powered uhci1: <Intel 631XESB/632XESB/3100 USB controller USB-2> port 0x1820-0x183f irq 21 at device 29.1 on pci0 uhci1: [GIANT-LOCKED] uhci1: [ITHREAD] usb1: <Intel 631XESB/632XESB/3100 USB controller USB-2> on uhci1 usb1: USB revision 1.0 uhub1: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb1 uhub1: 2 ports with 2 removable, self powered uhci2: <Intel 631XESB/632XESB/3100 USB controller USB-3> port 0x1840-0x185f irq 22 at device 29.2 on pci0 uhci2: [GIANT-LOCKED] uhci2: [ITHREAD] usb2: <Intel 631XESB/632XESB/3100 USB controller USB-3> on uhci2 usb2: USB revision 1.0 uhub2: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb2 uhub2: 2 ports with 2 removable, self powered ehci0: <Intel 63XXESB USB 2.0 controller> mem 0xd8704000-0xd87043ff irq 23 at device 29.7 on pci0 ehci0: [GIANT-LOCKED] ehci0: [ITHREAD] usb3: EHCI version 1.0 usb3: companion controllers, 2 ports each: usb0 usb1 usb2 usb3: <Intel 63XXESB USB 2.0 controller> on ehci0 usb3: USB revision 2.0 uhub3: <Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1> on usb3 uhub3: 6 ports with 6 removable, self powered pcib11: <MPTable PCI-PCI bridge> at device 30.0 on pci0 pci11: <PCI bus> on pcib11 vgapci0: <VGA-compatible display> port 0x5000-0x50ff mem 0xd0000000-0xd7ffffff,0xd8400000-0xd840ffff irq 18 at device 1.0 on pci11 isab0: <PCI-ISA bridge> at device 31.0 on pci0 isa0: <ISA bus> on isab0 atapci0: <Intel 63XXESB2 SATA300 controller> port 0x1880-0x1887,0x1874-0x1877,0x1878-0x187f,0x1870-0x1873,0x1860-0x186f mem 0xd8704400-0xd87047ff irq 17 at device 31.2 on pci0 atapci0: [ITHREAD] ata2: <ATA channel 0> on atapci0 ata2: [ITHREAD] ata3: <ATA channel 1> on atapci0 ata3: [ITHREAD] pci0: <serial bus, SMBus> at device 31.3 (no driver attached) pmtimer0 on isa0 orm0: <ISA Option ROM> at iomem 0xc0000-0xcafff pnpid ORM0000 on isa0 ata0 at port 0x1f0-0x1f7,0x3f6 irq 14 on isa0 ata0: [ITHREAD] ata1 at port 0x170-0x177,0x376 irq 15 on isa0 ata1: [ITHREAD] atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0 atkbd0: <AT Keyboard> irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] ppc0: parallel port not found. sc0: <System console> at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> sio0: configured irq 4 not in bitmap of probed irqs 0 sio0: port may not be enabled sio0: configured irq 4 not in bitmap of probed irqs 0 sio0: port may not be enabled sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0 sio0: type 8250 or not responding sio0: [FILTER] sio1: configured irq 3 not in bitmap of probed irqs 0 sio1: port may not be enabled vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 unknown: <PNP0c02> can't assign resources (memory) unknown: <PNP0c02> can't assign resources (memory) unknown: <PNP0c01> can't assign resources (memory) unknown: <PNP0303> can't assign resources (port) speaker0: <PC speaker> at port 0x61 pnpid PNP0800 on isa0 unknown: <PNP0c02> can't assign resources (memory) unknown: <PNP0c02> can't assign resources (memory) unknown: <INT0800> can't assign resources (memory) fdc1: No FDOUT register! Timecounters tick every 1.000 msec Fast IPsec: Initialized Security Association Processing. hptrr: no controller detected. ad4: 76319MB <WDC WD800AAJS-00PSA0 05.06H05> at ata2-master SATA150 SMP: AP CPU #1 Launched! SMP: AP CPU #3 Launched! SMP: AP CPU #2 Launched! WARNING: Expected rawoffset 0, found 63 Trying to mount root from ufs:/dev/ad4s1a em4: link state changed to UP em5: link state changed to UP em5: link state changed to DOWN em4: link state changed to DOWN pflog0: promiscuous mode enabled em5: link state changed to UP ukbd0: <Generic USB+PS2 Keyboard, class 0/0, rev 1.10/2.02, addr 2> on uhub0 kbd2 at ukbd0 uhid0: <Generic USB+PS2 Keyboard, class 0/0, rev 1.10/2.02, addr 2> on uhub0 em4: link state changed to UP ukbd0: at uhub0 port 1 (addr 2) disconnected ukbd0: detached uhid0: at uhub0 port 1 (addr 2) disconnected uhid0: detached em5: link state changed to DOWN   Thank You, Nathan Eisenberg Sr. Systems Administrator Atlas Networks, LLC    Atlas Networks is an Atlas Accelerator Company