Greetings, I administer a web server for my institution and last night we had a problem. The server is a 1U Intel Xeon E5620 machine. The on-board network interface is an Intel 82574L Gigabit Controller. Scientific Linux 6.4, kernel 2.6.32-431.1.2.el6.x86_64. At some point last night the network interface stopped working giving a backtrace on dev_watchdog. I could not restart the service network, it complained that the interface eth0 was not available. I tried to reconfigure it with NetworkManager, unsuccessfully. A full system reboot fixed the problem, although I couldn't identify the problem. I do not know if this matters, but this problem never occurred before the last yum update. Here below the portion of /var/log/messages that relates to the problem
================================================================= Dec 25 20:01:52 veritasm xinetd[1966]: EXIT: nrpe status=0 pid=20943 duration=0(sec) Dec 25 20:02:21 veritasm xinetd[1966]: START: nrpe pid=20947 from=::ffff:199.104.151.131 Dec 25 20:02:21 veritasm xinetd[1966]: EXIT: nrpe status=0 pid=20947 duration=0(sec) Dec 26 02:18:37 veritasm kernel: ------------[ cut here ]------------ Dec 26 02:18:37 veritasm kernel: WARNING: at net/sched/sch_generic.c:261 dev_watchdog+0x26b/0x280() (Not tainted) Dec 26 02:18:37 veritasm kernel: Hardware name: X8DTL Dec 26 02:18:37 veritasm kernel: NETDEV WATCHDOG: eth0 (e1000e): transmit queue 0 timed out Dec 26 02:18:37 veritasm kernel: Modules linked in: autofs4 8021q sunrpc garp stp llc cpufreq_ondemand acpi_cpufreq freq_table mperf ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 microcode iTCO_wdt iTCO_vendor_support sg i2c_i801 i2c_core lpc_ich mfd_core e1000e ptp pps_core ioatdma dca i7core_edac edac_core shpchp ext4 jbd2 mbcache raid1 sr_mod cdrom sd_mod crc_t10dif pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan] Dec 26 02:18:37 veritasm kernel: Pid: 130, comm: kipmi0 Not tainted 2.6.32-431.1.2.el6.x86_64 #1 Dec 26 02:18:37 veritasm kernel: Call Trace: Dec 26 02:18:37 veritasm kernel: <IRQ> [<ffffffff81071e27>] ? warn_slowpath_common+0x87/0xc0 Dec 26 02:18:37 veritasm kernel: [<ffffffff81071f16>] ? warn_slowpath_fmt+0x46/0x50 Dec 26 02:18:37 veritasm kernel: [<ffffffff8147b75b>] ? dev_watchdog+0x26b/0x280 Dec 26 02:18:37 veritasm kernel: [<ffffffff8105dd5c>] ? scheduler_tick+0xcc/0x260 Dec 26 02:18:37 veritasm kernel: [<ffffffff8147b4f0>] ? dev_watchdog+0x0/0x280 Dec 26 02:18:37 veritasm kernel: [<ffffffff81084b07>] ? run_timer_softirq+0x197/0x340 Dec 26 02:18:37 veritasm kernel: [<ffffffff810ac905>] ? tick_dev_program_event+0x65/0xc0 Dec 26 02:18:37 veritasm kernel: [<ffffffff8107a8e1>] ? __do_softirq+0xc1/0x1e0 Dec 26 02:18:37 veritasm kernel: [<ffffffff810ac9da>] ? tick_program_event+0x2a/0x30 Dec 26 02:18:37 veritasm kernel: [<ffffffff8100c30c>] ? call_softirq+0x1c/0x30 Dec 26 02:18:37 veritasm kernel: [<ffffffff8100fa75>] ? do_softirq+0x65/0xa0 Dec 26 02:18:37 veritasm kernel: [<ffffffff8107a795>] ? irq_exit+0x85/0x90 Dec 26 02:18:37 veritasm kernel: [<ffffffff815310ba>] ? smp_apic_timer_interrupt+0x4a/0x60 Dec 26 02:18:37 veritasm kernel: [<ffffffff8100bb93>] ? apic_timer_interrupt+0x13/0x20 Dec 26 02:18:37 veritasm kernel: <EOI> [<ffffffff8152a367>] ? _spin_unlock_irqrestore+0x17/0x20 Dec 26 02:18:37 veritasm kernel: [<ffffffff812e7790>] ? ipmi_thread+0x70/0x1c0 Dec 26 02:18:37 veritasm kernel: [<ffffffff812e7720>] ? ipmi_thread+0x0/0x1c0 Dec 26 02:18:37 veritasm kernel: [<ffffffff8109af06>] ? kthread+0x96/0xa0 Dec 26 02:18:37 veritasm kernel: [<ffffffff8100c20a>] ? child_rip+0xa/0x20 Dec 26 02:18:37 veritasm kernel: [<ffffffff8109ae70>] ? kthread+0x0/0xa0 Dec 26 02:18:37 veritasm kernel: [<ffffffff8100c200>] ? child_rip+0x0/0x20 Dec 26 02:18:37 veritasm kernel: ---[ end trace fc057a7fca6eff49 ]--- Dec 26 02:18:37 veritasm kernel: e1000e 0000:06:00.0: eth0: Reset adapter unexpectedly Dec 26 02:18:37 veritasm NetworkManager[1724]: <info> (eth0): carrier now OFF (device state 8, deferring action for 4 seconds) Dec 26 02:18:38 veritasm kernel: e1000e 0000:06:00.0: eth0: Timesync Tx Control register not set as expected Dec 26 02:18:41 veritasm NetworkManager[1724]: <info> (eth0): device state change: 8 -> 2 (reason 40) Dec 26 02:18:41 veritasm NetworkManager[1724]: <info> (eth0): deactivating device (reason: 40). Dec 26 02:18:43 veritasm ntpd[1974]: Deleting interface #5 eth0, 199.104.151.141#123, interface stats: received=789, sent=886, dropped=0, active_time=232100 secs Dec 26 08:58:36 veritasm kernel: fuse init (API version 7.13) Dec 26 08:58:36 veritasm rtkit-daemon[2379]: Sucessfully made thread 23005 of process 23005 (/usr/bin/pulseaudio) owned by '500' high priority at nice level -11. Dec 26 08:58:37 veritasm rtkit-daemon[2379]: Sucessfully made thread 23039 of process 23039 (/usr/bin/pulseaudio) owned by '500' high priority at nice level -11. Dec 26 08:58:37 veritasm pulseaudio[23039]: pid.c: Daemon already running. ===================================================================== and the dmesg portion too ====================================================================== e1000e 0000:06:00.0: eth0: registered PHC clock e1000e 0000:06:00.0: eth0: (PCI Express:2.5GT/s:Width x1) 00:25:90:c2:ec:00 e1000e 0000:06:00.0: eth0: Intel(R) PRO/1000 Network Connection e1000e 0000:06:00.0: eth0: MAC: 3, PHY: 8, PBA No: 0101FF-0FF ADDRCONF(NETDEV_UP): eth0: link is not ready e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready 8021q: adding VLAN 0 to HW filter on device eth0 eth0: no IPv6 routers present NETDEV WATCHDOG: eth0 (e1000e): transmit queue 0 timed out e1000e 0000:06:00.0: eth0: Reset adapter unexpectedly e1000e 0000:06:00.0: eth0: Timesync Tx Control register not set as expected ======================================================================= Hope somebody has an idea of where the problem might be. Regards, Nicola Galante
