Hello, ip_forward is disabled by default and I doesn't enable it for my DR setup.
I seems my version of ethtool 6 doesn't know anything about LRO. How could I see is LRO enabled or disabled? Regards, Aleksey -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Julian Anastasov Sent: Saturday, November 21, 2009 2:42 AM To: Aleksey Chudov Cc: 'LinuxVirtualServer.org users mailing list.' Subject: Re: [lvs-users] LVS and Broadcom bug Hello, On Fri, 20 Nov 2009, Aleksey Chudov wrote: > Hello! > > I have HP ProLiant BL460c G6 servers with Broadcom Corporation NetXtreme II > BCM57711E 10-Gigabit PCIe controller. After enabling Virtual Server in > kernel and placing server to the test environment with just 10Mbit/s traffic > I see a lot of error messages in logs. > > I try different kernel versions 2.6.27.39, 2.6.31.2, 2.6.31.6 but every time > I see the same error: > > With 2.6.31.x kernel: > > Nov 19 23:35:30 bsrvm8-1 kernel: ------------[ cut here ]------------ > Nov 19 23:35:30 bsrvm8-1 kernel: WARNING: at net/core/dev.c:1563 > skb_gso_segment+0x10f/0x260() > Nov 19 23:35:30 bsrvm8-1 kernel: Hardware name: ProLiant BL460c G6 > Nov 19 23:35:30 bsrvm8-1 kernel: bnx2x: caps=(0x198829, 0x0) len=719 > data_len=119 ip_summed=1Pid: 0, comm: swapper Tainted: G W caps 0x198829 has 0x8000 (NETIF_F_LRO). It seems LRO is not disabled for your devices, may be it should happen when you enable the IP forwarding for the concerned devices. It seems you didn't enabled IP forwarding. Can you confirm it? Also, may be LRO can be disabled with ethtool. The problem exists because IPVS does not disable LRO, it must be done under RTNL and IPVS never runs in this context. And LRO is not supported for forwarding: http://marc.info/?l=linux-netdev&m=121389887114416&w=2 IPVS does not call ip_forward for DR method, that is why you do not need forwarding and the LRO warning does not occur before hitting the GSO code. ip_forward just drops LRO packets: if (skb_warn_if_lro(skb)) goto drop; > 2.6.31.2-lv > s #1 > Nov 19 23:35:30 bsrvm8-1 kernel: Call Trace: > Nov 19 23:35:30 bsrvm8-1 kernel: <IRQ> [<ffffffff812c1b09>] ? > skb_gso_segment+0x10f/0x260 > Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff810352e1>] > warn_slowpath_common+0x72/0x8a > Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff8103536e>] > warn_slowpath_fmt+0x64/0x66 > Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff81255c2e>] ? > bnx2x_release_phy_lock+0x28/0x2d > Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff81261e51>] ? > bnx2x_get_drvinfo+0xea/0x121 > Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812c1b09>] > skb_gso_segment+0x10f/0x260 > Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812c1e04>] > dev_hard_start_xmit+0x1aa/0x2bc > Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812d0126>] > __qdisc_run+0xeb/0x1ef > Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812c4a08>] > dev_queue_xmit+0x1cd/0x2d9 > Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812e8549>] ? > ip_finish_output+0x0/0x264 > Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812e876d>] > ip_finish_output+0x224/0x264 > Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812e8adb>] ip_output+0xa9/0xad > Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812dc676>] > ip_vs_dr_xmit+0x137/0x16f > Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812d762f>] ip_vs_in+0x241/0x305 > Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812e2b63>] ? > ip_route_input+0xc84/0xdf3 > Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812d41f3>] nf_iterate+0x43/0x80 > Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812e4a26>] ? > ip_local_deliver_finish+0x0/0x176 > Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812d4291>] > nf_hook_slow+0x61/0xc3 > Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812e4a26>] ? > ip_local_deliver_finish+0x0/0x176 > Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812e4fe5>] > ip_local_deliver+0x63/0x7e > Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812e4a0c>] > ip_rcv_finish+0x308/0x322 > Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812e4f4d>] ip_rcv+0x2bd/0x2f2 > Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812c128e>] > netif_receive_skb+0x253/0x273 > Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff81257a5b>] > bnx2x_rx_int+0xbbc/0x1504 > Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff8102d223>] ? > enqueue_task_fair+0xcb/0xd8 > Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff8117fdc2>] ? > cpumask_next_and+0x2b/0x3d > Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff8102e884>] ? > find_busiest_group+0x22d/0x799 > Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff81265d17>] bnx2x_poll+0xd3/0x1bf > Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff81030ac1>] ? > run_rebalance_domains+0x183/0x481 > Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff812c3f10>] > net_rx_action+0x8b/0x146 > Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff81266d56>] ? > bnx2x_msix_fp_int+0x139/0x14b > Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff81039abf>] > __do_softirq+0xa2/0x13d > Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff8100bfec>] > call_softirq+0x1c/0x28 > Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff8100dba8>] do_softirq+0x42/0x88 > Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff81039a1b>] irq_exit+0x3f/0x41 > Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff8100d227>] do_IRQ+0xa3/0xba > Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff8100b813>] > ret_from_intr+0x0/0x11 > Nov 19 23:35:30 bsrvm8-1 kernel: <EOI> [<ffffffff810122aa>] ? > mwait_idle+0x67/0x72 > Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff81009bde>] ? > enter_idle+0x20/0x22 > Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff81009f6d>] ? cpu_idle+0x4e/0x6c > Nov 19 23:35:30 bsrvm8-1 kernel: [<ffffffff81338fb1>] ? > start_secondary+0x1a8/0x1ac > Nov 19 23:35:30 bsrvm8-1 kernel: ---[ end trace 5563ae7e045b479e ]--- Regards -- Julian Anastasov <[email protected]> _______________________________________________ Please read the documentation before posting - it's available at: http://www.linuxvirtualserver.org/ LinuxVirtualServer.org mailing list - [email protected] Send requests to [email protected] or go to http://lists.graemef.net/mailman/listinfo/lvs-users _______________________________________________ Please read the documentation before posting - it's available at: http://www.linuxvirtualserver.org/ LinuxVirtualServer.org mailing list - [email protected] Send requests to [email protected] or go to http://lists.graemef.net/mailman/listinfo/lvs-users
