>-----Original Message----- >From: J.B. Broccard [mailto:[email protected]] >Sent: Monday, June 03, 2013 3:58 PM >To: Jay Vosburgh; [email protected] >Cc: Tantilov, Emil S >Subject: RE: [E1000-devel] Bonding + ixgbe breaks with jumbo frames if the >MTU is not set on bond0 before adding slaves > > > >-----Original Message----- >From: Jay Vosburgh [mailto:[email protected]] >Sent: Friday, May 31, 2013 5:17 PM >To: J.B. Broccard >Cc: Tantilov, Emil S; [email protected] >Subject: Re: [E1000-devel] Bonding + ixgbe breaks with jumbo frames if the >MTU is not set on bond0 before adding slaves > >J.B. Broccard <[email protected]> wrote: >[...] >>Thanks for the insights. >>When I set the MTU to 9000 in the ifcfg-bond0 file, the behavior is the >>same: i.e. the eth3 becomes active instead of eth2 (which is wrong in >>my case), see dmesg sequence: >>[ 45.163896] bonding: bond0: setting mode to active-backup (1). >>[ 45.164045] bonding: bond0: Setting MII monitoring interval to 100. >>[ 45.164186] bonding: bond0: Setting use_carrier to 0. > > As an FYI, you almost certainly don't need to set use_carrier to 0; >it's a backwards compatibility option for really, really old devices (as in >10 Mb/sec era stuff). This is probably unrelated to your problem below. > >>[ 45.164323] bonding: bond0: Unable to set eth2 as primary slave as it >is >>not a slave. >>[ 45.165061] bonding: bond0: setting primary_reselect to failure (2). >>[ 45.266019] bonding: bond0: Adding slave eth2. >>[ 45.424272] bonding: bond0: enslaving eth2 as a backup interface with a >>down link. >>[ 45.430611] ixgbe 0000:b0:00.0: eth2: NIC Link is Up 10 Gbps, Flow >>Control: RX/TX >>[ 45.466570] bonding: bond0: link status definitely up for interface >>eth2. >>[ 45.466573] bonding: bond0: making interface eth2 the new active one. >>[ 45.466977] bonding: bond0: first active interface up! >>[ 45.522509] bonding: bond0: Adding slave eth3. >>[ 45.704223] bonding: bond0: enslaving eth3 as a backup interface with a >>down link. >>[ 45.706726] bonding: bond0: Setting eth2 as primary slave. >>[ 45.708377] ixgbe 0000:b0:00.0: eth2: Setting MTU > 1500 will disable >>legacy VFs >>[ 45.708382] ixgbe 0000:b0:00.0: eth2: changing MTU from 1500 to 9000 >>[ 45.710628] ixgbe 0000:b0:00.1: eth3: NIC Link is Up 10 Gbps, Flow >>Control: RX/TX >>[ 47.951705] ixgbe 0000:b0:00.1: eth3: Setting MTU > 1500 will disable >>legacy VFs >>[ 47.951709] ixgbe 0000:b0:00.1: eth3: changing MTU from 1500 to 9000 >>[ 50.259697] ixgbe 0000:b0:00.1: eth3: NIC Link is Up 10 Gbps, Flow >>Control: RX/TX >>[ 50.259846] ixgbe 0000:b0:00.0: eth2: NIC Link is Up 10 Gbps, Flow >>Control: RX/TX >>[ 50.262100] bonding: bond0: link status definitely down for interface >>eth2, disabling it >>[ 50.262363] bonding: bond0: now running without any active interface ! >>[ 50.262365] bonding: bond0: link status definitely up for interface >>eth3. >>[ 50.262367] bonding: bond0: making interface eth3 the new active one. >>[ 50.262887] bonding: bond0: first active interface up! >> >>How can I keep using the Linux configuration files (modprobe.conf and >>ifcfg >>files) and make sure that I set the MTU to 9K before enslaving? > > The sequence here seems kind of odd, because the slaves go link up, >and then the bond apparently has its MTU set after the slaves are added. >That in turn causes the link state to bounce, and eth3 happens to go link >up before eth2. > > You mentioned running OVM server, which is a distro I'm not familiar >with (at least by that name), but perhaps its initscripts / sysconfig / >whatever package it uses to configure the networking is setting the bond's >MTU after adding the slaves for some reason. > > Unless I'm missing something and this link bounce is due to something >in ixgbe, but that doesn't seem very likely. > > When I set up a similar situation here (where I set the bond's MTU to >1000, and one slave is set to 900, both in ifcfg-), I don't see the second >set of link state changes, the sequence looks like: > >bonding: bond0: setting mode to active-backup (1). >bonding: bond0: Setting MII monitoring interval to 100. >bonding: bond0: Recording eth3 as primary, but it has not been enslaved to >bond0 yet. >bonding: bond0: setting primary_reselect to failure (2). >IPv6: ADDRCONF(NETDEV_UP): bond0: link is not ready >bonding: bond0: Adding slave eth3. >e1000: eth3 changing MTU from 900 to 1500 >bonding: bond0: enslaving eth3 as a backup interface with a down link. >bonding: bond0: Adding slave eth4. >bonding: bond0: enslaving eth4 as a backup interface with a down link. >bonding: bond0: setting primary_reselect to failure (2). >e1000: eth3 changing MTU from 1500 to 1000 >e1000: eth4 changing MTU from 1500 to 1000 >e1000: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX >bonding: bond0: link status definitely up for interface eth3, 1000 Mbps >full duplex. >bonding: bond0: making interface eth3 the new active one. >bonding: bond0: first active interface up! >IPv6: ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready >e1000: eth4 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX >bonding: bond0: link status definitely up for interface eth4, 1000 Mbps >full duplex. > > I'm testing above with sysconfig-0.73.7, but a look at initscripts >9.42 suggests that it should behave similarly. > > What's happening in your case is that the two slaves are added more >or less simultaneously (within .25 seconds), and come up initially in >enslavement order (what you're relying on), but when the MTU changes later, >eth3 comes up first, and is thus made the active slave. > >[...] >>So, I tried to set MTU=9000 in ifcfg-eth2 and 3, tried to restart the >>network (unload/reload ixgbe and bonding modules), even tried to reboot >>the system ... the system seems to ignore the parameter, eth2/3 as well >>as bond0 are still at 1500. > >>I currently have primary_reselect set to 2 and we don't want to change >>this to 0 (automatic failback). > > The way bonding works, the MTU is "pushed" from the master to the >slaves at enslavement time. If slaves have a different MTU at enslavement >time, it is replaced with the MTU of the bond before the slave is brought >up during enslavement. It looks like your network config scripts may not >work this way. > > The usual way to set the MTU is what you had previously: Set MTU in >ifcfg-bond0. > > -J > >--- > -Jay Vosburgh, IBM Linux Technology Center, [email protected] > >------------------------------------------------------------------------ > >Jay, Emil, Experts, > >I think you got it right: the starting sequence of my VM is odd. >Apparently, adding MTU=9000 in ifcfg-bond0 causes the bonded interfaces >(eth2 and eth3) to be reset which causes the bonding failover. >What is even more strange is that, since it's a racing condition, I would >expect eth2 to come back as the primary as often as eth3, but it's not the >case, eth3 becomes always the active one. > >For information, I'm using OVM Server 303 based on a Xen kernel and the >bonding driver is 3.6.0. > >Anyways, I managed to work around the issue by inserting a RC script that >acts *after* modprobe.conf is read and creates bond0 and *before* ifcfg- >files are read. In other words, when alias bond0 bonding is read by the >system, bond0 exist but hasn't enslaved eth2/3 yet, then I set the MTU with >ifconfig command, and then I let /etc/rc3.d/network start the rest of the >network (i.e. enslavement of eth2/3). > >When I do this, the dmesg sequence looks like yours and I don't observe the >"Setting MTU > 1500 disabling legacy VFs" (something like that which >apparently is the message that precedes the reset of the eth2/3). When I >first read this thread, I figured that I was in the same situation of the >initiator and that Intel fixed this X-540 NICs, but the problem remains for >82599 NICs within the driver. Therefore, if someone from Intel could advise >on this, I'd really appreciate :)
What problem are you referring to? As I mentioned in my previous email, this thread was regarding a driver bug that cleared the MAC table on reset and is not dependent on HW. Your issue seems to be related to how your init scripts bring up the interfaces. Thanks, Emil > >Meanwhile, thanks all for the help. >JB > > > > > > ------------------------------------------------------------------------------ How ServiceNow helps IT people transform IT departments: 1. A cloud service to automate IT design, transition and operations 2. Dashboards that offer high-level views of enterprise services 3. A single system of record for all IT processes http://p.sf.net/sfu/servicenow-d2d-j _______________________________________________ E1000-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
