> -----Original Message----- > From: J.B. Broccard [mailto:[email protected]] > Sent: Monday, June 03, 2013 5:15 PM > To: Tantilov, Emil S; Jay Vosburgh; [email protected] > Subject: Re: [E1000-devel] Bonding + ixgbe breaks with jumbo frames if the > MTU is not set on bond0 before adding slaves > > > > -----Original Message----- > From: Tantilov, Emil S [mailto:[email protected]] > Sent: Monday, June 03, 2013 4:45 PM > To: J.B. Broccard; Jay Vosburgh; [email protected] > Subject: RE: [E1000-devel] Bonding + ixgbe breaks with jumbo frames if the > MTU is not set on bond0 before adding slaves > > > >-----Original Message----- > >From: J.B. Broccard [mailto:[email protected]] > >Sent: Monday, June 03, 2013 3:58 PM > >To: Jay Vosburgh; [email protected] > >Cc: Tantilov, Emil S > >Subject: RE: [E1000-devel] Bonding + ixgbe breaks with jumbo frames if > >the MTU is not set on bond0 before adding slaves > > > > > > > >-----Original Message----- > >From: Jay Vosburgh [mailto:[email protected]] > >Sent: Friday, May 31, 2013 5:17 PM > >To: J.B. Broccard > >Cc: Tantilov, Emil S; [email protected] > >Subject: Re: [E1000-devel] Bonding + ixgbe breaks with jumbo frames if > >the MTU is not set on bond0 before adding slaves > > > >J.B. Broccard <[email protected]> wrote: > >[...] > >>Thanks for the insights. > >>When I set the MTU to 9000 in the ifcfg-bond0 file, the behavior is > >>the > >>same: i.e. the eth3 becomes active instead of eth2 (which is wrong in > >>my case), see dmesg sequence: > >>[ 45.163896] bonding: bond0: setting mode to active-backup (1). > >>[ 45.164045] bonding: bond0: Setting MII monitoring interval to 100. > >>[ 45.164186] bonding: bond0: Setting use_carrier to 0. > > > > As an FYI, you almost certainly don't need to set use_carrier to 0; > >it's a backwards compatibility option for really, really old devices > >(as in > >10 Mb/sec era stuff). This is probably unrelated to your problem below. > > > >>[ 45.164323] bonding: bond0: Unable to set eth2 as primary slave as it > >is > >>not a slave. > >>[ 45.165061] bonding: bond0: setting primary_reselect to failure (2). > >>[ 45.266019] bonding: bond0: Adding slave eth2. > >>[ 45.424272] bonding: bond0: enslaving eth2 as a backup interface with a > >>down link. > >>[ 45.430611] ixgbe 0000:b0:00.0: eth2: NIC Link is Up 10 Gbps, Flow > >>Control: RX/TX > >>[ 45.466570] bonding: bond0: link status definitely up for interface > >>eth2. > >>[ 45.466573] bonding: bond0: making interface eth2 the new active one. > >>[ 45.466977] bonding: bond0: first active interface up! > >>[ 45.522509] bonding: bond0: Adding slave eth3. > >>[ 45.704223] bonding: bond0: enslaving eth3 as a backup interface with a > >>down link. > >>[ 45.706726] bonding: bond0: Setting eth2 as primary slave. > >>[ 45.708377] ixgbe 0000:b0:00.0: eth2: Setting MTU > 1500 will disable > >>legacy VFs > >>[ 45.708382] ixgbe 0000:b0:00.0: eth2: changing MTU from 1500 to 9000 > >>[ 45.710628] ixgbe 0000:b0:00.1: eth3: NIC Link is Up 10 Gbps, Flow > >>Control: RX/TX > >>[ 47.951705] ixgbe 0000:b0:00.1: eth3: Setting MTU > 1500 will disable > >>legacy VFs > >>[ 47.951709] ixgbe 0000:b0:00.1: eth3: changing MTU from 1500 to 9000 > >>[ 50.259697] ixgbe 0000:b0:00.1: eth3: NIC Link is Up 10 Gbps, Flow > >>Control: RX/TX > >>[ 50.259846] ixgbe 0000:b0:00.0: eth2: NIC Link is Up 10 Gbps, Flow > >>Control: RX/TX > >>[ 50.262100] bonding: bond0: link status definitely down for interface > >>eth2, disabling it > >>[ 50.262363] bonding: bond0: now running without any active interface ! > >>[ 50.262365] bonding: bond0: link status definitely up for interface > >>eth3. > >>[ 50.262367] bonding: bond0: making interface eth3 the new active one. > >>[ 50.262887] bonding: bond0: first active interface up! > >> > >>How can I keep using the Linux configuration files (modprobe.conf and > >>ifcfg > >>files) and make sure that I set the MTU to 9K before enslaving? > > > > The sequence here seems kind of odd, because the slaves go link up, > >and then the bond apparently has its MTU set after the slaves are added. > >That in turn causes the link state to bounce, and eth3 happens to go > >link up before eth2. > > > > You mentioned running OVM server, which is a distro I'm not familiar > >with (at least by that name), but perhaps its initscripts / sysconfig / > >whatever package it uses to configure the networking is setting the > >bond's MTU after adding the slaves for some reason. > > > > Unless I'm missing something and this link bounce is due to > something > >in ixgbe, but that doesn't seem very likely. > > > > When I set up a similar situation here (where I set the bond's MTU to > >1000, and one slave is set to 900, both in ifcfg-), I don't see the > >second set of link state changes, the sequence looks like: > > > >bonding: bond0: setting mode to active-backup (1). > >bonding: bond0: Setting MII monitoring interval to 100. > >bonding: bond0: Recording eth3 as primary, but it has not been enslaved > >to > >bond0 yet. > >bonding: bond0: setting primary_reselect to failure (2). > >IPv6: ADDRCONF(NETDEV_UP): bond0: link is not ready > >bonding: bond0: Adding slave eth3. > >e1000: eth3 changing MTU from 900 to 1500 > >bonding: bond0: enslaving eth3 as a backup interface with a down link. > >bonding: bond0: Adding slave eth4. > >bonding: bond0: enslaving eth4 as a backup interface with a down link. > >bonding: bond0: setting primary_reselect to failure (2). > >e1000: eth3 changing MTU from 1500 to 1000 > >e1000: eth4 changing MTU from 1500 to 1000 > >e1000: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX > >bonding: bond0: link status definitely up for interface eth3, 1000 Mbps > >full duplex. > >bonding: bond0: making interface eth3 the new active one. > >bonding: bond0: first active interface up! > >IPv6: ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready > >e1000: eth4 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX > >bonding: bond0: link status definitely up for interface eth4, 1000 Mbps > >full duplex. > > > > I'm testing above with sysconfig-0.73.7, but a look at initscripts > >9.42 suggests that it should behave similarly. > > > > What's happening in your case is that the two slaves are added more > or > >less simultaneously (within .25 seconds), and come up initially in > >enslavement order (what you're relying on), but when the MTU changes > >later, > >eth3 comes up first, and is thus made the active slave. > > > >[...] > >>So, I tried to set MTU=9000 in ifcfg-eth2 and 3, tried to restart the > >>network (unload/reload ixgbe and bonding modules), even tried to > >>reboot the system ... the system seems to ignore the parameter, eth2/3 > >>as well as bond0 are still at 1500. > > > >>I currently have primary_reselect set to 2 and we don't want to change > >>this to 0 (automatic failback). > > > > The way bonding works, the MTU is "pushed" from the master to the > >slaves at enslavement time. If slaves have a different MTU at > >enslavement time, it is replaced with the MTU of the bond before the > >slave is brought up during enslavement. It looks like your network > >config scripts may not work this way. > > > > The usual way to set the MTU is what you had previously: Set MTU in > >ifcfg-bond0. > > > > -J > > > >--- > > -Jay Vosburgh, IBM Linux Technology Center, [email protected] > > > >----------------------------------------------------------------------- > >- > > > >Jay, Emil, Experts, > > > >I think you got it right: the starting sequence of my VM is odd. > >Apparently, adding MTU=9000 in ifcfg-bond0 causes the bonded interfaces > >(eth2 and eth3) to be reset which causes the bonding failover. > >What is even more strange is that, since it's a racing condition, I > >would expect eth2 to come back as the primary as often as eth3, but > >it's not the case, eth3 becomes always the active one. > > > >For information, I'm using OVM Server 303 based on a Xen kernel and the > >bonding driver is 3.6.0. > > > >Anyways, I managed to work around the issue by inserting a RC script > >that acts *after* modprobe.conf is read and creates bond0 and *before* > >ifcfg- files are read. In other words, when alias bond0 bonding is read > >by the system, bond0 exist but hasn't enslaved eth2/3 yet, then I set > >the MTU with ifconfig command, and then I let /etc/rc3.d/network start > >the rest of the network (i.e. enslavement of eth2/3). > > > >When I do this, the dmesg sequence looks like yours and I don't observe > >the "Setting MTU > 1500 disabling legacy VFs" (something like that > >which apparently is the message that precedes the reset of the eth2/3). > >When I first read this thread, I figured that I was in the same > >situation of the initiator and that Intel fixed this X-540 NICs, but > >the problem remains for > >82599 NICs within the driver. Therefore, if someone from Intel could > >advise on this, I'd really appreciate :) > > What problem are you referring to? As I mentioned in my previous email, this > thread was regarding a driver bug that cleared the MAC table on reset and is > not dependent on HW. Your issue seems to be related to how your init > scripts bring up the interfaces. > > > Thanks, > Emil > > > > >Meanwhile, thanks all for the help. > >JB > > > > > > > > > > > > > > Hi Emil, > Originally the issue was: the bond fails over when I set the MTU after the > enslavement. > I related my issue to that one because I was able to reproduce this as well :) > I take a note on the init scripts. But, I am even able to reproduce the > failover > just by doing a simple: "ifconfig bond0 mtu 9000" while interfaces are already > enslaved. > JB
Hey JB, This is because the driver will reset on a new MTU as it must renegotiate with the link partner. So anytime you change the MTU a reset will occur and if your set to failover on a reset then you failover. This is why Emil suggested you sent your MTU size before setting up bonding so that the reset occurs before the bonding failure is enabled. Thanks, -Don Skidmore <[email protected]> ------------------------------------------------------------------------------ How ServiceNow helps IT people transform IT departments: 1. A cloud service to automate IT design, transition and operations 2. Dashboards that offer high-level views of enterprise services 3. A single system of record for all IT processes http://p.sf.net/sfu/servicenow-d2d-j _______________________________________________ E1000-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
