J.B. Broccard <[email protected]> wrote:
[...]
>Thanks for the insights.
>When I set the MTU to 9000 in the ifcfg-bond0 file, the behavior is the
>same: i.e. the eth3 becomes active instead of eth2 (which is wrong in
>my case), see dmesg sequence:
>[ 45.163896] bonding: bond0: setting mode to active-backup (1).
>[ 45.164045] bonding: bond0: Setting MII monitoring interval to 100.
>[ 45.164186] bonding: bond0: Setting use_carrier to 0.
As an FYI, you almost certainly don't need to set use_carrier to
0; it's a backwards compatibility option for really, really old devices
(as in 10 Mb/sec era stuff). This is probably unrelated to your problem
below.
>[ 45.164323] bonding: bond0: Unable to set eth2 as primary slave as it is
>not a slave.
>[ 45.165061] bonding: bond0: setting primary_reselect to failure (2).
>[ 45.266019] bonding: bond0: Adding slave eth2.
>[ 45.424272] bonding: bond0: enslaving eth2 as a backup interface with a
>down link.
>[ 45.430611] ixgbe 0000:b0:00.0: eth2: NIC Link is Up 10 Gbps, Flow
>Control: RX/TX
>[ 45.466570] bonding: bond0: link status definitely up for interface
>eth2.
>[ 45.466573] bonding: bond0: making interface eth2 the new active one.
>[ 45.466977] bonding: bond0: first active interface up!
>[ 45.522509] bonding: bond0: Adding slave eth3.
>[ 45.704223] bonding: bond0: enslaving eth3 as a backup interface with a
>down link.
>[ 45.706726] bonding: bond0: Setting eth2 as primary slave.
>[ 45.708377] ixgbe 0000:b0:00.0: eth2: Setting MTU > 1500 will disable
>legacy VFs
>[ 45.708382] ixgbe 0000:b0:00.0: eth2: changing MTU from 1500 to 9000
>[ 45.710628] ixgbe 0000:b0:00.1: eth3: NIC Link is Up 10 Gbps, Flow
>Control: RX/TX
>[ 47.951705] ixgbe 0000:b0:00.1: eth3: Setting MTU > 1500 will disable
>legacy VFs
>[ 47.951709] ixgbe 0000:b0:00.1: eth3: changing MTU from 1500 to 9000
>[ 50.259697] ixgbe 0000:b0:00.1: eth3: NIC Link is Up 10 Gbps, Flow
>Control: RX/TX
>[ 50.259846] ixgbe 0000:b0:00.0: eth2: NIC Link is Up 10 Gbps, Flow
>Control: RX/TX
>[ 50.262100] bonding: bond0: link status definitely down for interface
>eth2, disabling it
>[ 50.262363] bonding: bond0: now running without any active interface !
>[ 50.262365] bonding: bond0: link status definitely up for interface
>eth3.
>[ 50.262367] bonding: bond0: making interface eth3 the new active one.
>[ 50.262887] bonding: bond0: first active interface up!
>
>How can I keep using the Linux configuration files (modprobe.conf and
>ifcfg
>files) and make sure that I set the MTU to 9K before enslaving?
The sequence here seems kind of odd, because the slaves go link
up, and then the bond apparently has its MTU set after the slaves are
added. That in turn causes the link state to bounce, and eth3 happens
to go link up before eth2.
You mentioned running OVM server, which is a distro I'm not
familiar with (at least by that name), but perhaps its initscripts /
sysconfig / whatever package it uses to configure the networking is
setting the bond's MTU after adding the slaves for some reason.
Unless I'm missing something and this link bounce is due to
something in ixgbe, but that doesn't seem very likely.
When I set up a similar situation here (where I set the bond's
MTU to 1000, and one slave is set to 900, both in ifcfg-), I don't see
the second set of link state changes, the sequence looks like:
bonding: bond0: setting mode to active-backup (1).
bonding: bond0: Setting MII monitoring interval to 100.
bonding: bond0: Recording eth3 as primary, but it has not been enslaved to
bond0 yet.
bonding: bond0: setting primary_reselect to failure (2).
IPv6: ADDRCONF(NETDEV_UP): bond0: link is not ready
bonding: bond0: Adding slave eth3.
e1000: eth3 changing MTU from 900 to 1500
bonding: bond0: enslaving eth3 as a backup interface with a down link.
bonding: bond0: Adding slave eth4.
bonding: bond0: enslaving eth4 as a backup interface with a down link.
bonding: bond0: setting primary_reselect to failure (2).
e1000: eth3 changing MTU from 1500 to 1000
e1000: eth4 changing MTU from 1500 to 1000
e1000: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
bonding: bond0: link status definitely up for interface eth3, 1000 Mbps full
duplex.
bonding: bond0: making interface eth3 the new active one.
bonding: bond0: first active interface up!
IPv6: ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready
e1000: eth4 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
bonding: bond0: link status definitely up for interface eth4, 1000 Mbps full
duplex.
I'm testing above with sysconfig-0.73.7, but a look at
initscripts 9.42 suggests that it should behave similarly.
What's happening in your case is that the two slaves are added
more or less simultaneously (within .25 seconds), and come up initially
in enslavement order (what you're relying on), but when the MTU changes
later, eth3 comes up first, and is thus made the active slave.
[...]
>So, I tried to set MTU=9000 in ifcfg-eth2 and 3, tried to restart the
>network (unload/reload ixgbe and bonding modules), even tried to reboot
>the system ... the system seems to ignore the parameter, eth2/3 as well
>as bond0 are still at 1500.
>I currently have primary_reselect set to 2 and we don't want to change
>this to 0 (automatic failback).
The way bonding works, the MTU is "pushed" from the master to
the slaves at enslavement time. If slaves have a different MTU at
enslavement time, it is replaced with the MTU of the bond before the
slave is brought up during enslavement. It looks like your network
config scripts may not work this way.
The usual way to set the MTU is what you had previously: Set MTU
in ifcfg-bond0.
-J
---
-Jay Vosburgh, IBM Linux Technology Center, [email protected]
------------------------------------------------------------------------------
Get 100% visibility into Java/.NET code with AppDynamics Lite
It's a free troubleshooting tool designed for production
Get down to code-level detail for bottlenecks, with <2% overhead.
Download for free and get started troubleshooting in minutes.
http://p.sf.net/sfu/appdyn_d2d_ap2
_______________________________________________
E1000-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel® Ethernet, visit
http://communities.intel.com/community/wired
