Rick Jones <[EMAIL PROTECTED]> wrote: [...] >> Note that this out of order delivery occurs when both the >> sending and receiving systems are utilizing a multiple >> interface bond. Consider a configuration in which a >> balance-rr bond feeds into a single higher capacity network >> channel (e.g., multiple 100Mb/sec ethernets feeding a single >> gigabit ethernet via an etherchannel capable switch). In this >> configuration, traffic sent from the multiple 100Mb devices to >> a destination connected to the gigabit device will not see >> packets out of order. > >My first reaction was that this was incorrect - it didn't matter if the >receiver was using a single link or not because the packets flowing across >the multiple 100Mb links could hit the intermediate device out of order >and so stay that way across the GbE link.
Usually it does matter, at least at the time I tested this. Usually, the even striping of traffic from the balance-rr mode will deliver in-order to a single higher speed link (e.g., N 100Mb feeding a single 1Gb). I say "usually" because, although I don't see it happen with the equipment I have, I'm willing to believe that there are gizmos that would "bundle" packets arriving on the switch ports. The reordering (usually) occurs when packet coalescing stuff (either interrupt mitigation on the device, or NAPI) happens at the receiver end, after the packets are striped evenly into the interfaces, e.g., eth0 eth1 eth2 P1 P2 P3 P4 P5 P6 P7 P8 P9 and then eth0 goes and grabs a bunch of its packets, then eth1, and eth2 do the same afterwards, so the received order ends up something like P1, P4, P7, P2, P5, P8, P3, P6, P9. In Ye Olde Dayes Of Yore, with one packet per interrupt at 10 Mb/sec, this type of configuration wouldn't reorder (or at least not as badly). The text probably is lacking in some detail, though. The real key is that the last sender before getting to the destination system has to do the round-robin striping. Most switches that I'm familiar with (again, never seen one, but willing to believe there is one) don't have round-robin as a load balance option for etherchannel, and thus won't evenly stripe traffic, but instead do some math on the packets so that a given "connection" isn't split across ports. That said, it's certainly plausible that, for a given set of N ethernets all enslaved to a single bonding balance-rr, the individual ethernets could get out of sync, as it were (e.g., one running a fuller tx ring, and thus running "behind" the others). If bonding is the only feeder of the devices, then for a continuous flow of traffic, all the slaves will generally receive packets (from the kernel, for transmission) at pretty much the same rate, and so they won't tend to get ahead or behind. I haven't investigated into this deeply for a few years, but this is my recollection of what happened with the tests I did then. I did testing with multiple 100Mb devices feeding either other sets of 100Mb devices or single gigabit devices. I'm willing to believe that things have changed, and an N feeding into one configuration can reorder, but I haven't seen it (or really looked for it; balance-rr isn't much the rage these days). -J --- -Jay Vosburgh, IBM Linux Technology Center, [EMAIL PROTECTED] - To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html