Hi Ben,
On Dec 13, 2006, at 4:02 AM, Ben Rockwood wrote:
I've got a network on which a number of Galaxy systems are using
link aggregation on 2 e1000g interfaces. It works well. And thats
what puzzles me. None of the switches were configured to aggregate
ports, LACP isn't enabled on the switch or on the systems, and yet
the aggr's seem fine, we're not seeing errors on the ports, etc.
It seems to me that this shouldn't work.
Can anyone help me understand why aggr's work without the switch
having any knowledge of the configuration on the system? Does this
present problems that I haven't yet encountered or perhaps simply
don't know to look for? Given that we're using Dell PowerConect
switches which limit aggr groups to 8 even on a 48 port switch I'm
inclined to leave the setup alone unless I find a good reason to
stop aggregating in this way. Inquiring minds want to know.
It appears to be working, but there might be side effects which are
not obviously visible but can bite later. Let me try to describe what
happens in this case and these possible problems:
When LACP is turned off, Solaris assumes that a port is part of the
aggregation as soon as its link is up and its link speed and duplex
status compatible with the other ports of the aggregation. Packets
will be sent through the members of the aggregation according to the
configured outbound port policy. The switch will receive these
packets. Unicast packets will be delivered by the switch, but a first
problem is that broadcast packets sent by one of the constituent of
the aggregation will be sent by the switch to the other members of
the aggregation. This will cause unexpected broadcast packets to be
received by the host, which can cause problems such as these to show
up in your logs:
Feb 16 15:01:00 host ip: [ID 903730 kern.warning] WARNING: IP:
Hardware address 'xx:xx:xx:xx:xx:xx' trying to be our address
yyy.yyy.yyy.yyy!
On the path to the Solaris host, the misconfigured switch doesn't
know that the ports connected to the host are aggregated. However the
switch saw packets with a source address corresponding to the
aggregation MAC address coming from these ports. So the switch will
pick one of these ports to send packets to the host. Even though
packets appear to be flowing, it can be problematic since (a)
duplicate broadcast packets will be sent to the host, (b) traffic
might not be properly spread through the different ports of the
aggregation, and (c) packets for the same connection could arrive
from different NICs, which can cause reordering of packets, and a
mismatch between the interrupted CPUs and the CPUs to which squeues
are bound.
These side effects can cause performance problems, hard to diagnose
error messages, and in general suboptimal use of the hardware resources.
With LACP, such misconfigurations are a lot easier to detect, since
ports will not be enabled until the aggregation successfully
completes the LACP exchange with the remote peer. To avoid cases like
this, we're planning to enable LACP by default (see 6433652).
Let me know you have further questions.
Nicolas.
--
Nicolas Droux - Solaris Networking - Sun Microsystems, Inc.
[EMAIL PROTECTED] - http://blogs.sun.com/droux
_______________________________________________
networking-discuss mailing list
[email protected]
