Hi tech@. Two machines (A and B) running recent 5.4-stable plugged into same switch.
A has: em0 at pci4 dev 0 function 0 "Intel 82573E" rev 0x03: msi, address 00:30:48:66:a0:ec em1 at pci5 dev 0 function 0 "Intel 82573L" rev 0x00: msi, address 00:30:48:66:a0:ed B has: bnx0 at pci2 dev 0 function 0 "Broadcom BCM5716" rev 0x20: apic 0 int 16 bnx1 at pci2 dev 0 function 1 "Broadcom BCM5716" rev 0x20: apic 0 int 17 bnx0: address b8:ac:6f:91:48:da brgphy0 at bnx0 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8 bnx1: address b8:ac:6f:91:48:db brgphy1 at bnx1 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8 Both servers have LACP trunk(4)s built on-top the above mentioned interfaces: A has: em0: flags=28b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST,NOINET6> mtu 1500 lladdr 00:30:48:66:a0:ec priority: 0 trunk: trunkdev trunk0 media: Ethernet autoselect (1000baseT full-duplex,master) status: active em1: flags=28b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST,NOINET6> mtu 1500 lladdr 00:30:48:66:a0:ec priority: 0 trunk: trunkdev trunk0 media: Ethernet autoselect (1000baseT full-duplex,master) status: active trunk0: flags=28943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,NOINET6> mtu 1500 lladdr 00:30:48:66:a0:ec priority: 0 trunk: trunkproto lacp trunk id: [(8000,00:30:48:66:a0:ec,402C,0000,0000), (0001,ec:30:91:25:c0:4f,03E8,0000,0000)] trunkport em1 active,collecting,distributing trunkport em0 active,collecting,distributing groups: trunk media: Ethernet autoselect status: active B has: bnx0: flags=28b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST,NOINET6> mtu 1500 lladdr b8:ac:6f:91:48:da priority: 0 trunk: trunkdev trunk0 media: Ethernet autoselect (1000baseT full-duplex) status: active bnx1: flags=28b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST,NOINET6> mtu 1500 lladdr b8:ac:6f:91:48:da priority: 0 trunk: trunkdev trunk0 media: Ethernet autoselect (1000baseT full-duplex,master) status: active trunk0: flags=28843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,NOINET6> mtu 1500 lladdr b8:ac:6f:91:48:da priority: 0 trunk: trunkproto lacp trunk id: [(8000,b8:ac:6f:91:48:da,402C,0000,0000), (0001,ec:30:91:25:c0:4f,03EA,0000,0000)] trunkport bnx1 active,collecting,distributing trunkport bnx0 active,collecting,distributing groups: trunk media: Ethernet autoselect status: active Now about the difference. The A receives on both em0 and em1 and transmits on em0: IFACE STATE DESC IPKTS IBYTES IERRS OPKTS OBYTES OERRS COLLS em0 up:U 2711 2859K 0 5593 5222K 0 0 em1 up:U 2867 2343K 0 10 3226 0 0 trunk0 up:U 5578 5202K 0 5603 5225K 0 0 The B receives *only* on bnx0 and transmits *only* on bnx1: IFACE STATE DESC IPKTS IBYTES IERRS OPKTS OBYTES OERRS COLLS bnx0 up:U 2873 2956K 0 2 977 0 0 bnx1 up:U 5 360 0 3119 2604K 0 0 trunk0 up:U 2878 2956K 0 3121 2605K 0 0 The only difference ifconfig shows, both em(4)s are master interfaces on A, but only bnx1 is a master interface on B (I haven't found any description of "master" media option in ifconfig man page, trunk man page saying about "master" but only wrt failover mode). Whole situation smells like trunk(4) receives on *all* master interfaces, but transmits on *first available* master. The question here is, why both em(4)s are master interfaces on A, but only bnx1 is master interface on B? Another question is, what is transmit hash policy for a trunk in LACP mode? If it matters, while testing on B, different MACs and different VLANs been used, but effect is same: bnx0 only receives, bnx1 only transmits. Anybody with trunking experience please speak up. Thanks, Alexey