Re: Dropped packets with bonded interface
On 15.11.2011 07:18, Albert Chin wrote: On Mon, Nov 14, 2011 at 11:56:20AM +0100, Stefan Bader wrote: My best guess about the dropped packages would be some that arrive before being set up... (like broadcasts) But even after bond0 is configured, up, and operational, the dropped packets continue. Hm, right. I see this as well. Though it seems to be quite minimal and I see a similar number on one of the enslaved interfaces. Not adding up completely. Just as a note that the recommendation for the bond-slaves option is none. This is because the bond interface should get the slaves added by the slave interface. Probably requiring auto lines for the slaves, too (the README and examples coming with the package should reflect this by now). The interface did not come up before because there was not network configuration set for the bond interface. Yeah. I read through the README and made the change. However, the issue of bond0 not being UP and the dropped packets remain even after this change. In my setup this seems to work. But I am only running in failover mode. Could be that the problem is related to the mode. From the logs it looks like at least the ip address is set and the interface is deemed up for bond0. But in the bonding status the mii status is down for bond0 but up for both of the slaves... Seems like being halfway through setup. I wonder whether ifdown -a followed by a ifup -a always end up in the stuck case or come up... -Stefan -- ubuntu-server mailing list ubuntu-server@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-server More info: https://wiki.ubuntu.com/ServerTeam
Re: Dropped packets with bonded interface
On 12.11.2011 11:07, Albert Chin wrote: On Sat, Nov 12, 2011 at 01:49:40PM +1100, James Gray wrote: On 12/11/2011, at 3:58 AM, Albert Chin wrote: I'm running 11.10 on an Intel SR2625URLXR system with an Intel S5520UR motherboard and an Intel E1G42ET Dual Port Server Adapter. I'm getting dropped packets on a bonded interface: $ ifconfig -a ... bond0 Link encap:Ethernet HWaddr 00:1b:21:b7:21:ea BROADCAST PROMISC MASTER MULTICAST MTU:1500 Metric:1 RX packets:2472 errors:0 dropped:2472 overruns:0 frame:0 TX packets:4 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:306528 (306.5 KB) TX bytes:272 (272.0 B) --- 8 --- Snipped $ cat /etc/network/interfaces ... auto bond0 iface bond0 inet manual bond-slaves none bond_mode 802.3ad bond_miimon 100 auto eth2 iface eth2 inet manual bond-master bond0 auto eth3 iface eth3 inet manual bond-master bond0 2. Why isn't the bond0 interface up? The dropped packets on bond0 are probably control packets the kernel is ignoring because you haven't enslaved any interfaces. Which leads to question 2. Try modifying the iface bond0 stanza in /etc/network/interfaces: auto bond0 iface bond0 inet manual bond-slaves eth2 eth3 # No slaves == nothing for bond0 to bind to! bond_mode 802.3ad bond_miimon 100 You might want to add an updelay in there too if your interfaces take a little while to synchronise with the switch (easy for the switch to block the port until spanning tree has done its thing unless you've specify portfast on the switch port). Then bounce the network and see what happens. When it's all running properly, you should see something like this: Thanks. I tried that but it didn't work. I updated bond0 in /etc/network/interfaces to: auto bond0 iface bond0 inet static address 10.191.62.2 netmask 255.255.255.0 broadcast 10.191.62.255 bond-slaves eth2 eth3 bond-primary eth2 eth3 bond-mode 802.3ad bond-lacp_rate fast bond-miimon 100 When I boot with this, I get: # ifconfig -a bond0 Link encap:Ethernet HWaddr 00:1b:21:b7:21:ea inet addr:10.191.62.2 Bcast:10.191.62.255 Mask:255.255.255.0 inet6 addr: fe80::21b:21ff:feb7:21ea/64 Scope:Link UP BROADCAST MASTER MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) eth2 Link encap:Ethernet HWaddr 00:1b:21:b7:21:ea UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) Memory:b242-b244 eth3 Link encap:Ethernet HWaddr 00:1b:21:b7:21:ea UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) Memory:b240-b242 # cat /proc/net/bonding/bond0 Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011) Bonding Mode: IEEE 802.3ad Dynamic link aggregation Transmit Hash Policy: layer2 (0) MII Status: down MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 802.3ad info LACP rate: fast Aggregator selection policy (ad_select): stable bond bond0 has no active aggregator Slave Interface: eth2 MII Status: up Speed: 1000 Mbps Duplex: full Link Failure Count: 1 Permanent HW addr: 00:1b:21:b7:21:ea Aggregator ID: N/A Slave queue ID: 0 Slave Interface: eth3 MII Status: up Speed: 1000 Mbps Duplex: full Link Failure Count: 1 Permanent HW addr: 00:1b:21:b7:21:eb Aggregator ID: N/A Slave queue ID: 0 What's odd is that after the system comes up with the above config, if I do the following: # ip link set dev bond0 up # ifenslave bond0 eth2 eth3 # ifconfig bond0 10.191.62.2 netmask 255.255.255.0 # ifconfig bond0 bond0 Link encap:Ethernet HWaddr 00:1b:21:b7:21:ea inet addr:10.191.62.2 Bcast:10.191.62.255 Mask:255.255.255.0 inet6 addr: fe80::21b:21ff:feb7:21ea/64 Scope:Link UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 RX packets:17 errors:0 dropped:17 overruns:0 frame:0 TX packets:27 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:2108 (2.1 KB) TX bytes:3126 (3.1 KB) # cat /proc/net/bonding
Re: Dropped packets with bonded interface
On Mon, Nov 14, 2011 at 11:56:20AM +0100, Stefan Bader wrote: My best guess about the dropped packages would be some that arrive before being set up... (like broadcasts) But even after bond0 is configured, up, and operational, the dropped packets continue. Just as a note that the recommendation for the bond-slaves option is none. This is because the bond interface should get the slaves added by the slave interface. Probably requiring auto lines for the slaves, too (the README and examples coming with the package should reflect this by now). The interface did not come up before because there was not network configuration set for the bond interface. Yeah. I read through the README and made the change. However, the issue of bond0 not being UP and the dropped packets remain even after this change. -- albert chin (ch...@thewrittenword.com) -- ubuntu-server mailing list ubuntu-server@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-server More info: https://wiki.ubuntu.com/ServerTeam
Re: Dropped packets with bonded interface
On Sat, Nov 12, 2011 at 04:07:22AM -0600, Albert Chin wrote: On Sat, Nov 12, 2011 at 01:49:40PM +1100, James Gray wrote: The dropped packets on bond0 are probably control packets the kernel is ignoring because you haven't enslaved any interfaces. Which leads to question 2. Try modifying the iface bond0 stanza in /etc/network/interfaces: auto bond0 iface bond0 inet manual bond-slaves eth2 eth3 # No slaves == nothing for bond0 to bind to! bond_mode 802.3ad bond_miimon 100 You might want to add an updelay in there too if your interfaces take a little while to synchronise with the switch (easy for the switch to block the port until spanning tree has done its thing unless you've specify portfast on the switch port). Then bounce the network and see what happens. When it's all running properly, you should see something like this: Thanks. I tried that but it didn't work. ... [[ snip snip ]] What's odd is that after the system comes up with the above config, if I do the following: # ip link set dev bond0 up # ifenslave bond0 eth2 eth3 # ifconfig bond0 10.191.62.2 netmask 255.255.255.0 # ifconfig bond0 bond0 Link encap:Ethernet HWaddr 00:1b:21:b7:21:ea inet addr:10.191.62.2 Bcast:10.191.62.255 Mask:255.255.255.0 inet6 addr: fe80::21b:21ff:feb7:21ea/64 Scope:Link UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 RX packets:17 errors:0 dropped:17 overruns:0 frame:0 TX packets:27 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:2108 (2.1 KB) TX bytes:3126 (3.1 KB) # cat /proc/net/bonding/bond0 [[ output snipped ]] So it seems something odd is going on. Notice that I continue to get dropped packets after this working config. I filed a bug. Looks like others are experiencing this as well. https://bugs.launchpad.net/ubuntu/+source/ifupdown/+bug/889423/ -- albert chin (ch...@thewrittenword.com) -- ubuntu-server mailing list ubuntu-server@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-server More info: https://wiki.ubuntu.com/ServerTeam
Re: Dropped packets with bonded interface
On Sat, Nov 12, 2011 at 01:49:40PM +1100, James Gray wrote: On 12/11/2011, at 3:58 AM, Albert Chin wrote: I'm running 11.10 on an Intel SR2625URLXR system with an Intel S5520UR motherboard and an Intel E1G42ET Dual Port Server Adapter. I'm getting dropped packets on a bonded interface: $ ifconfig -a ... bond0 Link encap:Ethernet HWaddr 00:1b:21:b7:21:ea BROADCAST PROMISC MASTER MULTICAST MTU:1500 Metric:1 RX packets:2472 errors:0 dropped:2472 overruns:0 frame:0 TX packets:4 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:306528 (306.5 KB) TX bytes:272 (272.0 B) --- 8 --- Snipped $ cat /etc/network/interfaces ... auto bond0 iface bond0 inet manual bond-slaves none bond_mode 802.3ad bond_miimon 100 auto eth2 iface eth2 inet manual bond-master bond0 auto eth3 iface eth3 inet manual bond-master bond0 2. Why isn't the bond0 interface up? The dropped packets on bond0 are probably control packets the kernel is ignoring because you haven't enslaved any interfaces. Which leads to question 2. Try modifying the iface bond0 stanza in /etc/network/interfaces: auto bond0 iface bond0 inet manual bond-slaves eth2 eth3 # No slaves == nothing for bond0 to bind to! bond_mode 802.3ad bond_miimon 100 You might want to add an updelay in there too if your interfaces take a little while to synchronise with the switch (easy for the switch to block the port until spanning tree has done its thing unless you've specify portfast on the switch port). Then bounce the network and see what happens. When it's all running properly, you should see something like this: Thanks. I tried that but it didn't work. I updated bond0 in /etc/network/interfaces to: auto bond0 iface bond0 inet static address 10.191.62.2 netmask 255.255.255.0 broadcast 10.191.62.255 bond-slaves eth2 eth3 bond-primary eth2 eth3 bond-mode 802.3ad bond-lacp_rate fast bond-miimon 100 When I boot with this, I get: # ifconfig -a bond0 Link encap:Ethernet HWaddr 00:1b:21:b7:21:ea inet addr:10.191.62.2 Bcast:10.191.62.255 Mask:255.255.255.0 inet6 addr: fe80::21b:21ff:feb7:21ea/64 Scope:Link UP BROADCAST MASTER MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) eth2 Link encap:Ethernet HWaddr 00:1b:21:b7:21:ea UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) Memory:b242-b244 eth3 Link encap:Ethernet HWaddr 00:1b:21:b7:21:ea UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) Memory:b240-b242 # cat /proc/net/bonding/bond0 Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011) Bonding Mode: IEEE 802.3ad Dynamic link aggregation Transmit Hash Policy: layer2 (0) MII Status: down MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 802.3ad info LACP rate: fast Aggregator selection policy (ad_select): stable bond bond0 has no active aggregator Slave Interface: eth2 MII Status: up Speed: 1000 Mbps Duplex: full Link Failure Count: 1 Permanent HW addr: 00:1b:21:b7:21:ea Aggregator ID: N/A Slave queue ID: 0 Slave Interface: eth3 MII Status: up Speed: 1000 Mbps Duplex: full Link Failure Count: 1 Permanent HW addr: 00:1b:21:b7:21:eb Aggregator ID: N/A Slave queue ID: 0 What's odd is that after the system comes up with the above config, if I do the following: # ip link set dev bond0 up # ifenslave bond0 eth2 eth3 # ifconfig bond0 10.191.62.2 netmask 255.255.255.0 # ifconfig bond0 bond0 Link encap:Ethernet HWaddr 00:1b:21:b7:21:ea inet addr:10.191.62.2 Bcast:10.191.62.255 Mask:255.255.255.0 inet6 addr: fe80::21b:21ff:feb7:21ea/64 Scope:Link UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 RX packets:17 errors:0 dropped:17 overruns:0 frame:0 TX packets:27 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:2108 (2.1 KB) TX bytes:3126 (3.1 KB) # cat /proc/net/bonding/bond0 Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011) Bonding Mode: IEEE 802.3ad
Dropped packets with bonded interface
I'm running 11.10 on an Intel SR2625URLXR system with an Intel S5520UR motherboard and an Intel E1G42ET Dual Port Server Adapter. I'm getting dropped packets on a bonded interface: $ ifconfig -a ... bond0 Link encap:Ethernet HWaddr 00:1b:21:b7:21:ea BROADCAST PROMISC MASTER MULTICAST MTU:1500 Metric:1 RX packets:2472 errors:0 dropped:2472 overruns:0 frame:0 TX packets:4 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:306528 (306.5 KB) TX bytes:272 (272.0 B) br0 Link encap:Ethernet HWaddr 00:1b:21:b7:21:ea inet6 addr: fe80::21b:21ff:feb7:21ea/64 Scope:Link UP BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:7 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:554 (554.0 B) ... eth2 Link encap:Ethernet HWaddr 00:1b:21:b7:21:ea UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:1236 errors:0 dropped:0 overruns:0 frame:0 TX packets:2 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:153264 (153.2 KB) TX bytes:116 (116.0 B) Memory:b242-b244 eth3 Link encap:Ethernet HWaddr 00:1b:21:b7:21:ea UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:1236 errors:0 dropped:0 overruns:0 frame:0 TX packets:2 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:153264 (153.2 KB) TX bytes:156 (156.0 B) Memory:b240-b242 $ cat /etc/network/interfaces ... auto bond0 iface bond0 inet manual bond-slaves none bond_mode 802.3ad bond_miimon 100 auto eth2 iface eth2 inet manual bond-master bond0 auto eth3 iface eth3 inet manual bond-master bond0 auto br0 iface br0 inet manual bridge_ports bond0 bridge_stp off eth2 and eth3 are on the Intel E1G42ET Dual Port Server Adapter. The switch these two ports are connected to is a Summit 400-48t, with the ports they're connected to configured for LACP: Summit400-48t:1 # show configuration ... configure sharing address-based ip-source ... enable sharing 17 grouping 17,21 dynamic ... Summit400-48t:1 # show ports sharing Load Sharing Monitor ConfigCurrentLd ShareLd Share Link Link MasterMaster TypeGroup Status Ups == 17 dy 17 ND 689 dy 21 ND 115 Link Status: (A) Active, (D) Disabled, (LB) Loopback, (ND) Not Distributing (NP) Not Present, (R) Ready Ld Share Type: (a) address based, (dy) dynamic 1. If eth2 and eth3, which make up bond0, show no dropped packets, why am I seeing dropped packets on bond0? I've tested the network cables and they're ok. I've tested the two ports on the Intel E1G42ET Dual Port Server Adapter and they're ok. I've tested the ports on the switch and they're ok. I'm only seeing dropped packets when I enable bonding. 2. Why isn't the bond0 interface up? -- albert chin (ch...@thewrittenword.com) -- ubuntu-server mailing list ubuntu-server@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-server More info: https://wiki.ubuntu.com/ServerTeam
Re: Dropped packets with bonded interface
Hi Albert, See below… On 12/11/2011, at 3:58 AM, Albert Chin wrote: I'm running 11.10 on an Intel SR2625URLXR system with an Intel S5520UR motherboard and an Intel E1G42ET Dual Port Server Adapter. I'm getting dropped packets on a bonded interface: $ ifconfig -a ... bond0 Link encap:Ethernet HWaddr 00:1b:21:b7:21:ea BROADCAST PROMISC MASTER MULTICAST MTU:1500 Metric:1 RX packets:2472 errors:0 dropped:2472 overruns:0 frame:0 TX packets:4 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:306528 (306.5 KB) TX bytes:272 (272.0 B) --- 8 --- Snipped $ cat /etc/network/interfaces ... auto bond0 iface bond0 inet manual bond-slaves none bond_mode 802.3ad bond_miimon 100 auto eth2 iface eth2 inet manual bond-master bond0 auto eth3 iface eth3 inet manual bond-master bond0 2. Why isn't the bond0 interface up? The dropped packets on bond0 are probably control packets the kernel is ignoring because you haven't enslaved any interfaces. Which leads to question 2. Try modifying the iface bond0 stanza in /etc/network/interfaces: auto bond0 iface bond0 inet manual bond-slaves eth2 eth3 # No slaves == nothing for bond0 to bind to! bond_mode 802.3ad bond_miimon 100 You might want to add an updelay in there too if your interfaces take a little while to synchronise with the switch (easy for the switch to block the port until spanning tree has done its thing unless you've specify portfast on the switch port). Then bounce the network and see what happens. When it's all running properly, you should see something like this: # cat /proc/net/bonding/bond0 Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009) Bonding Mode: IEEE 802.3ad Dynamic link aggregation Transmit Hash Policy: layer2 (0) MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 802.3ad info LACP rate: slow Aggregator selection policy (ad_select): stable Active Aggregator Info: Aggregator ID: 1 Number of ports: 2 Actor Key: 17 Partner Key: 53 Partner Mac Address: 74:ea:3a:ba:35:e4 Slave Interface: eth0 MII Status: up Link Failure Count: 0 Permanent HW addr: 00:08:9b:c4:a6:f4 Aggregator ID: 1 Slave Interface: eth1 MII Status: up Link Failure Count: 0 Permanent HW addr: 00:08:9b:c4:a6:f5 Aggregator ID: 1 As always, there's some documentation to read: https://help.ubuntu.com/community/UbuntuBonding Cheers, James smime.p7s Description: S/MIME cryptographic signature -- ubuntu-server mailing list ubuntu-server@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-server More info: https://wiki.ubuntu.com/ServerTeam
