Re: Multipath and iSCSI on 11.10
On Fri, Dec 23, 2011 at 07:20:01PM -0600, Albert Chin wrote:
> I have device-mapper-multipath configured with iSCSI. I have two
> interfaces configured to communicate with the iSCSI server. I discover
> and connect with the iSCSI targets as follows:
> (1) # iscsiadm -m discovery -t st -p 10.191.61.1:3260 -I iface1 -P 1
> (2) # iscsiadm -m discovery -t st -p 10.191.62.1:3260 -I iface2 -P 1
> (3) # iscsiadm -m node -p 10.191.61.1:3260 -I iface1 -l
> (4) # iscsiadm -m node -p 10.191.62.1:3260 -I iface2 -l
> # cat /etc/multipath.conf
> defaults {
> selector "round-robin 0"
> udev_dir /dev
> user_friendly_names yes
> }
> [[ snip snip ]]
> # multipath -ll
> winry (3600144f057774b004eeab58a0001) dm-24 OI,COMSTAR
> size=500G features='0' hwhandler='0' wp=rw
> |-+- policy='round-robin 0' prio=1 status=active
> | `- 12:0:0:0 sdd 8:48 active ready running
> `-+- policy='round-robin 0' prio=1 status=enabled
> `- 41:0:0:0 sdae 65:224 active ready running
> sai (3600144f057774b004ebf539e001d) dm-9 OI,COMSTAR
> size=500G features='0' hwhandler='0' wp=rw
> |-+- policy='round-robin 0' prio=1 status=active
> | `- 17:0:0:0 sdv 65:80 active ready running
> `-+- policy='round-robin 0' prio=1 status=enabled
> `- 54:0:0:0 sdai 66:32 active ready running
> ...
>
> 1. Based on the above multipath -ll output, it seems that I am not
>load-balancing across both NICs. Why not?
Adding path_grouping_policy multibus fixed this.
> 2. When I boot the system, the initiator does not log in to any
>target. Therefore, I need to rerun (3) and (4) above. Why?
Still not sure about this one.
--
albert chin (ch...@thewrittenword.com)
--
ubuntu-server mailing list
ubuntu-server@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server
More info: https://wiki.ubuntu.com/ServerTeam
Multipath and iSCSI on 11.10
I have device-mapper-multipath configured with iSCSI. I have two
interfaces configured to communicate with the iSCSI server. I discover
and connect with the iSCSI targets as follows:
(1) # iscsiadm -m discovery -t st -p 10.191.61.1:3260 -I iface1 -P 1
(2) # iscsiadm -m discovery -t st -p 10.191.62.1:3260 -I iface2 -P 1
(3) # iscsiadm -m node -p 10.191.61.1:3260 -I iface1 -l
(4) # iscsiadm -m node -p 10.191.62.1:3260 -I iface2 -l
# cat /etc/multipath.conf
defaults {
selector "round-robin 0"
udev_dir /dev
user_friendly_names yes
}
blacklist {
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^hd[a-z]"
devnode "^sda[0-9]"
devnode "^sdb[0-9]"
}
multipaths {
multipath {
wwid 3600144f057774b004ebf539e001d
alias sai
}
multipath {
wwid 3600144f057774b004eeab58a0001
alias winry
}
multipath {
device {
vendor OI
product COMSTAR
}
}
}
# multipath -ll
winry (3600144f057774b004eeab58a0001) dm-24 OI,COMSTAR
size=500G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 12:0:0:0 sdd 8:48 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
`- 41:0:0:0 sdae 65:224 active ready running
sai (3600144f057774b004ebf539e001d) dm-9 OI,COMSTAR
size=500G features='0' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| `- 17:0:0:0 sdv 65:80 active ready running
`-+- policy='round-robin 0' prio=1 status=enabled
`- 54:0:0:0 sdai 66:32 active ready running
...
1. Based on the above multipath -ll output, it seems that I am not
load-balancing across both NICs. Why not?
2. When I boot the system, the initiator does not log in to any
target. Therefore, I need to rerun (3) and (4) above. Why?
--
albert chin (ch...@thewrittenword.com)
--
ubuntu-server mailing list
ubuntu-server@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server
More info: https://wiki.ubuntu.com/ServerTeam
Re: Dropped packets with bonded interface
On Mon, Nov 14, 2011 at 11:56:20AM +0100, Stefan Bader wrote: > My best guess about the dropped packages would be some that arrive > before being set up... (like broadcasts) But even after bond0 is configured, up, and operational, the dropped packets continue. > Just as a note that the recommendation for the bond-slaves option is > none. This is because the bond interface should get the slaves added > by the slave interface. Probably requiring auto lines for the > slaves, too (the README and examples coming with the package should > reflect this by now). The interface did not come up before because > there was not network configuration set for the bond interface. Yeah. I read through the README and made the change. However, the issue of bond0 not being UP and the dropped packets remain even after this change. -- albert chin (ch...@thewrittenword.com) -- ubuntu-server mailing list ubuntu-server@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-server More info: https://wiki.ubuntu.com/ServerTeam
Re: Dropped packets with bonded interface
On Sat, Nov 12, 2011 at 04:07:22AM -0600, Albert Chin wrote: > On Sat, Nov 12, 2011 at 01:49:40PM +1100, James Gray wrote: > > The dropped packets on bond0 are probably control packets the kernel > > is ignoring because you haven't enslaved any interfaces. Which > > leads to question 2. Try modifying the "iface bond0" stanza in > > /etc/network/interfaces: > > > > auto bond0 > > iface bond0 inet manual > > bond-slaves eth2 eth3 # No slaves == nothing for bond0 to bind to! > > bond_mode 802.3ad > > bond_miimon 100 > > > > You might want to add an "updelay" in there too if your interfaces > > take a little while to synchronise with the switch (easy for the > > switch to block the port until spanning tree has done its thing > > unless you've specify "portfast" on the switch port). Then bounce > > the network and see what happens. When it's all running properly, > > you should see something like this: > > Thanks. I tried that but it didn't work. ... > [[ snip snip ]] > > What's odd is that after the system comes up with the above config, if > I do the following: > # ip link set dev bond0 up > # ifenslave bond0 eth2 eth3 > # ifconfig bond0 10.191.62.2 netmask 255.255.255.0 > # ifconfig bond0 > bond0 Link encap:Ethernet HWaddr 00:1b:21:b7:21:ea > inet addr:10.191.62.2 Bcast:10.191.62.255 Mask:255.255.255.0 > inet6 addr: fe80::21b:21ff:feb7:21ea/64 Scope:Link > UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 > RX packets:17 errors:0 dropped:17 overruns:0 frame:0 > TX packets:27 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:0 > RX bytes:2108 (2.1 KB) TX bytes:3126 (3.1 KB) > > # cat /proc/net/bonding/bond0 > [[ output snipped ]] > > So it seems something odd is going on. Notice that I continue to get > dropped packets after this "working" config. I filed a bug. Looks like others are experiencing this as well. https://bugs.launchpad.net/ubuntu/+source/ifupdown/+bug/889423/ -- albert chin (ch...@thewrittenword.com) -- ubuntu-server mailing list ubuntu-server@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-server More info: https://wiki.ubuntu.com/ServerTeam
Re: Dropped packets with bonded interface
On Sat, Nov 12, 2011 at 01:49:40PM +1100, James Gray wrote: > On 12/11/2011, at 3:58 AM, Albert Chin wrote: > > > I'm running 11.10 on an Intel SR2625URLXR system with an Intel S5520UR > > motherboard and an Intel E1G42ET Dual Port Server Adapter. I'm getting > > dropped packets on a bonded interface: > > > > $ ifconfig -a > > ... > > bond0 Link encap:Ethernet HWaddr 00:1b:21:b7:21:ea > > BROADCAST PROMISC MASTER MULTICAST MTU:1500 Metric:1 > > RX packets:2472 errors:0 dropped:2472 overruns:0 frame:0 > > TX packets:4 errors:0 dropped:0 overruns:0 carrier:0 > > collisions:0 txqueuelen:0 > > RX bytes:306528 (306.5 KB) TX bytes:272 (272.0 B) > > --- >8 --- Snipped > > > $ cat /etc/network/interfaces > > ... > > auto bond0 > > iface bond0 inet manual > > bond-slaves none > > bond_mode 802.3ad > > bond_miimon 100 > > > > auto eth2 > > iface eth2 inet manual > > bond-master bond0 > > > > auto eth3 > > iface eth3 inet manual > > bond-master bond0 > > > > 2. Why isn't the bond0 interface "up"? > > The dropped packets on bond0 are probably control packets the kernel > is ignoring because you haven't enslaved any interfaces. Which > leads to question 2. Try modifying the "iface bond0" stanza in > /etc/network/interfaces: > > auto bond0 > iface bond0 inet manual > bond-slaves eth2 eth3 # No slaves == nothing for bond0 to bind to! > bond_mode 802.3ad > bond_miimon 100 > > You might want to add an "updelay" in there too if your interfaces > take a little while to synchronise with the switch (easy for the > switch to block the port until spanning tree has done its thing > unless you've specify "portfast" on the switch port). Then bounce > the network and see what happens. When it's all running properly, > you should see something like this: Thanks. I tried that but it didn't work. I updated bond0 in /etc/network/interfaces to: auto bond0 iface bond0 inet static address 10.191.62.2 netmask 255.255.255.0 broadcast 10.191.62.255 bond-slaves eth2 eth3 bond-primary eth2 eth3 bond-mode 802.3ad bond-lacp_rate fast bond-miimon 100 When I boot with this, I get: # ifconfig -a bond0 Link encap:Ethernet HWaddr 00:1b:21:b7:21:ea inet addr:10.191.62.2 Bcast:10.191.62.255 Mask:255.255.255.0 inet6 addr: fe80::21b:21ff:feb7:21ea/64 Scope:Link UP BROADCAST MASTER MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) eth2 Link encap:Ethernet HWaddr 00:1b:21:b7:21:ea UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) Memory:b242-b244 eth3 Link encap:Ethernet HWaddr 00:1b:21:b7:21:ea UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B) Memory:b240-b242 # cat /proc/net/bonding/bond0 Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011) Bonding Mode: IEEE 802.3ad Dynamic link aggregation Transmit Hash Policy: layer2 (0) MII Status: down MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 802.3ad info LACP rate: fast Aggregator selection policy (ad_select): stable bond bond0 has no active aggregator Slave Interface: eth2 MII Status: up Speed: 1000 Mbps Duplex: full Link Failure Count: 1 Permanent HW addr: 00:1b:21:b7:21:ea Aggregator ID: N/A Slave queue ID: 0 Slave Interface: eth3 MII Status: up Speed: 1000 Mbps Duplex: full Link Failure Count: 1 Permanent HW addr: 00:1b:21:b7:21:eb Aggregator ID: N/A Slave queue ID: 0 What's odd is that after the system comes up with the above config, if I do the following: # ip link set dev bond0 up # ifenslave bond0 eth2 eth3 # ifconfig bond0 10.191.62.2 netmask 255.255.255.0 # ifconfig bond0 bond0 Link encap:Ethernet HWaddr 00:1b:21:b7:21:ea inet addr:10.191.62.2 Bcast:10.191.62.255 Mask:255.255.255.0 inet6 addr: fe80::21b:21ff:feb7:21ea/64 Scope:Link
Re: Dropped packets with bonded interface
On Fri, Nov 11, 2011 at 10:58:41AM -0600, Albert Chin wrote: > I'm running 11.10 on an Intel SR2625URLXR system with an Intel S5520UR > motherboard and an Intel E1G42ET Dual Port Server Adapter. I'm getting > dropped packets on a bonded interface: > > [[ ifconfig and /etc/network/interfaces output snipped ]] Forgot to list output of /proc/net/bonding/bond0: $ cat /proc/net/bonding/bond0 Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011) Bonding Mode: IEEE 802.3ad Dynamic link aggregation Transmit Hash Policy: layer2 (0) MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0 802.3ad info LACP rate: slow Aggregator selection policy (ad_select): stable bond bond0 has no active aggregator Slave Interface: eth2 MII Status: up Speed: 100 Mbps Duplex: full Link Failure Count: 0 Permanent HW addr: 00:1b:21:b7:21:ea Aggregator ID: N/A Slave queue ID: 0 Slave Interface: eth3 MII Status: up Speed: 100 Mbps Duplex: full Link Failure Count: 0 Permanent HW addr: 00:1b:21:b7:21:eb Aggregator ID: N/A Slave queue ID: 0 -- albert chin (ch...@thewrittenword.com) -- ubuntu-server mailing list ubuntu-server@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-server More info: https://wiki.ubuntu.com/ServerTeam
Dropped packets with bonded interface
I'm running 11.10 on an Intel SR2625URLXR system with an Intel S5520UR motherboard and an Intel E1G42ET Dual Port Server Adapter. I'm getting dropped packets on a bonded interface: $ ifconfig -a ... bond0 Link encap:Ethernet HWaddr 00:1b:21:b7:21:ea BROADCAST PROMISC MASTER MULTICAST MTU:1500 Metric:1 RX packets:2472 errors:0 dropped:2472 overruns:0 frame:0 TX packets:4 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:306528 (306.5 KB) TX bytes:272 (272.0 B) br0 Link encap:Ethernet HWaddr 00:1b:21:b7:21:ea inet6 addr: fe80::21b:21ff:feb7:21ea/64 Scope:Link UP BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:7 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:554 (554.0 B) ... eth2 Link encap:Ethernet HWaddr 00:1b:21:b7:21:ea UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:1236 errors:0 dropped:0 overruns:0 frame:0 TX packets:2 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:153264 (153.2 KB) TX bytes:116 (116.0 B) Memory:b242-b244 eth3 Link encap:Ethernet HWaddr 00:1b:21:b7:21:ea UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:1236 errors:0 dropped:0 overruns:0 frame:0 TX packets:2 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:153264 (153.2 KB) TX bytes:156 (156.0 B) Memory:b240-b242 $ cat /etc/network/interfaces ... auto bond0 iface bond0 inet manual bond-slaves none bond_mode 802.3ad bond_miimon 100 auto eth2 iface eth2 inet manual bond-master bond0 auto eth3 iface eth3 inet manual bond-master bond0 auto br0 iface br0 inet manual bridge_ports bond0 bridge_stp off eth2 and eth3 are on the Intel E1G42ET Dual Port Server Adapter. The switch these two ports are connected to is a Summit 400-48t, with the ports they're connected to configured for LACP: Summit400-48t:1 # show configuration ... configure sharing address-based ip-source ... enable sharing 17 grouping 17,21 dynamic ... Summit400-48t:1 # show ports sharing Load Sharing Monitor ConfigCurrentLd ShareLd Share Link Link MasterMaster TypeGroup Status Ups == 17 dy 17 ND 689 dy 21 ND 115 Link Status: (A) Active, (D) Disabled, (LB) Loopback, (ND) Not Distributing (NP) Not Present, (R) Ready Ld Share Type: (a) address based, (dy) dynamic 1. If eth2 and eth3, which make up bond0, show no dropped packets, why am I seeing dropped packets on bond0? I've tested the network cables and they're ok. I've tested the two ports on the Intel E1G42ET Dual Port Server Adapter and they're ok. I've tested the ports on the switch and they're ok. I'm only seeing dropped packets when I enable bonding. 2. Why isn't the bond0 interface "up"? -- albert chin (ch...@thewrittenword.com) -- ubuntu-server mailing list ubuntu-server@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-server More info: https://wiki.ubuntu.com/ServerTeam
