Re: [openib-general] [openfabrics-ewg] Problems with OFED IPoIB HA on SLES10
Hi Scott, You have an old version of ipoibtools package (ipoib_ha.pl). All issues you are talking about were fixed in the new version which will be available in OFED-1.1-rc7. You can also download it from SVN: https://openib.org/svn/gen2/branches/1.1/src/userspace/ipoibtools Thanks, Regards, Vladimir On Tue, 2006-10-03 at 14:53 -0700, Scott Weitzenkamp (sweitzen) wrote: Vlad, thaks for the fast response. I have some followup questions about configuring IPoIB HA, see below. 3) I got IPoIB HA working on SLES 10, but the documentation is a little lacking. Looks like I have to put the same IP address in ifcfg-ib0 and ifcfg-ib1, is this correct? Yes, IP address should be the same. Actually the configuration of the secondary interface does not matter. The High Availability daemon reads the configuration of the primary interface and migrates it between the interfaces in case of failure. If I don't have an ifcfg-ib1 file, then ipoib_ha.pl won't start. If I don't have an ifcfg-ib1, then ipoib_ha.pl won't start. I would prefer to not configure ifcfg-ib1 since I don't plan to use it. # ipoib_ha.pl --with-arping --with-multicast -v Can't open conf /etc/sysconfig/network/ifcfg-ib1: No such file or directory Can't open conf /etc/sysconfig/network/ifcfg-ib1: No such file or directory Can't open conf /etc/sysconfig/network/ifcfg-ib1: No such file or directory Can't open conf /etc/sysconfig/network/ifcfg-ib1: No such file or directory Can't open conf /etc/sysconfig/network/ifcfg-ib1: No such file or directory ... If I put different IP addresses in ifcfg-ib0 and ifcfg-ib1, then the ifcfg-ib1 IP address is used for both ib0 and ib1! # pwd /etc/sysconfig/network # cat ifcfg-ib0 DEVICE=ib0 BOOTPROTO=static IPADDR=192.168.2.46 NETMASK=255.255.255.0 > # cat ifcfg-ib1 DEVICE=ib1 BOOTPROTO=static IPADDR=192.168.6.46 NETMASK=255.255.255.0 > # /etc/init.d/openibd start Loading HCA driver and Access Layer: [ OK ] Setting up InfiniBand network interfaces: ib0 device: Mellanox Technologies MT25208 InfiniHost III Ex (Tavor com patibility mode) (rev 20) ib0 configuration: ib1 Bringing up interface ib0: [ OK ] ib1 device: Mellanox Technologies MT25208 InfiniHost III Ex (Tavor com patibility mode) (rev 20) Bringing up interface ib1: [ OK ] Setting up service network . . . [ done ] # ifconfig ib0 ib0 Link encap:UNSPEC HWaddr 00-00-04-04-FE-80-00-00-00-00-00-00-00-00-00 -00 inet addr:192.168.6.46 Bcast:192.168.6.255 Mask:255.255.255.0 inet6 addr: fe80::202:c902:21:700d/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:2044 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:3 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:128 RX bytes:0 (0.0 b) TX bytes:224 (224.0 b) # ifconfig ib1 ib1 Link encap:UNSPEC HWaddr 00-00-04-05-FE-80-00-00-00-00-00-00-00-00-00 -00 inet addr:192.168.6.46 Bcast:192.168.6.255 Mask:255.255.255.0 inet6 addr: fe80::202:c902:21:700e/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:2044 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:4 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:128 RX bytes:0 (0.0 b) TX bytes:304 (304.0 b) Notice how both ib0 and ib1 have the IP address from ifcfg-ib1. This contradicts this info from ipoib_release_notes.txt: b. The ib1 interface uses the configuration script of ib0. Scott ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [openfabrics-ewg] Problems with OFED IPoIB HA on SLES10
}
{:ib_mad:ib_mad_complete_send_wr+421}
{:ib_mad:ib_mad_completion_handler+947}
{:ib_mad:ib_mad_completion_handler+0}
{run_workqueue+153}
{worker_thread+0}
{keventd_create_kthread+0}
{worker_thread+265}
{__wake_up_common+62}
{default_wake_function+0}
{keventd_create_kthread+0}
{kthread+236}
{child_rip+8}
{keventd_create_kthread+0}
{kthread+0}
{child_rip+0}
Code: f0 ff 0f 0f 88 29 01 00 00 c3 fa f0 ff 0f 0f 88 2a 01
00 00RIP {_spin_lock_irqsave+3} RSP
Scott Weitzenkamp
SQA and Release Manager
Server Virtualization Business
Unit
Cisco Systems
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Scott
Weitzenkamp (sweitzen)Sent: Tuesday, October 03, 2006 2:53
PMTo: Vladimir SokolovskyCc: EWG;
openib-GeneralSubject: Re: [openib-general] [openfabrics-ewg]
Problems with OFED IPoIB HA on SLES10
Vlad, thaks for the fast response. I have some
followup questions about configuring IPoIB HA, see below.
3) I got IPoIB HA
working on SLES 10, but the documentation is a little lacking.
Looks like I have to put the same IP address in ifcfg-ib0 and ifcfg-ib1, is
this
correct?
Yes, IP address should be the same. Actually the configuration of the
secondary interface does not
matter. The High Availability
daemon reads the configuration of the primary interface and migrates it
between the interfaces in case of
failure. If
I don't have an ifcfg-ib1 file, then ipoib_ha.pl won't start.
If I don't have an ifcfg-ib1, then
ipoib_ha.pl won't start. I would prefer to not configure ifcfg-ib1 since
I don't plan to use it.
# ipoib_ha.pl --with-arping
--with-multicast -vCan't open conf /etc/sysconfig/network/ifcfg-ib1: No
such file or directoryCan't open conf /etc/sysconfig/network/ifcfg-ib1: No
such file or directoryCan't open conf /etc/sysconfig/network/ifcfg-ib1: No
such file or directoryCan't open conf /etc/sysconfig/network/ifcfg-ib1: No
such file or directoryCan't open conf /etc/sysconfig/network/ifcfg-ib1: No
such file or directory...
If I put
different IP addresses in ifcfg-ib0 and ifcfg-ib1, then the ifcfg-ib1 IP
address is used for both ib0 and ib1!
#
pwd/etc/sysconfig/network# cat
ifcfg-ib0DEVICE=ib0BOOTPROTO=staticIPADDR=192.168.2.46NETMASK=255.255.255.0>#
cat
ifcfg-ib1DEVICE=ib1BOOTPROTO=staticIPADDR=192.168.6.46NETMASK=255.255.255.0>#
/etc/init.d/openibd startLoading HCA driver and Access
Layer:
[ OK ]Setting up InfiniBand network
interfaces: ib0
device: Mellanox Technologies MT25208 InfiniHost III Ex (Tavor
compatibility mode) (rev 20)
ib0 configuration: ib1Bringing up
interface
ib0:
[ OK ]
ib1 device: Mellanox Technologies MT25208
InfiniHost III Ex (Tavor compatibility mode) (rev 20)Bringing up
interface
ib1:
[ OK ]Setting up service network . .
.
[ done ]# ifconfig
ib0ib0 Link
encap:UNSPEC HWaddr
00-00-04-04-FE-80-00-00-00-00-00-00-00-00-00-00
inet addr:192.168.6.46 Bcast:192.168.6.255
Mask:255.255.255.0
inet6 addr: fe80::202:c902:21:700d/64
Scope:Link UP
BROADCAST RUNNING MULTICAST MTU:2044
Metric:1 RX
packets:0 errors:0 dropped:0 overruns:0
frame:0 TX packets:3
errors:0 dropped:0 overruns:0
carrier:0
collisions:0
txqueuelen:128 RX
bytes:0 (0.0 b) TX bytes:224 (224.0 b)
# ifconfig
ib1ib1 Link
encap:UNSPEC HWaddr
00-00-04-05-FE-80-00-00-00-00-00-00-00-00-00-00
inet addr:192.168.6.46 Bcast:192.168.6.255
Mask:255.255.255.0
inet6 addr: fe80::202:c902:21:700e/64
Scope:Link UP
BROADCAST RUNNING MULTICAST MTU:2044
Metric:1 RX
packets:0 errors:0 dropped:0 overruns:0
frame:0 TX packets:4
errors:0 dropped:0 overruns:0
carrier:0
collisions:0
txqueuelen:128 RX
bytes:0 (0.0 b) TX bytes:304 (304.0 b)
Notice how both ib0 and ib1
have the IP address from ifcfg-ib1. This contradicts this info from
ipoib_release_notes.txt:
b.
The ib1 interface uses the configuration script of
ib0.
Scott
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [openfabrics-ewg] Problems with OFED IPoIB HA on SLES10
Vlad, thaks for the fast response. I have some followup questions about configuring IPoIB HA, see below. 3) I got IPoIB HA working on SLES 10, but the documentation is a little lacking. Looks like I have to put the same IP address in ifcfg-ib0 and ifcfg-ib1, is this correct? Yes, IP address should be the same. Actually the configuration of the secondary interface does not matter. The High Availability daemon reads the configuration of the primary interface and migrates it between the interfaces in case of failure. If I don't have an ifcfg-ib1 file, then ipoib_ha.pl won't start. If I don't have an ifcfg-ib1, then ipoib_ha.pl won't start. I would prefer to not configure ifcfg-ib1 since I don't plan to use it. # ipoib_ha.pl --with-arping --with-multicast -vCan't open conf /etc/sysconfig/network/ifcfg-ib1: No such file or directoryCan't open conf /etc/sysconfig/network/ifcfg-ib1: No such file or directoryCan't open conf /etc/sysconfig/network/ifcfg-ib1: No such file or directoryCan't open conf /etc/sysconfig/network/ifcfg-ib1: No such file or directoryCan't open conf /etc/sysconfig/network/ifcfg-ib1: No such file or directory... If I put different IP addresses in ifcfg-ib0 and ifcfg-ib1, then the ifcfg-ib1 IP address is used for both ib0 and ib1! # pwd/etc/sysconfig/network# cat ifcfg-ib0DEVICE=ib0BOOTPROTO=staticIPADDR=192.168.2.46NETMASK=255.255.255.0># cat ifcfg-ib1DEVICE=ib1BOOTPROTO=staticIPADDR=192.168.6.46NETMASK=255.255.255.0># /etc/init.d/openibd startLoading HCA driver and Access Layer: [ OK ]Setting up InfiniBand network interfaces: ib0 device: Mellanox Technologies MT25208 InfiniHost III Ex (Tavor compatibility mode) (rev 20) ib0 configuration: ib1Bringing up interface ib0: [ OK ] ib1 device: Mellanox Technologies MT25208 InfiniHost III Ex (Tavor compatibility mode) (rev 20)Bringing up interface ib1: [ OK ]Setting up service network . . . [ done ]# ifconfig ib0ib0 Link encap:UNSPEC HWaddr 00-00-04-04-FE-80-00-00-00-00-00-00-00-00-00-00 inet addr:192.168.6.46 Bcast:192.168.6.255 Mask:255.255.255.0 inet6 addr: fe80::202:c902:21:700d/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:2044 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:3 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:128 RX bytes:0 (0.0 b) TX bytes:224 (224.0 b) # ifconfig ib1ib1 Link encap:UNSPEC HWaddr 00-00-04-05-FE-80-00-00-00-00-00-00-00-00-00-00 inet addr:192.168.6.46 Bcast:192.168.6.255 Mask:255.255.255.0 inet6 addr: fe80::202:c902:21:700e/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:2044 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:4 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:128 RX bytes:0 (0.0 b) TX bytes:304 (304.0 b) Notice how both ib0 and ib1 have the IP address from ifcfg-ib1. This contradicts this info from ipoib_release_notes.txt: b. The ib1 interface uses the configuration script of ib0. Scott ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [openfabrics-ewg] Problems with OFED IPoIB HA on SLES10
Hi Scott,
Please see my comments below:
On Mon, 2006-10-02 at 23:24 -0700, Scott Weitzenkamp (sweitzen) wrote:
Vlad,
I filed a bug for these issues.
1) If I start IPoIB HA with ib0 IB port shut down (from IB switch) and ib1 IB port enabled, then IPoIB does not work because "ip monitor link all" does not report NO-CARRIER at startup like ipoib_ha.pl is looking for. This is a major hole.
Fixed, will be updated in OFED-1.1-rc7.
2) /etc/init.d/openibd runs ipoib_ha.pl with its stdout and stderr redirected to /dev/null, should we run with -v for verbose instead and redirect log file to /var/log?
# fgrep ipoib_ha.pl /etc/init.d/openibd
ipoib_ha.pl -p ${PRIMARY_IPOIB_DEV} -s ${SECONDARY_IPOIB_DEV} --
with-arping --with-multicast > /dev/null 2>&1 &
Added /var/log/ipoib_ha.log and the verbose output of ipoib_ha.pl redirected into this log file.
3) I got IPoIB HA working on SLES 10, but the documentation is a little lacking. Looks like I have to put the same IP address in ifcfg-ib0 and ifcfg-ib1, is this correct?
Yes, IP address should be the same. Actually the configuration of the secondary interface does not matter.
The High Availability daemon reads the configuration of the primary interface and migrates it between the interfaces in case of failure.
# pwd
/etc/sysconfig/network
# cat ifcfg-ib0
DEVICE=ib0
BOOTPROTO=static
IPADDR=192.168.2.46
NETMASK=255.255.255.0
>
# cat ifcfg-ib1
DEVICE=ib1
BOOTPROTO=static
IPADDR=192.168.2.46
NETMASK=255.255.255.0
>
4) If I shutdown ib0 IB port, I see this from "/usr/local/ofed/bin/ipoib_ha.pl -v --with-arping --with-multicast"
Use of uninitialized value in concatenation (.) or string at /usr/local/ofed/bin/ipoib_ha.pl line 287.
Fixed.
Scott Weitzenkamp
SQA and Release Manager
Server Virtualization Business Unit
Cisco Systems
___
openfabrics-ewg mailing list
[EMAIL PROTECTED]
http://openib.org/mailman/listinfo/openfabrics-ewg
Best Regards,
Vladimir Sokolovsky
Software Integration Engineer
Mellanox Technologies Ltd.
Tell: +972 (4) 909-7200 ext. 338
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
