Hi Hal,
I also found this in opensm.log on the SM machine (on a different server):
Jul 22 14:39:29 646241 [4AD86940] 0x01 -> mcmr_rcv_join_mgrp: ERR 1B11:
method = SubnAdmSet, scope_state = 0x1, component mask =
0x0000000000010083, expected comp mask = 0x00000000000130c7, MGID:
ff12:401b:8005::16 from port 0x0002c90200252841 (labdisk01 HCA-1)
Jul 22 14:39:31 379196 [48582940] 0x01 -> mcmr_rcv_join_mgrp: ERR 1B11:
method = SubnAdmSet, scope_state = 0x1, component mask =
0x0000000000010083, expected comp mask = 0x00000000000130c7, MGID:
ff12:601b:8005::16 from port 0x0002c90200252841 (labdisk01 HCA-1)
Jul 22 14:39:32 015121 [4357A940] 0x01 -> mcmr_rcv_join_mgrp: ERR 1B11:
method = SubnAdmSet, scope_state = 0x1, component mask =
0x0000000000010083, expected comp mask = 0x00000000000130c7, MGID:
ff12:601b:8005::2 from port 0x0002c90200252841 (labdisk01 HCA-1)
On 7/22/2010 2:49 PM, Tom Ammon wrote:
Hal,
Thanks for looking at all of this with me. ifconfig output is below.
On 7/22/2010 12:08 PM, Hal Rosenstock wrote:
Tom,
On Thu, Jul 22, 2010 at 1:19 PM, Tom Ammon<tom.am...@utah.edu> wrote:
Hal,
On 7/21/2010 2:45 PM, Hal Rosenstock wrote:
Hi Tom,
On 7/19/10, Tom Ammon<tom.am...@utah.edu> wrote:
I'm trying to set up partitions in a little test environment, and I'm
having trouble.
I have opensm running on a machine attached to the fabric, and sminfo on
the other machines confirm that this is indeed the master SM. Here's my
/etc/opensm/partitions.conf:
Default=0xffff , ipoib : ALL, SELF=full ;
PartitionBlue=0x8004, ipoib : 0x0002c9030009cb3f=full,
0x0002c90200252841=full, 0x0002c90200243471=full ;
PartitionRed=0x8005, ipoib : 0x0002c90200252841=full,
0x0002c90200243591=full, 0x0002c9030009cb2b=full ;
You don't really need the 0x8000 bit on in the pkeys but I don't think
it does any harm.
But when I go to the machine with port GUID 0x0002c90200243471, it
doesn't appear that it's getting the pkey I wanted:
[r...@stagnate ~]# ibstat
CA 'mthca0'
CA type: MT23108
Number of ports: 2
Firmware version: 3.3.5
Hardware version: a1
Node GUID: 0x0002c90200243470
System image GUID: 0x0002c90200243473
Port 1:
State: Active
Physical state: LinkUp
Rate: 10
Base lid: 10
LMC: 0
SM lid: 4
Capability mask: 0x02510a68
Port GUID: 0x0002c90200243471
Port 2:
State: Down
Physical state: Polling
Rate: 2
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x02510a68
Port GUID: 0x0002c90200243472
[r...@stagnate ~]# cat /sys/class/net/ib0/pkey
0xffff
What does:
smpquery pkeys 10 1
say ? Do you see the other pkey(s) on that port ?
[r...@stagnate ~]# smpquery pkeys 10 1
0: 0x7fff 0x8004 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
8: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
16: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
24: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
32: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
40: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
48: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
56: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
64 pkeys capacity for this port
So I see that both 7fff and 8004 are being assigned to this port. Is that
okay?
Yes.
Is there any problem with the machine also being in the default
partition?
No.
As I look around at all of the machines with smpquery, it appears that they
are all being assigned 7fff and the pkey that I assigned in partitions.conf.
Good.
But the machine that I want to run 2 child interfaces on is having issues.
It's at LID 7 and here's what smpquery says:
[r...@stagnate ~]# smpquery pkeys 7 1
0: 0x7fff 0x8004 0x8005 0x0000 0x0000 0x0000 0x0000 0x0000
8: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
16: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
24: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
32: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
40: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
48: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
56: 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000
64 pkeys capacity for this port
So that's fine, but when I try to create a child interface I get this:
[r...@labdisk01 ~]# echo 0x8004> /sys/class/net/ib0/create_child
-bash: echo: write error: Name not unique on network
I don't know what cause that error. Maybe someone else can help here.
Are you sure the ib0 interface is OK ? What does ifconfig ib0 say ?
Here's ifconfig ib0:
ib0 Link encap:InfiniBand HWaddr
80:00:04:04:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00
inet6 addr: fe80::202:c902:25:2841/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:65520 Metric:1
RX packets:1 errors:0 dropped:0 overruns:0 frame:0
TX packets:17 errors:0 dropped:7 overruns:0 carrier:0
collisions:0 txqueuelen:256
RX bytes:56 (56.0 b) TX bytes:3529 (3.4 KiB)
Then I brought up the "sub"interfaces with "ifup ib0.8004" "ifup
ib0.8005" . Still get the "Name not unique on network" message if I
switch the order and do ifup followed by echo 0x8004....etc.
ib0.8004 Link encap:InfiniBand HWaddr
80:00:04:06:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00
inet addr:10.0.0.2 Bcast:10.0.0.255 Mask:255.255.255.0
inet6 addr: fe80::202:c902:25:2841/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:65520 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:78 errors:0 dropped:17 overruns:0 carrier:0
collisions:0 txqueuelen:256
RX bytes:0 (0.0 b) TX bytes:14620 (14.2 KiB)
ib0.8005 Link encap:InfiniBand HWaddr
80:00:04:07:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00
inet addr:192.168.10.2 Bcast:192.168.10.255 Mask:255.255.255.0
inet6 addr: fe80::202:c902:25:2841/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:65520 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:72 errors:0 dropped:18 overruns:0 carrier:0
collisions:0 txqueuelen:256
RX bytes:0 (0.0 b) TX bytes:14269 (13.9 KiB)
Also, here's some junk from /var/log/messages, seemed like it might be
relevant, but maybe this is just IP stuff:
Jul 22 14:38:37 labdisk01 kernel: ADDRCONF(NETDEV_UP): ib0.8004: link is
not ready
Jul 22 14:38:37 labdisk01 kernel: ADDRCONF(NETDEV_CHANGE): ib0.8004:
link becomes ready
Jul 22 14:38:39 labdisk01 avahi-daemon[4056]: New relevant interface
ib0.8004.IPv6 for mDNS.
Jul 22 14:38:39 labdisk01 avahi-daemon[4056]: Joining mDNS multicast
group on interface ib0.8004.IPv6 with address fe80::202:c902:25:2841.
Jul 22 14:38:39 labdisk01 avahi-daemon[4056]: Registering new address
record for fe80::202:c902:25:2841 on ib0.8004.
Jul 22 14:38:41 labdisk01 avahi-daemon[4056]: New relevant interface
ib0.8004.IPv4 for mDNS.
Jul 22 14:38:41 labdisk01 avahi-daemon[4056]: Joining mDNS multicast
group on interface ib0.8004.IPv4 with address 10.0.0.2.
Jul 22 14:38:41 labdisk01 avahi-daemon[4056]: Registering new address
record for 10.0.0.2 on ib0.8004.
Jul 22 14:39:22 labdisk01 kernel: ADDRCONF(NETDEV_UP): ib0.8005: link is
not ready
Jul 22 14:39:22 labdisk01 kernel: ADDRCONF(NETDEV_CHANGE): ib0.8005:
link becomes ready
Jul 22 14:39:24 labdisk01 avahi-daemon[4056]: New relevant interface
ib0.8005.IPv6 for mDNS.
Jul 22 14:39:24 labdisk01 avahi-daemon[4056]: Joining mDNS multicast
group on interface ib0.8005.IPv6 with address fe80::202:c902:25:2841.
Jul 22 14:39:24 labdisk01 avahi-daemon[4056]: Registering new address
record for fe80::202:c902:25:2841 on ib0.8005.
Jul 22 14:39:26 labdisk01 avahi-daemon[4056]: New relevant interface
ib0.8005.IPv4 for mDNS.
Jul 22 14:39:26 labdisk01 avahi-daemon[4056]: Joining mDNS multicast
group on interface ib0.8005.IPv4 with address 192.168.10.2.
Jul 22 14:39:26 labdisk01 avahi-daemon[4056]: Registering new address
record for 192.168.10.2 on ib0.8005.
My plan was to create two child interfaces (0x8004 and 0x8005) and then
ifconfig ib0.8004 and ifconfig ib0.8005 to assign them to separate subnets.
That should be fine.
-- Hal
Tom
The pkey you are seeing is the only one for ib0 interface.
If you want to have IPoIB interfaces on the other partitions too, you
need to set this up by creating a child interface on those nodes; you
had asked about that in a previous email
(http://www.mail-archive.com/linux-rdma@vger.kernel.org/msg04728.html).
-- Hal
I'm trying to run one ipoib subnet in each partition, and then
eventually the goal is to have a different server that has 2 child
interfaces, one on each subnet. But it doesn't appear that my partition
configuration is even correct. Is there a syntax error, or something
else I am missing?
Thanks,
Tom
--
Tom Ammon
Network Engineer
Office: 801.587.0976
Mobile: 801.674.9273
Center for High Performance Computing
University of Utah
http://www.chpc.utah.edu
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Tom Ammon
Network Engineer
Office: 801.587.0976
Mobile: 801.674.9273
Center for High Performance Computing
University of Utah
http://www.chpc.utah.edu
--
Tom Ammon
Network Engineer
Office: 801.587.0976
Mobile: 801.674.9273
Center for High Performance Computing
University of Utah
http://www.chpc.utah.edu
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
