Hello Yevgeny,
OK, so there are three possible reasons that I can think of:
1. Something is wrong in the configuration.
2. The application does not saturate the link, thus QoS
and the whole VL arbitration thing doesn't kick in.
3. There's some bug, somewhere.
Let's start with reason no. 1.
Please shut off each of the SLs one by one, and
make sure that the application gets zero BW on
these SLs. You can do it by mapping SL to VL15:
qos_sl2vl 0,15,2,3,4,5,6,7,8,9,10,11,12,13,14,15
If I shut down this SL by moving it to VL15, the interfaces stop
pinging.
This is probably because some IPoIB multicast traffic gets cut off
for
pkey 0x7fff .. ?
Could be, or because ALL interfaces are mapped to
SL1, which is what the results below suggest.
Yes, you are right (see below).
So no results for this one.
and then
qos_sl2vl 0,1,15,3,4,5,6,7,8,9,10,11,12,13,14,15
With this setup, and the following QoS settings:
qos_max_vls 8
qos_high_limit 1
qos_vlarb_high 0:0,1:0,2:0,3:0,4:0,5:0
qos_vlarb_low 0:1,1:64,2:128,3:192,4:0,5:0
qos_sl2vl 0,1,15,3,4,5,6,7,8,9,10,11,12,13,14,15
I get roughly the same values for SL 1 to SL3:
That doesn't look right.
You have shut off SL2, so you can't see same
BW for this SL. Looks like there is a problem
in configuration (or bug in SM).
Yes, that's correct: There could be a configuration issue or a bug in
SM:
Current setup and results:
qos_max_vls 8
qos_high_limit 1
qos_vlarb_high 0:0,1:0,2:0,3:0,4:0,5:0
qos_vlarb_low 0:1,1:64,2:128,3:192,4:0,5:0
qos_sl2vl 0,1,15,3,4,5,6,7,8,9,10,11,12,13,14,15
[r...@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-ic0 -t
10 -P 8 2>&1; done | grep SUM
[SUM] 0.0-10.1 sec 9.78 GBytes 8.28 Gbits/sec
[SUM] 0.0-10.0 sec 5.69 GBytes 4.89 Gbits/sec
[SUM] 0.0-10.0 sec 4.30 GBytes 3.69 Gbits/sec
[r...@pichu22 ~]# while test -e keep_going; do iperf -c
pichu16-backbone
-t 10 -P 8 2>&1; done | grep SUM
[SUM] 0.0-10.2 sec 6.44 GBytes 5.45 Gbits/sec
[SUM] 0.0-10.1 sec 6.64 GBytes 5.66 Gbits/sec
[SUM] 0.0-10.0 sec 6.03 GBytes 5.15 Gbits/sec
[r...@pichu22 ~]# while test -e keep_going; do iperf -c
pichu16-admin -t
10 -P 8 2>&1; done | grep SUM
[SUM] 0.0-10.0 sec 5.80 GBytes 4.98 Gbits/sec
[SUM] 0.0-10.0 sec 7.04 GBytes 6.02 Gbits/sec
[SUM] 0.0-10.0 sec 6.60 GBytes 5.67 Gbits/sec
The -backbone bandwidth should be 0 here.
Have you validated somehow that the interfaces
have been mapped to the right SLs?
Two things:
1/ Either the interface have not been mapped properly to the right
SL's,
but given the config files below, I doubt it:
[r...@pichu22 ~]# tail -n 5 /etc/sysconfig/network-scripts/ifcfg-ib0*
==> /etc/sysconfig/network-scripts/ifcfg-ib0 <==
BOOTPROTO=static
IPADDR=10.12.1.10
NETMASK=255.255.0.0
ONBOOT=yes
MTU=2000
==> /etc/sysconfig/network-scripts/ifcfg-ib0.8001 <==
BOOTPROTO=static
IPADDR=10.13.1.10
NETMASK=255.255.0.0
ONBOOT=yes
MTU=2000
==> /etc/sysconfig/network-scripts/ifcfg-ib0.8002 <==
BOOTPROTO=static
IPADDR=10.14.1.10
NETMASK=255.255.0.0
ONBOOT=yes
MTU=2000
partitions.conf:
-----------------
default=0x7fff,ipoib : ALL=full;
ip_backbone=0x0001,ipoib : ALL=full;
ip_admin=0x0002,ipoib : ALL=full;
qos-policy.conf:
----------------
qos-ulps
default : 0 # default SL
ipoib, pkey 0x7FFF : 1 # IP with default pkey 0x7FFF
ipoib, pkey 0x1 : 2 # backbone IP with pkey 0x1
ipoib, pkey 0x2 : 3 # admin IP with pkey 0x2
end-qos-ulps
ib0.8001 maps to pkey 1 (with MSB set to 1 due to full membership =>
0x8001 = (1<<16 | 1)
ib0.8002 maps to pkey 2 (with MSB set to 1 due to full membership =>
0x8002 = (1<<16 | 2)
2/ Somehow, the qos policy parsing does not map pkeys as we would
expect, which is what the opensm messages would suggest:
Nov 25 13:13:05 664690 [373E910] 0x01 -> __qos_policy_validate_pkey:
ERR
AC15: pkey 0x0002 in match rule - overriding partition SL (0) with QoS
Level SL (3)
Nov 25 13:13:05 664681 [373E910] 0x01 -> __qos_policy_validate_pkey:
ERR
AC15: pkey 0x0001 in match rule - overriding partition SL (0) with QoS
Level SL (2)
Nov 25 13:13:05 664670 [373E910] 0x01 -> __qos_policy_validate_pkey:
ERR
AC15: pkey 0x7FFF in match rule - overriding partition SL (0) with QoS
Level SL (1)
If the messages are correct and do reflect what opensm is actually
doing, this would explain why shutting down SL1 (by moving it to VL15)
prevented all interfaces from running.
