Yevgeny Kliteynik wrote:
> Vincent Ficet wrote:
>> Hello Yevgeny,
>>
>>>>> OK, so there are three possible reasons that I can think of:
>>>>> 1. Something is wrong in the configuration.
>>>>> 2. The application does not saturate the link, thus QoS
>>>>> and the whole VL arbitration thing doesn't kick in.
>>>>> 3. There's some bug, somewhere.
>>>>>
>>>>> Let's start with reason no. 1.
>>>>> Please shut off each of the SLs one by one, and
>>>>> make sure that the application gets zero BW on
>>>>> these SLs. You can do it by mapping SL to VL15:
>>>>>
>>>>> qos_sl2vl 0,15,2,3,4,5,6,7,8,9,10,11,12,13,14,15
>>>> If I shut down this SL by moving it to VL15, the interfaces stop
>>>> pinging.
>>>> This is probably because some IPoIB multicast traffic gets cut off for
>>>> pkey 0x7fff .. ?
>>> Could be, or because ALL interfaces are mapped to
>>> SL1, which is what the results below suggest.
>> Yes, you are right (see below).
>>>> So no results for this one.
>>>>> and then
>>>>> qos_sl2vl 0,1,15,3,4,5,6,7,8,9,10,11,12,13,14,15
>>>>>
>>>> With this setup, and the following QoS settings:
>>>>
>>>> qos_max_vls 8
>>>> qos_high_limit 1
>>>> qos_vlarb_high 0:0,1:0,2:0,3:0,4:0,5:0
>>>> qos_vlarb_low 0:1,1:64,2:128,3:192,4:0,5:0
>>>> qos_sl2vl 0,1,15,3,4,5,6,7,8,9,10,11,12,13,14,15
>>>>
>>>> I get roughly the same values for SL 1 to SL3:
>>> That doesn't look right.
>>> You have shut off SL2, so you can't see same
>>> BW for this SL. Looks like there is a problem
>>> in configuration (or bug in SM).
>> Yes, that's correct: There could be a configuration issue or a bug in
>> SM:
>>
>> Current setup and results:
>>
>> qos_max_vls 8
>> qos_high_limit 1
>> qos_vlarb_high 0:0,1:0,2:0,3:0,4:0,5:0
>> qos_vlarb_low 0:1,1:64,2:128,3:192,4:0,5:0
>> qos_sl2vl 0,1,15,3,4,5,6,7,8,9,10,11,12,13,14,15
>>
>> [r...@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-ic0 -t
>> 10 -P 8 2>&1; done | grep SUM
>> [SUM] 0.0-10.1 sec 9.78 GBytes 8.28 Gbits/sec
>> [SUM] 0.0-10.0 sec 5.69 GBytes 4.89 Gbits/sec
>> [SUM] 0.0-10.0 sec 4.30 GBytes 3.69 Gbits/sec
>> [r...@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-backbone
>> -t 10 -P 8 2>&1; done | grep SUM
>> [SUM] 0.0-10.2 sec 6.44 GBytes 5.45 Gbits/sec
>> [SUM] 0.0-10.1 sec 6.64 GBytes 5.66 Gbits/sec
>> [SUM] 0.0-10.0 sec 6.03 GBytes 5.15 Gbits/sec
>> [r...@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-admin -t
>> 10 -P 8 2>&1; done | grep SUM
>> [SUM] 0.0-10.0 sec 5.80 GBytes 4.98 Gbits/sec
>> [SUM] 0.0-10.0 sec 7.04 GBytes 6.02 Gbits/sec
>> [SUM] 0.0-10.0 sec 6.60 GBytes 5.67 Gbits/sec
>>
>> The -backbone bandwidth should be 0 here.
>>
>>> Have you validated somehow that the interfaces
>>> have been mapped to the right SLs?
>> Two things:
>> 1/ Either the interface have not been mapped properly to the right SL's,
>> but given the config files below, I doubt it:
>>
>> [r...@pichu22 ~]# tail -n 5 /etc/sysconfig/network-scripts/ifcfg-ib0*
>> ==> /etc/sysconfig/network-scripts/ifcfg-ib0 <==
>> BOOTPROTO=static
>> IPADDR=10.12.1.10
>> NETMASK=255.255.0.0
>> ONBOOT=yes
>> MTU=2000
>>
>> ==> /etc/sysconfig/network-scripts/ifcfg-ib0.8001 <==
>> BOOTPROTO=static
>> IPADDR=10.13.1.10
>> NETMASK=255.255.0.0
>> ONBOOT=yes
>> MTU=2000
>>
>> ==> /etc/sysconfig/network-scripts/ifcfg-ib0.8002 <==
>> BOOTPROTO=static
>> IPADDR=10.14.1.10
>> NETMASK=255.255.0.0
>> ONBOOT=yes
>> MTU=2000
>>
>> partitions.conf:
>> -----------------
>>
>> default=0x7fff,ipoib : ALL=full;
>> ip_backbone=0x0001,ipoib : ALL=full;
>> ip_admin=0x0002,ipoib : ALL=full;
>>
>> qos-policy.conf:
>> ----------------
>> qos-ulps
>> default : 0 # default SL
>> ipoib, pkey 0x7FFF : 1 # IP with default pkey 0x7FFF
>> ipoib, pkey 0x1 : 2 # backbone IP with pkey 0x1
>> ipoib, pkey 0x2 : 3 # admin IP with pkey 0x2
>> end-qos-ulps
>>
>> ib0.8001 maps to pkey 1 (with MSB set to 1 due to full membership =>
>> 0x8001 = (1<<16 | 1)
>> ib0.8002 maps to pkey 2 (with MSB set to 1 due to full membership =>
>> 0x8002 = (1<<16 | 2)
>>
>> 2/ Somehow, the qos policy parsing does not map pkeys as we would
>> expect, which is what the opensm messages would suggest:
>>
>> Nov 25 13:13:05 664690 [373E910] 0x01 -> __qos_policy_validate_pkey: ERR
>> AC15: pkey 0x0002 in match rule - overriding partition SL (0) with QoS
>> Level SL (3)
>> Nov 25 13:13:05 664681 [373E910] 0x01 -> __qos_policy_validate_pkey: ERR
>> AC15: pkey 0x0001 in match rule - overriding partition SL (0) with QoS
>> Level SL (2)
>> Nov 25 13:13:05 664670 [373E910] 0x01 -> __qos_policy_validate_pkey: ERR
>> AC15: pkey 0x7FFF in match rule - overriding partition SL (0) with QoS
>> Level SL (1)
>>
>> If the messages are correct and do reflect what opensm is actually
>> doing, this would explain why shutting down SL1 (by moving it to VL15)
>> prevented all interfaces from running.
>
> What SM are you using?
OpenSM 3.3.2
> Does it have the following bug fix:
>
> http://www.openfabrics.org/git/?p=~sashak/management.git;a=commit;h=ef4c8ac3fdd50bb0b7af06887abdb5b73b7ed8c3
>
Yes it does.
The most recent git commit (sorted by date) is for this rpm is:
* Sun Aug 23 2009 Sasha Khapyorsky <sas...@voltaire.com>
commit 3f4954c73add5e7b598883242782607f87c482b4
Apart from the following commit (with a bogus date):
* Tue Jul 24 2035 Keshetti Mahesh <keshetti.mah...@gmail.com>
commit a0c23ed2194e96816744a075d405ff34c8373fa3
Thanks,
Vincent
>
> -- Yevgeny
>
>>>> [r...@pichu22 ~]# while test -e keep_going; do iperf -c pichu16-ic0 -t
>>>> 10 -P 8 2>&1; done | grep SUM
>>>> [SUM] 0.0-10.0 sec 6.15 GBytes 5.28 Gbits/sec
>>>> [SUM] 0.0-10.0 sec 6.00 GBytes 5.16 Gbits/sec
>>>> [SUM] 0.0-10.1 sec 5.38 GBytes 4.59 Gbits/sec
>>>>
>>>> [r...@pichu22 ~]# while test -e keep_going; do iperf -c
>>>> pichu16-backbone
>>>> -t 10 -P 8 2>&1; done | grep SUM
>>>> [SUM] 0.0-10.0 sec 6.09 GBytes 5.23 Gbits/sec
>>>> [SUM] 0.0-10.0 sec 6.41 GBytes 5.51 Gbits/sec
>>>> [SUM] 0.0-10.0 sec 4.72 GBytes 4.05 Gbits/sec
>>>>
>>>> [r...@pichu22 ~]# while test -e keep_going; do iperf -c
>>>> pichu16-admin -t
>>>> 10 -P 8 2>&1; done | grep SUM
>>>> [SUM] 0.0-10.1 sec 6.96 GBytes 5.92 Gbits/sec
>>>> [SUM] 0.0-10.1 sec 5.89 GBytes 5.00 Gbits/sec
>>>> [SUM] 0.0-10.0 sec 5.35 GBytes 4.58 Gbits/sec
>>>>
>>>>> and then
>>>>> qos_sl2vl 0,1,2,15,4,5,6,7,8,9,10,11,12,13,14,15
>>>> Same results as the previous 0,1,15,3,... SL2vl mapping.
>>>>> If this part works well, then we will continue to
>>>>> reason no. 2.
>>>> In the above tests, I used -P8 to force 8 threads on the client
>>>> side for
>>>> each test.
>>>> I have one quad core CPU(Intel E55400).
>>>> This makes 24 iperf threads on 4 cores, which __should__ be fine
>>>> (well I
>>>> suppose ...)
>>> Best would be having one qperf per CPU core,
>>> which is 4 qperf's in your case.
>>>
>>> What is your subnet setup?
>> Nothing fancy for this test: I just bounce the taffic through a switch;
>>
>> [r...@pichu16 ~]# ibtracert 49 53
>>> From ca {0x2c9000100d00056c} portnum 1 lid 49-49 "pichu16 HCA-1"
>> [1] -> switch port {0x0002c9000100d0d4}[22] lid 58-58 "bullX chassis 36
>> port QDR switch"
>> [28] -> ca port {0x2c9000100d000679}[1] lid 53-53 "pichu22 HCA-1"
>> To ca {0x2c9000100d000678} portnum 1 lid 53-53 "pichu22 HCA-1"
>>
>> Vincent
>>
>>> -- Yevgeny
>>>
>>>
>>>> And regarding reason #3. I still get the error I got yesterday, which
>>>> you told me was not important because the SL's set in partitions.conf
>>>> would override what was read from qos-policy.conf in the first place.
>>>>
>>>> Nov 25 13:13:05 664690 [373E910] 0x01 ->
>>>> __qos_policy_validate_pkey: ERR
>>>> AC15: pkey 0x0002 in match rule - overriding partition SL (0) with QoS
>>>> Level SL (3)
>>>> Nov 25 13:13:05 664681 [373E910] 0x01 ->
>>>> __qos_policy_validate_pkey: ERR
>>>> AC15: pkey 0x0001 in match rule - overriding partition SL (0) with QoS
>>>> Level SL (2)
>>>> Nov 25 13:13:05 664670 [373E910] 0x01 ->
>>>> __qos_policy_validate_pkey: ERR
>>>> AC15: pkey 0x7FFF in match rule - overriding partition SL (0) with QoS
>>>> Level SL (1)
>>>>
>>>> Thanks for your help.
>>>>
>>>> Vincent
>>>>
>>>
>>>
>>
>>
>
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
