Re: RoCE passive side failures on 4.4-rc5

[Doug Ledford]

On 12/21/2015 10:08 AM, Matan Barak wrote:
> On Sun, Dec 20, 2015 at 9:29 AM, Or Gerlitz  wrote:
>> On 12/17/2015 3:58 PM, Or Gerlitz wrote:
>>>
>>> Using 4.4-rc5+ [1] and **not** applying any of the patches I sent today,
>>> I noted that RoCE passive side isn't working (rdma-cm, ibv_rc_pingpong
>>> works).
>>>
>>> I have two nodes in ConnectX3 VPI config (port1 IB and port2 Eth), the one
>>> with the 4.4-rc5 kernel can act as both (rping) client/server for IB links
>>> but only (rping) client for RoCE.
>>>
>>> I tried both inter-node and loopback runs, in all cases, the client side
>>> getsCM
>>> reject with reason 28, see [2], tried both iser and rping. Eth (ICMP, TCP)
>>> works OK.
>>
>>
>> OK, small progress, when the force Eth link type on my IB port (using mlx4
>> sysfs), things work.
>>
>> You should be able to reproduce it on your non-VPI systems the other way
>> around, by
>> forcing IB link type on one of the Eth ports and see the failure.
>>
>> I Saw the same behavior with both 4.4-rc2 and 4.4-rc5
>>
>> Or.
> 
> I've posted a patch that fixes that, please take a look at [1].
> 
> Regards,
> Matan
> 
> [1] https://www.mail-archive.com/linux-rdma@vger.kernel.org/msg30777.html

I've been seeing this too in my 4.4-rc testing, so I'll have test
results today.


-- 
Doug Ledford 
  GPG KeyID: 0E572FDD




signature.asc
Description: OpenPGP digital signature

Re: RoCE passive side failures on 4.4-rc5

[Matan Barak]

On Sun, Dec 20, 2015 at 9:29 AM, Or Gerlitz  wrote:
> On 12/17/2015 3:58 PM, Or Gerlitz wrote:
>>
>> Using 4.4-rc5+ [1] and **not** applying any of the patches I sent today,
>> I noted that RoCE passive side isn't working (rdma-cm, ibv_rc_pingpong
>> works).
>>
>> I have two nodes in ConnectX3 VPI config (port1 IB and port2 Eth), the one
>> with the 4.4-rc5 kernel can act as both (rping) client/server for IB links
>> but only (rping) client for RoCE.
>>
>> I tried both inter-node and loopback runs, in all cases, the client side
>> getsCM
>> reject with reason 28, see [2], tried both iser and rping. Eth (ICMP, TCP)
>> works OK.
>
>
> OK, small progress, when the force Eth link type on my IB port (using mlx4
> sysfs), things work.
>
> You should be able to reproduce it on your non-VPI systems the other way
> around, by
> forcing IB link type on one of the Eth ports and see the failure.
>
> I Saw the same behavior with both 4.4-rc2 and 4.4-rc5
>
> Or.

I've posted a patch that fixes that, please take a look at [1].

Regards,
Matan

[1] https://www.mail-archive.com/linux-rdma@vger.kernel.org/msg30777.html

>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RoCE passive side failures on 4.4-rc5

[Or Gerlitz]

On 12/17/2015 3:58 PM, Or Gerlitz wrote:

Using 4.4-rc5+ [1] and **not** applying any of the patches I sent today,
I noted that RoCE passive side isn't working (rdma-cm, ibv_rc_pingpong 
works).


I have two nodes in ConnectX3 VPI config (port1 IB and port2 Eth), the 
one with the 4.4-rc5 kernel can act as both (rping) client/server for 
IB links but only (rping) client for RoCE.


I tried both inter-node and loopback runs, in all cases, the client 
side getsCM
reject with reason 28, see [2], tried both iser and rping. Eth (ICMP, 
TCP) works OK.


OK, small progress, when the force Eth link type on my IB port (using 
mlx4 sysfs), things work.


You should be able to reproduce it on your non-VPI systems the other way 
around, by

forcing IB link type on one of the Eth ports and see the failure.

I Saw the same behavior with both 4.4-rc2 and 4.4-rc5

Or.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RoCE passive side failures on 4.4-rc5

[Or Gerlitz]

On 12/17/2015 4:18 PM, Sagi Grimberg wrote:
I'm using 4.4-rc2+ and I have RoCE working. 


I tried 4.4-rc2 and I see the same problem. I will make my .config
available to you and Moni so you can try it out on your systems Sunday.

I see this over both CX2 and CX3-pro, both are in VPI config.

BTW getting the 28 (consumer defined) reject reason is something I saw 
coming

and going from time to time in various weird cases over the past years.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: RoCE passive side failures on 4.4-rc5

[Sagi Grimberg]

Guys,

Using 4.4-rc5+ [1] and **not** applying any of the patches I sent today,
I noted that RoCE passive side isn't working (rdma-cm, ibv_rc_pingpong
works).


I'm using 4.4-rc2+ and I have RoCE working.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RoCE passive side failures on 4.4-rc5

[Or Gerlitz]

Guys,

Using 4.4-rc5+ [1] and **not** applying any of the patches I sent today,
I noted that RoCE passive side isn't working (rdma-cm, ibv_rc_pingpong 
works).


I have two nodes in ConnectX3 VPI config (port1 IB and port2 Eth), the 
one with the 4.4-rc5 kernel can act as both (rping) client/server for IB 
links but only (rping) client for RoCE.


I tried both inter-node and loopback runs, in all cases, the client side 
getsCM
reject with reason 28, see [2], tried both iser and rping. Eth (ICMP, 
TCP) works OK.


Or.

[1] Linus tree up to commit a5e90b1 "Merge branch 'fixes' of 
git://ftp.arm.linux.org.uk/~rmk/linux-arm"


[2] $ rping -d -v -c -a 192.168.31.17 -C 1
verbose
client
count 1
created cm_id 0x60cd90
cma_event type RDMA_CM_EVENT_ADDR_RESOLVED cma_id 0x60cd90 (parent)
cma_event type RDMA_CM_EVENT_ROUTE_RESOLVED cma_id 0x60cd90 (parent)
rdma_resolve_addr - rdma_resolve_route successful
created pd 0x60cfe0
created channel 0x608510
created cq 0x607f50
created qp 0x60d000
rping_setup_buffers called on cb 0x606010
allocated & registered buffers...
cq_thread started.
wait for CONNECTED state 10
cma_event type RDMA_CM_EVENT_REJECTED cma_id 0x60cd90 (parent)
connect error -1
rping_free_buffers called on cb 0x606010
cma event RDMA_CM_EVENT_REJECTED, error 28

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html