Re: Problem running rping over Intel adapters
On 10/4/2012 10:01 AM, Hefty, Sean wrote: retry_count, I assume, is for connection establishment? For the Chelsio devices, this is all up to the TCP stack and thus this counter is not used. If there is no TCP listener, then the connection establishment will fail immediately. If the node is not reachable, then normal TCP retransmit rules apply, but once TCP gives up, then the rdma connection establishment will fail. On IB, this is a transport level retry of a lost request. Ah, ok. Regardless, TCP handles all that for IWARP and we currently don't provide the rdma application any way to manipulate the TCP behavior. Steve. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: Problem running rping over Intel adapters
> retry_count, I assume, is for connection establishment? For the Chelsio > devices, this is all up to the TCP stack and thus this counter is not > used. If there is no TCP listener, then the connection establishment > will fail immediately. If the node is not reachable, then normal TCP > retransmit rules apply, but once TCP gives up, then the rdma connection > establishment will fail. On IB, this is a transport level retry of a lost request. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Problem running rping over Intel adapters
On 10/4/2012 12:41 AM, Or Gerlitz wrote: On 04/10/2012 03:47, Steve Wise wrote: Not used by iwarp drivers... Which one, the retry counter or the RNR retry counter? rnr_retry_count has no meaning in iWARP. There is no RNR message in the iwarp protocol. For some iwarp devices (cxgb3 for example), an ingress send with no recv buffer available results in immediate connection termination. Some devices, (eg cxgb4), the ingress SEND is buffered in the adapter until a recv buf is posted. Both behaviors are allowed in the iwarp specs. retry_count, I assume, is for connection establishment? For the Chelsio devices, this is all up to the TCP stack and thus this counter is not used. If there is no TCP listener, then the connection establishment will fail immediately. If the node is not reachable, then normal TCP retransmit rules apply, but once TCP gives up, then the rdma connection establishment will fail. Also, the RDMA_CM and IW_CM don't even pass these two counters down to the drivers' connect function. See struct iw_cm_conn_param and cma.c/cma_connect_iw() iwcm.c/iw_cm_connect(). Steve. Or. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: Problem running rping over Intel adapters
The IBTA spec clearly binds the IB HCA to a max of 7:
C9-142: For an HCA requester using Reliable Connection service, to prevent
the requester from retrying the request forever, the requester shall
maintain a 3 bit retry counter which is used to count the number of times
a particular request packet has been retried and timed out. This counter
shall be decremented each time the transport timer expires for a given request
packet. The counter shall be re-loaded whenever a given outstanding
request is cleared.
The language round modify QP probably validates a test on the Retry count:
C11-9: If any of the QP attributes to be modified are invalid or the requested
state transition is invalid, none of the QP attributes shall be modified.
An immediate error shall be returned and the QP state shall remain
unchanged.
A brief look at Mellanox code indicates that the "invalid" value is trusted and
or'ed into the modify mechanism.
Mike
> -Original Message-
> From: Hefty, Sean
> Sent: Wednesday, October 03, 2012 6:39 PM
> To: Doug Ledford; Marciniszyn, Mike; linux-rdma@vger.kernel.org
> Subject: RE: Problem running rping over Intel adapters
>
> > 1) rping, on the client side, clears the conn_params for the newly to
> > be attempted connection, then sets:
> >
> > conn_param.responder_resources = 1;
> > conn_param.initiator_depth = 1;
> > conn_param.retry_count = 10;
> >
> > On the accept side, rping clears the conn_params and then sets just
> > the responder_resources and initiator_depth, without even checking the
> > incoming requested conn_param values from the incoming cm_id. So, OK,
> > you can get away with that since this is a simple test program, but
> > still not "best programming practices". However, the important part
> > here is the retry_count of 10. That won't work on Intel/QLogic hardware.
>
> I pushed in the following fix to rping. Thanks
>
> ---
>
> rping: Reduce retry_count to fit in 3-bits
>
> From: Sean Hefty
>
> retry_count is a 3 bit value on IB, reduce it from
> 10 to 7.
>
> A value of 10 prevents rping from working over the Intel IB HCA. Problem
> reported by Doug Ledford
>
> The retry_count is also not set when calling rdma_accept.
> Rather than passing different values into rdma_accept than what was specified
> by the remote side, use the values given in the connection request.
>
> Signed-off-by: Sean Hefty
> ---
> examples/rping.c |9 ++---
> 1 files changed, 2 insertions(+), 7 deletions(-)
>
> diff --git a/examples/rping.c b/examples/rping.c index 785338e..32bd70a
> 100644
> --- a/examples/rping.c
> +++ b/examples/rping.c
> @@ -342,16 +342,11 @@ error:
>
> static int rping_accept(struct rping_cb *cb) {
> - struct rdma_conn_param conn_param;
> int ret;
>
> DEBUG_LOG("accepting client connection request\n");
>
> - memset(&conn_param, 0, sizeof conn_param);
> - conn_param.responder_resources = 1;
> - conn_param.initiator_depth = 1;
> -
> - ret = rdma_accept(cb->child_cm_id, &conn_param);
> + ret = rdma_accept(cb->child_cm_id, NULL);
> if (ret) {
> perror("rdma_accept");
> return ret;
> @@ -975,7 +970,7 @@ static int rping_connect_client(struct rping_cb *cb)
> memset(&conn_param, 0, sizeof conn_param);
> conn_param.responder_resources = 1;
> conn_param.initiator_depth = 1;
> - conn_param.retry_count = 10;
> + conn_param.retry_count = 7;
>
> ret = rdma_connect(cb->cm_id, &conn_param);
> if (ret) {
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Problem running rping over Intel adapters
On 04/10/2012 03:47, Steve Wise wrote: Not used by iwarp drivers... Which one, the retry counter or the RNR retry counter? Or. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Problem running rping over Intel adapters
On 10/3/2012 5:02 PM, Hefty, Sean wrote: 1) rping, on the client side, clears the conn_params for the newly to be attempted connection, then sets: conn_param.responder_resources = 1; conn_param.initiator_depth = 1; conn_param.retry_count = 10; 2) the qib driver enforces a maximum of 7 for retry_count. I don't see anything in the spec that specifies a maximum for this entry, and in particular I know it doesn't call out for 7 to mean infinite retries like it does for rnr_retry_count. On IB, retry count is a 3-bit value. See the CM REQ message. The kernel rdma cm should probably check for a valid value and reduce it accordingly. Then it's also fair to say that right now rping passes a known invalid value and needs fixed ;-) For IB, but I wasn't sure about iWarp. It doesn't look like it's used there though. If not, I'll apply a patch to rping, plus submit a fix for the kernel to validate the value. -- Not used by iwarp drivers... -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: Problem running rping over Intel adapters
> 1) rping, on the client side, clears the conn_params for the newly to be
> attempted connection, then sets:
>
> conn_param.responder_resources = 1;
> conn_param.initiator_depth = 1;
> conn_param.retry_count = 10;
>
> On the accept side, rping clears the conn_params and then sets just the
> responder_resources and initiator_depth, without even checking the
> incoming requested conn_param values from the incoming cm_id. So, OK,
> you can get away with that since this is a simple test program, but
> still not "best programming practices". However, the important part
> here is the retry_count of 10. That won't work on Intel/QLogic hardware.
I pushed in the following fix to rping. Thanks
---
rping: Reduce retry_count to fit in 3-bits
From: Sean Hefty
retry_count is a 3 bit value on IB, reduce it from
10 to 7.
A value of 10 prevents rping from working over the Intel
IB HCA. Problem reported by Doug Ledford
The retry_count is also not set when calling rdma_accept.
Rather than passing different values into rdma_accept than
what was specified by the remote side, use the values given
in the connection request.
Signed-off-by: Sean Hefty
---
examples/rping.c |9 ++---
1 files changed, 2 insertions(+), 7 deletions(-)
diff --git a/examples/rping.c b/examples/rping.c
index 785338e..32bd70a 100644
--- a/examples/rping.c
+++ b/examples/rping.c
@@ -342,16 +342,11 @@ error:
static int rping_accept(struct rping_cb *cb)
{
- struct rdma_conn_param conn_param;
int ret;
DEBUG_LOG("accepting client connection request\n");
- memset(&conn_param, 0, sizeof conn_param);
- conn_param.responder_resources = 1;
- conn_param.initiator_depth = 1;
-
- ret = rdma_accept(cb->child_cm_id, &conn_param);
+ ret = rdma_accept(cb->child_cm_id, NULL);
if (ret) {
perror("rdma_accept");
return ret;
@@ -975,7 +970,7 @@ static int rping_connect_client(struct rping_cb *cb)
memset(&conn_param, 0, sizeof conn_param);
conn_param.responder_resources = 1;
conn_param.initiator_depth = 1;
- conn_param.retry_count = 10;
+ conn_param.retry_count = 7;
ret = rdma_connect(cb->cm_id, &conn_param);
if (ret) {
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Problem running rping over Intel adapters
On 10/3/2012 6:02 PM, Hefty, Sean wrote: 1) rping, on the client side, clears the conn_params for the newly to be attempted connection, then sets: conn_param.responder_resources = 1; conn_param.initiator_depth = 1; conn_param.retry_count = 10; >>> >>> 2) the qib driver enforces a maximum of 7 for retry_count. I don't see anything in the spec that specifies a maximum for this entry, and in particular I know it doesn't call out for 7 to mean infinite retries like it does for rnr_retry_count. >>> >>> On IB, retry count is a 3-bit value. See the CM REQ message. >>> >>> The kernel rdma cm should probably check for a valid value and >>> reduce it >> accordingly. >> >> Then it's also fair to say that right now rping passes a known >> invalid value and needs fixed ;-) > > For IB, but I wasn't sure about iWarp. It doesn't look like it's > used there though. I would hope it's not used on iWARPit's running over it's own link layer where this is neither defined nor applicable. > If not, I'll apply a patch to rping, plus submit a fix for the > kernel to validate the value. -- Doug Ledford GPG KeyID: 0E572FDD http://people.redhat.com/dledford Infiniband specific RPMs available at http://people.redhat.com/dledford/Infiniband signature.asc Description: OpenPGP digital signature
RE: Problem running rping over Intel adapters
> >> 1) rping, on the client side, clears the conn_params for the newly to be > >> attempted connection, then sets: > >> > >>conn_param.responder_resources = 1; > >>conn_param.initiator_depth = 1; > >>conn_param.retry_count = 10; > > > > > >> 2) the qib driver enforces a maximum of 7 for retry_count. I don't see > >> anything in the spec that specifies a maximum for this entry, and in > >> particular I know it doesn't call out for 7 to mean infinite retries > >> like it does for rnr_retry_count. > > > > On IB, retry count is a 3-bit value. See the CM REQ message. > > > > The kernel rdma cm should probably check for a valid value and reduce it > accordingly. > > Then it's also fair to say that right now rping passes a known invalid > value and needs fixed ;-) For IB, but I wasn't sure about iWarp. It doesn't look like it's used there though. If not, I'll apply a patch to rping, plus submit a fix for the kernel to validate the value. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Problem running rping over Intel adapters
On 10/3/2012 5:50 PM, Hefty, Sean wrote: >> 1) rping, on the client side, clears the conn_params for the newly to be >> attempted connection, then sets: >> >> conn_param.responder_resources = 1; >> conn_param.initiator_depth = 1; >> conn_param.retry_count = 10; > > >> 2) the qib driver enforces a maximum of 7 for retry_count. I don't see >> anything in the spec that specifies a maximum for this entry, and in >> particular I know it doesn't call out for 7 to mean infinite retries >> like it does for rnr_retry_count. > > On IB, retry count is a 3-bit value. See the CM REQ message. > > The kernel rdma cm should probably check for a valid value and reduce it > accordingly. Then it's also fair to say that right now rping passes a known invalid value and needs fixed ;-) -- Doug Ledford GPG KeyID: 0E572FDD http://people.redhat.com/dledford Infiniband specific RPMs available at http://people.redhat.com/dledford/Infiniband signature.asc Description: OpenPGP digital signature
RE: Problem running rping over Intel adapters
> 1) rping, on the client side, clears the conn_params for the newly to be > attempted connection, then sets: > > conn_param.responder_resources = 1; > conn_param.initiator_depth = 1; > conn_param.retry_count = 10; > 2) the qib driver enforces a maximum of 7 for retry_count. I don't see > anything in the spec that specifies a maximum for this entry, and in > particular I know it doesn't call out for 7 to mean infinite retries > like it does for rnr_retry_count. On IB, retry count is a 3-bit value. See the CM REQ message. The kernel rdma cm should probably check for a valid value and reduce it accordingly. - Sean -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Problem running rping over Intel adapters
We ran into this problem when testing rping over Intel/QLogic hardware: [root@rdmaperf3 ~]# rping -s -a 172.31.2.103 -v wait for CONNECTED state 10 connect error -1 cma event RDMA_CM_EVENT_REJECTED, error 28 [root@rdmaperf3 ~]# [root@rdmaperf8 ~]# rping -c -a 172.31.2.103 -v -C 5 cma event RDMA_CM_EVENT_CONNECT_ERROR, error -1 wait for CONNECTED state 4 connect error -1 [root@rdmaperf8 ~]# Turns out this is because of a couple things: 1) rping, on the client side, clears the conn_params for the newly to be attempted connection, then sets: conn_param.responder_resources = 1; conn_param.initiator_depth = 1; conn_param.retry_count = 10; On the accept side, rping clears the conn_params and then sets just the responder_resources and initiator_depth, without even checking the incoming requested conn_param values from the incoming cm_id. So, OK, you can get away with that since this is a simple test program, but still not "best programming practices". However, the important part here is the retry_count of 10. That won't work on Intel/QLogic hardware. 2) the qib driver enforces a maximum of 7 for retry_count. I don't see anything in the spec that specifies a maximum for this entry, and in particular I know it doesn't call out for 7 to mean infinite retries like it does for rnr_retry_count. I don't think the spec really cares how we solve this, and I don't think there is a hard limit of 7 for the retry_count like the qib driver enforces. On the other hand, the spec doesn't call out a limit on the retry_count but I would assume each driver has the option to implement their own "reasonable, implementation defined" limit in a case like this. So, do we make qib more liberal in its acceptance of retry_count or do we fix rping to use a smaller number? Matters not to me... -- Doug Ledford GPG KeyID: 0E572FDD http://people.redhat.com/dledford Infiniband specific RPMs available at http://people.redhat.com/dledford/Infiniband signature.asc Description: OpenPGP digital signature
