I've done some additional investigation and found the following:
int rdma_accept (called on the passive side with a valid qp_num set in
rdma_conn_param), on line 1535 in current git, there is the following:
[...]
if (!ucma_is_ud_qp(id->qp_type)) {
ret = ucma_modify_qp_rtr(id, id_priv->responder_resources);
if (ret)
return ret;
ret = ucma_modify_qp_rts(id, id_priv->initiator_depth);
if (ret)
return ret;
}
[...]
In this case, id->qp_type is IBV_QPT_RC, so ucma_modify_qp_rtr is called.
Unfortunately, ucma_modify_qp_rtr() contains the following on line:
static int ucma_modify_qp_rtr(struct rdma_cm_id *id, uint8_t resp_res)
{
struct ibv_qp_attr qp_attr;
int qp_attr_mask, ret;
if (!id->qp)
return ERR(EINVAL);
[...]
Since (as explained in my other email), rdma_create_qp() has not been called,
id->qp is NULL, causing ucma_modify_qp_rtr() and subsequently rdma_accept()
failure.
Other than directly setting id->qp, I don't see how rdma_accept() can be used
without calling rdma_create_qp() first.
Is the conclusion that rdmacm doesn't support outside QP creation/destruction
(at least not for the passive side) and that the documentation (rdma_accept
manpage) is incorrect?
Thanks,
Dries
* 'dr...@kimpe.be' (dries.ki...@gmail.com) <dries.ki...@gmail.com> [2014-08-20
12:59:47]:
> Hi,
>
> I've been having some issues in using RDMA CM with QPs created and owned
> by the applications, i.e. created with ibv_create_qp as opposed to
> rdma_create_qp.
>
> The man pages for both rdma_accept and rdma_connect hint that this is
> possible.
>
> In my application, whem I'm using rdma_create_qp, everything works.
> If I change to using ibv_create_qp (but everything else the same:
> qp_init_attr, PD, ... -- only change is rdma_create_qp -> ibv_create_qp)
> the call to rdma_accept fails with 'invalid argument'.
>
> [I do correctly specify the QP number in the accept parameters and
> connect parameters in both cases. The documentation states that rdmacm
> ignores qp_num in case rdma_create_qp was called]
>
> Some more details:
> - RC connection
> - Ofed version doesn't seem to matter (tried up to 2.1).
> - Using my own protection domain (not created by RDMA CM) but derived
> from the same context to which the ID is bound.
> - Using outside (i.e. not RDMA CM created) completion queue
>
> Things I've verified/tried:
>
> - verify context for RDMA cm id and QP created by ibv_create_qp is the
> same.
> - verify protection domains belong to the same context.
> - transition QP to INIT on passive side before calling rdma_accept.
> (I checked the rdma cm source code and noticed that rdma_create_qp
> also does this)
> - Checked the kernel code to see where an invaliad argument parameter
> could be returned but nothing stood out.
> - Verified sequence of calls on both active and passive side
> as described in rdma_cm manpage.
>
> I wasn't able to find any example code using rdma cm but not
> rdma_create_qp.
> It would be very helpful if somebody could list the differences or point
> me to some code.
>
> Thanks,
> Dries
>
_______________________________________________
ewg mailing list
ewg@lists.openfabrics.org
http://lists.openfabrics.org/mailman/listinfo/ewg
