On 1/10/07, Sean Hefty <[EMAIL PROTECTED]> wrote: >> How about adding sendonly param to the ABI and having the ucma kernel >> code returning -EINVAL if someone tries to set it to true. Such code can >> be pushed to 2.6.21 and when you have the time to complete the >> implementation you can complete this?
> I don't think adding this is a huge deal; I just haven't gotten to it yet. > However, I'd like to make sure there's enough time once the change is made to > verify that we have the right result before pushing it upstream. Compared to the amount of functionality (ib_sa ref mcast counting and rdma cm u/k mcast API) provided by this code, i really think it can be done in two phases, first push a code that does not support sendonly and later once its implemented push a patch that adds this. Anyway, per the best of my knowledge you would not be able to fully verify the "right" result , i understand the opensm does not really support a sendonly join (*) in the sense of that it does not support configuring "one-way" MFTs for a subset of the branches/leaves of the multicast group spanning tree, but this is orthogonal to your code. (*) there are some more issues here which need to be addressed, see for example the "Some SMs don't support send-only yet" weird comment at ipoib_mcast_sendonly_join() > Woody has also seen this issue. And of course, I can't reproduce it on my > systems, but I'm actively looking into the problem. It looks like some sort > of > issue with ipoib trying to join a non-existent multicast group. mmm, weird, are there active rdmacm multicast consumers involved in this settings? >> Looking on the code, i understand that if an multicast consumer attempts >> to join a group for which another consumer is already joined then it >> just gets the group params, that is the mgid is your discriminator (with >> the exception of an all zeros mgid which has a different treatment) >> which makes much sense to me. > Not exactly. The rdma_cm consumer gets the group parameters for the ipoib > broadcast group. It uses this information as a template for joining new > groups. OK, got you at last (sorry but i have somehow ignored the call to ib_addr_get_mgid() at the rdmacm code). So to achieve interop with IPoIB all we need to do is remove the rdmacm signature bit and not to over-write the rdmacm qkey on the the qkey of the ipoib ipv4 broadcast group, are you ok with that? > One issue is that an rdma_cm consumer can first allocate a UD QP to use with > UD > traffic. When it later joins a multicast group, the qkey must be the same. > How > does ipoib handle this? Well, ipoib first sets a zero qkey into its qp in ipoib_init_qp() and later in ipoib_mcast_attach() does another qp modify providing priv->qkey which is the ipv4 broadcast group one, the rdma_cm can mimic this behaviour for qps created with rdma_create_qp and also for those created by the user. >> Since for our apps needs we do intend to join the 224.0.0.1 group, >> resolving a) above is fine for us --> we will join 224.0.0.1 above, >> provide the qkey to the rdma cm and it will join to the other group (eg >> 224.5.5.5) with this qkey. > I'm not completely following you on this yet. forget this, there is no need in such tricks since the rdmacm has the ipv4 broadcast group qkey. >> can you remind me what the idea/trick here, aren't you supposed to >> generate an mgid for this case? > This either returns an existing MCMemberRecord that this node has joined, or > it > fills out an MCMemberRecord that can be used to join a new group. If the mgid > is zero, the SA will assign one. thanks Or. _______________________________________________ openib-general mailing list [email protected] http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
