Re: [openib-general] multicast code/merge status
> This is fine, but it may change when the user needs to make this > choice. E.g. when creating the QP, versus joining the multicast group, > in order to support the valid options. The selection also needs to be > conveyed to the kernel somehow. At this point, maybe we just need to > start looking at specific implementations. Indeed. I will send a patch early next week. Or. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] multicast code/merge status
> makes sense, so my suggestion is "leave this (using the ipoib qkey) to > the user" This is fine, but it may change when the user needs to make this choice. E.g. when creating the QP, versus joining the multicast group, in order to support the valid options. The selection also needs to be conveyed to the kernel somehow. At this point, maybe we just need to start looking at specific implementations. > I don't think there is any limitation today in the verbs layer, > actually for our testing so far we patches the rdmacm not set the sig > byte and use the ipoib (ie not override it in core/cma.c) and we > manage to interop fine with ipoib. Maybe this shouldn't be allowed for any random application. Perhaps there's a way to treat controlled qkeys similar to the privileged port space. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] multicast code/merge status
On 1/17/07, Sean Hefty <[EMAIL PROTECTED]> wrote: > > +1 used only for unicast > > +2 used only for multicast > > +3 used for both unicast and multicast > If you view this as the use case for one side only, we also have option 3 > communicating with options 1 and 2. I would list these as: OK > +4 unicast QP to unicast and multicast QP i think you mean 3 <--> 1 that is unicast and multicast QP to unicast QP > +5 multicast QP to unicast and multicast QP i think you mean 3 <--> 2 that is unicast and multicast QP to multicast QP > Today, all of these work. What you're wanting to add is the ability to > communicate with an ipoib multicast group. I'd like to do this without > breaking > any of the existing communications, or treat ipoib separately for security > reasons. makes sense, so my suggestion is "leave this (using the ipoib qkey) to the user" if you prefer to have two group types: rdmacm and ipoib - that's fine. we would use ipoib type groups and in the envs that seting the qkey to be the ipoib would not break our communication (that is where we do need to interop with IPoIB) - we would do it, else we would do nothing. > > To make things simple, the solution i suggest is that that the RDMA CM > > would --not-- do this modify QP/QKEY (that is would set the 0x12345678 > > qkey on the modify qp to init) and rather leave it to the RDMA CM > > consumer --if-- they wish to do so. However it will use the ipv4 > > broadcast group qkey for doing mcast joins and report this qkey to the > > user in the ud param of the event. > > We need to be able to handle options 4 and 5 as well. indeed, i have addressed that above. > > this (what qkey is assigned to the ipv4 broadcast group by different > > SAs) is orthogonal to the discussion we do here. > This depends on whether verbs allows, or if it should allow, a user to > specify a > controlled qkey when configuring their QP. I don't think there is any limitation today in the verbs layer, actually for our testing so far we patches the rdmacm not set the sig byte and use the ipoib (ie not override it in core/cma.c) and we manage to interop fine with ipoib. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] multicast code/merge status
On 1/17/07, Michael S. Tsirkin <[EMAIL PROTECTED]> wrote: > > not following you here, how does qkey relates to RC QPs ? > Currently you can block userspace from creating QPs by unloading uverbs > module. > Maybe we should make it possible to block creating UD QPs from userspace > as a separate security measure. I don't think this is valid option for most of the IB production env. but if you want to add blocking UD QP creation to ib_uverbs as mod param whose default value is --unset--, i don't really care. Or. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] multicast code/merge status
> +1 used only for unicast > +2 used only for multicast > +3 used for both unicast and multicast If you view this as the use case for one side only, we also have option 3 communicating with options 1 and 2. I would list these as: +4 unicast QP to unicast and multicast QP +5 multicast QP to unicast and multicast QP Today, all of these work. What you're wanting to add is the ability to communicate with an ipoib multicast group. I'd like to do this without breaking any of the existing communications, or treat ipoib separately for security reasons. > To make things simple, the solution i suggest is that that the RDMA CM > would --not-- do this modify QP/QKEY (that is would set the 0x12345678 > qkey on the modify qp to init) and rather leave it to the RDMA CM > consumer --if-- they wish to do so. However it will use the ipv4 > broadcast group qkey for doing mcast joins and report this qkey to the > user in the ud param of the event. We need to be able to handle options 4 and 5 as well. > this (what qkey is assigned to the ipv4 broadcast group by different > SAs) is orthogonal to the discussion we do here. This depends on whether verbs allows, or if it should allow, a user to specify a controlled qkey when configuring their QP. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] multicast code/merge status
> > Maybe a machanism should be in place to control access to this separately > > from regular rdma cm for RC QPs? > > not following you here, how does qkey relates to RC QPs ? Currently you can block userspace from creating QPs by unloading uverbs module. Maybe we should make it possible to block creating UD QPs from userspace as a separate security measure. -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] multicast code/merge status
Michael S. Tsirkin wrote: >> Quoting Sean Hefty <[EMAIL PROTECTED]>: >> Subject: Re: multicast code/merge status >> >>> sure, it can use the rdmacm qkey (0x1234567 etc) when it creates the QP >>> and later --if-- the user joins a multicast group modify the qp state >>> with the group qkey and report it in the cma event such that the >>> consumer of the rdmacm would set this into his IB UD TX WR >> Changing the qkey would break its existing UD communication. >> >>> Bottom line, Looking in the IB SPEC and IPoIB RFC i did not see >>> mentioning of privileged QKEY. >> From RFC 4391 (ipoib RFC), 4.1: >> >> 2. Q_Key >> >>It is RECOMMENDED that a controlled Q_Key be used with the >>high-order bit set. This is to prevent non-privileged >>software from fabricating and sending out bogus IP datagrams. > > BTW, should we be worried that proposed extension (passing qkey in rdma cm > param > list) seems to expose this qkey to non-privileged software? As was said over related threads here and elsewhere, multicast has its in nature non safeties and having IB implement broadcast over multicast adds more in safety to the party. Specifically, as Roland has commented, a user can attach his user space UD QP to the MGID of the ipv4 broadcast(if ipoib is running on this node it will join the group) and start making this IP subnet go crazy. We only want interop with IPoIB and we don't need to join/attach the ipv4 broadcast group just have an option for the rdmacm to use its qkey for joins and later either the rdmacm or the consumer will also set this qkey into the QP and the UD TX WR > Maybe a machanism should be in place to control access to this separately > from regular rdma cm for RC QPs? not following you here, how does qkey relates to RC QPs ? Or. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] multicast code/merge status
> Quoting Sean Hefty <[EMAIL PROTECTED]>: > Subject: Re: multicast code/merge status > > > sure, it can use the rdmacm qkey (0x1234567 etc) when it creates the QP > > and later --if-- the user joins a multicast group modify the qp state > > with the group qkey and report it in the cma event such that the > > consumer of the rdmacm would set this into his IB UD TX WR > > Changing the qkey would break its existing UD communication. > > > Bottom line, Looking in the IB SPEC and IPoIB RFC i did not see > > mentioning of privileged QKEY. > > From RFC 4391 (ipoib RFC), 4.1: > > 2. Q_Key > >It is RECOMMENDED that a controlled Q_Key be used with the >high-order bit set. This is to prevent non-privileged >software from fabricating and sending out bogus IP datagrams. BTW, should we be worried that proposed extension (passing qkey in rdma cm param list) seems to expose this qkey to non-privileged software? Maybe a machanism should be in place to control access to this separately from regular rdma cm for RC QPs? -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] multicast code/merge status
Sean Hefty wrote: >> sure, it can use the rdmacm qkey (0x1234567 etc) when it creates the >> QP and later --if-- the user joins a multicast group modify the qp >> state with the group qkey and report it in the cma event such that the >> consumer of the rdmacm would set this into his IB UD TX WR > Changing the qkey would break its existing UD communication. OK, so we have three use cases here for a UD QP +1 used only for unicast +2 used only for multicast +3 used for both unicast and multicast and my suggestion (default qkey, when join is completed do qp modify with the group qkey) would work fine for use cases 1 - since the user never joins to anything and 2 - same as it works in ipoib so we are left with use case 3. To make things simple, the solution i suggest is that that the RDMA CM would --not-- do this modify QP/QKEY (that is would set the 0x12345678 qkey on the modify qp to init) and rather leave it to the RDMA CM consumer --if-- they wish to do so. However it will use the ipv4 broadcast group qkey for doing mcast joins and report this qkey to the user in the ud param of the event. So users that don't care about their qkey would never bother to do this modify qp and users who do care would do it and have to take caution if their QP is of type 3 (both unicast and mcast). If you don't like this direction, your idea from below to have two option for group type - rdmacm or ipoib and have the consumer specify it, so for group type ipoib you will use the ipv4 brd qkey for both join and modify qp and for group type rdmacm you would just use the rdmacm qkey and do no modify qp - this is fine for us as well. >> Bottom line, Looking in the IB SPEC and IPoIB RFC i did not see >> mentioning of privileged QKEY. > > From RFC 4391 (ipoib RFC), 4.1: > > 2. Q_Key > > It is RECOMMENDED that a controlled Q_Key be used with the > high-order bit set. This is to prevent non-privileged > software from fabricating and sending out bogus IP datagrams. > > I don't know what qkey is actually assigned, however. this (what qkey is assigned to the ipv4 broadcast group by different SAs) is orthogonal to the discussion we do here. > I have some path forward related tasks that I would like to complete > before starting on this. I hope to finish that before the end of this > week. I don't want to rush on the multicast support and miss > something. For the rdma cm, we may need to let the user set some > options when joining a multicast group. Maybe something like: join type > (send-only or send-receive), group type (ipoib or rdma defined), etc. As I see it, the group type (or having no types and being always interoperable with ipoib as i suggest above) seems easy to add to the current implementation and would put it in acceptable state for upstream pushing to 2.6.21 and inclusion in OFED 1.2 . As for the join type, as i told you before, I think it should --not-- delay the upstream nor the ofed 1.2 push - if you have the time add this to the user/kernel ABI and have ucma kernel return -EINVAL if someone attempts to to send-only join. And if you don't have the time for that, it can be added later. Actually, as you can see in the ipoib code, it never does a send-only-non-member join, so my take here is that till the ipoib issue is resolved there is no reason to have this complexity in the rdmacm. > I do plan on requesting that the core multicast changes to ib_sa and > ib_ipoib be pulled into 2.6.21. This is great news but again I think the "nobody perfect" rule applies well here, the current rdmacm multicast support (which the little fixes we discuss over this thread) can be pushed to 2.6.21 and be enhanced later. Or. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] multicast code/merge status
> sure, it can use the rdmacm qkey (0x1234567 etc) when it creates the QP > and later --if-- the user joins a multicast group modify the qp state > with the group qkey and report it in the cma event such that the > consumer of the rdmacm would set this into his IB UD TX WR Changing the qkey would break its existing UD communication. > Bottom line, Looking in the IB SPEC and IPoIB RFC i did not see > mentioning of privileged QKEY. From RFC 4391 (ipoib RFC), 4.1: 2. Q_Key It is RECOMMENDED that a controlled Q_Key be used with the high-order bit set. This is to prevent non-privileged software from fabricating and sending out bogus IP datagrams. I don't know what qkey is actually assigned, however. I have some path forward related tasks that I would like to complete before starting on this. I hope to finish that before the end of this week. I don't want to rush on the multicast support and miss something. For the rdma cm, we may need to let the user set some options when joining a multicast group. Maybe something like: join type (send-only or send-receive), group type (ipoib or rdma defined), etc. I do plan on requesting that the core multicast changes to ib_sa and ib_ipoib be pulled into 2.6.21. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] multicast code/merge status
Sean Hefty wrote:
>> mimic IPoIB qkey flow:
>> +3 on rdma_create_qp do modify qp with some def qkey (eg zero)
>> +4 on the join completion path before attaching a qp to the associated
>> mgid, do modify qp with this mrec qkey (=ipv4 broadcast one)
> The rdma cm allows UD QP communication, which requires a valid qkey
> before or without joining a multicast group. I'd like to find a way to
> continue to support this.
sure, it can use the rdmacm qkey (0x1234567 etc) when it creates the QP
and later --if-- the user joins a multicast group modify the qp state
with the group qkey and report it in the cma event such that the
consumer of the rdmacm would set this into his IB UD TX WR
>> +3 on rdma_create_qp do modify qp with some def qkey (eg zero)
>> +4 on the join completion path before attaching a qp to the associated
>> mgid, do modify qp with this mrec qkey (=ipv4 broadcast one)
> Isn't the ipoib qkey a privileged qkey?
looking in ipoib code you can see the following code in
ipoib_mcast_join_task
> if (!test_bit(IPOIB_MCAST_FLAG_ATTACHED, &priv->broadcast->flags)) {
> ipoib_mcast_join(dev, priv->broadcast, 0);
> return;
> }
so ipoib_mcast_join is called with create=0 for the broadcast group and
this makes
it provide a component mask of
> comp_mask =
> IB_SA_MCMEMBER_REC_MGID |
> IB_SA_MCMEMBER_REC_PORT_GID |
> IB_SA_MCMEMBER_REC_PKEY |
> IB_SA_MCMEMBER_REC_JOIN_STATE;
that is the SA sets the QKEY, RATE, MTU, SL etc etc for the broadcast
group and later other
any joins done by ipoib uses the priv->broadcast->mcmember fields
So the broadcast qkey is basically what the SA has set when it created
the group.
During my talking here i got a pointer to section 10 in the IPoIB RFC
(4391)
mentioning something like "some 3rd party --has-- to create the
broadcast group":
> 10. Sending and Receiving IP Multicast Packets
> A node joining an IP multicast group must first construct an MGID
>according to the rule described in section 4 above. Once the correct
>MGID is calculated, the node must call the SA of the outbound link to
>attempt a "FullMember" join of the IB multicast group corresponding
>to the MGID. If the IB multicast group does not already exist, one
>must be created first with the IPoIB link MTU. The MGID MUST use the
>same P_Key, Q_Key, SL, MTU, and HopLimit as those used in the
>broadcast-GID. The rest of attributes SHOULD follow the values used
>in the broadcast-GID as well.
Bottom line, Looking in the IB SPEC and IPoIB RFC i did not see
mentioning of privileged QKEY.
Or.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] multicast code/merge status
> mimic IPoIB qkey flow: > > +3 on rdma_create_qp do modify qp with some def qkey (eg zero) > +4 on the join completion path before attaching a qp to the associated > mgid, do modify qp with this mrec qkey (=ipv4 broadcast one) The rdma cm allows UD QP communication, which requires a valid qkey before or without joining a multicast group. I'd like to find a way to continue to support this. > +3 on rdma_create_qp do modify qp with some def qkey (eg zero) > +4 on the join completion path before attaching a qp to the associated > mgid, do modify qp with this mrec qkey (=ipv4 broadcast one) Isn't the ipoib qkey a privileged qkey? - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] multicast code/merge status
Sean Hefty wrote: >> OK, got you at last (sorry but i have somehow ignored the call to >> ib_addr_get_mgid() at the rdmacm code). So to achieve interop with >> IPoIB all we need to do is remove the rdmacm signature bit and not to >> over-write the rdmacm qkey on the the qkey of the ipoib ipv4 broadcast >> group, are you ok with that? > I believe this would achieve interop with ipoib. However, overwriting > the qkey may break any existing UD communication that the user may > have. I just need to think about this more, and see what we can come up > with. Hi Sean, Based on our communication so far, the elements which are missing are ++ on the rdmacm kernel code: (drivers/infiniband/core/cma.c) +1 remove the rdmacm signature byte from the mgid +2 get the qkey used by the ipv4 broadcast group and use it mimic IPoIB qkey flow: +3 on rdma_create_qp do modify qp with some def qkey (eg zero) +4 on the join completion path before attaching a qp to the associated mgid, do modify qp with this mrec qkey (=ipv4 broadcast one) ++ on the rdmacm user space code: (librdmacm/src/cma.c) +3 on rdma_create_qp do modify qp with some def qkey (eg zero) +4 on the join completion path before attaching a qp to the associated mgid, do modify qp with this mrec qkey (=ipv4 broadcast one) With the time frame for 2.6.21 and OFED 1.2 becoming short, can you update of the multicast patch series status? We really want it in for this time frame, please let me know if you prefer to get patches that implement the above (eg as reference) or do it yourself... thanks, Or. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] multicast code/merge status
On Tue, 2007-01-09 at 13:06, Sean Hefty wrote: > > My thought re this was that since the rest of the original patch > > sequence specifically the rdma_cm UDP and ucma code are merged in > > 2.6.20-rcX which is the code OFED 1.2 is based on, the easy path for you > > would be to stage the multicast code for upstream push to 2.6.21 and > > then push the code as to OFED 1.2, what do you think? > > I think shooting for 2.6.21 is fine, but... > > > Other then that, as we discussed in SC06 there are some changes that > > need to be integrated in the code to allow for interoperability between > > a multicast rdma cm based app to IPoIB, specifically removing the RDMA > > CM signature from the mgid which generated from the ip addr and pkey, > > but not only. > > ...I have not completed these changes yet. Specifically, I have not added a > send only join parameter or changed the qkey. > > I have also not full examined an issue where the SM log fills up with bad > multicast join requests. Are these send only joins ? If so, it is because the group does not yet exist (you need a receiver first or it to be precreated). The later is problematic for dynamic IPmc groups. > > The second change is related to the qkey, looking in the current code > > of cma_join_ib_multicast() (at the multicast-sa_cache branch of the > > rdma-dev git) i see that the qkey is the mc ip address, which is not > > consistent with what librdmacm is assuming (0x1234567 etc). > > This is a bug in the kernel code. It should be using the standard qkey of > 0x12345678 - for now anyway. Where does this standard qkey come from ? (I missed it). -- Hal > > Anyway, what we need here is to plug into the scheme of ipoib which uses > > the qkey associated with the ipv4 broadcast multicast group. It turns > > out that there is some twilight zone here which i am working to > > understand better. You can see that for the ipv4 brd group ipoib lets > > the SM to allocate the group and qkey (ie the create param of > > ipoib_mcast_join is zero), i will give it some thought and let you know > > how i think the rdma cm can plug into this scheme, will be happy to get > > other ideas as well. > > The rdma_cm knows the qkey that ipoib uses before it joins a multicast group. > See cma_join_ib_multicast() - call to ib_sa_get_mcmember_rec(). > > - Sean > > ___ > openib-general mailing list > openib-general@openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] multicast code/merge status
On Wed, 2007-01-10 at 14:34, Or Gerlitz wrote: > On 1/10/07, Sean Hefty <[EMAIL PROTECTED]> wrote: > > > mmm, weird, are there active rdmacm multicast consumers involved in > > > this settings? > > > > I believe the issue is restricted to ipoib only. > > What does it means in this context that ipoib attempts to join a non > exisiting group, does it mean that ipoib assumes that the sm would > create the group? IPmc senders do send only joins and in IB this is insufficient to create the group. The group would need to be precreated or setup by the receiver prior to the sender. -- Hal > > > OK, got you at last (sorry but i have somehow ignored the call to > > > ib_addr_get_mgid() at the rdmacm code). So to achieve interop with > > > IPoIB all we need to do is remove the rdmacm signature bit and not to > > > over-write the rdmacm qkey on the the qkey of the ipoib ipv4 broadcast > > > group, are you ok with that? > > > I believe this would achieve interop with ipoib. However, overwriting the > > qkey > > may break any existing UD communication that the user may have. I just > > need to > > think about this more, and see what we can come up with. > > Basically, since there is only one QKEY per UD QP, a user IB UD app > should either set a qp per qkey or take the same approach as IPoIB > does, one qp, one qkey, all unicast/mcast traffic and mcast joins use > the same qkey, it sounds fine to me. > > Or. > > ___ > openib-general mailing list > openib-general@openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] multicast code/merge status
On 10 Jan 2007 14:31:38 -0500, Hal Rosenstock <[EMAIL PROTECTED]> wrote: > On Wed, 2007-01-10 at 13:47, Or Gerlitz wrote: > > (*) there are some more issues here which need to be addressed, see > > for example the "Some SMs don't support send-only yet" weird comment > > at ipoib_mcast_sendonly_join() > It's more likely an SA issue but I'm only guessing... It may also be > historical... We are not a huge community, how about asking the person who put this comment to come and say "i did it" and "it was done b/c this or that" Or. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] multicast code/merge status
On 1/10/07, Sean Hefty <[EMAIL PROTECTED]> wrote: > > mmm, weird, are there active rdmacm multicast consumers involved in > > this settings? > > I believe the issue is restricted to ipoib only. What does it means in this context that ipoib attempts to join a non exisiting group, does it mean that ipoib assumes that the sm would create the group? > > OK, got you at last (sorry but i have somehow ignored the call to > > ib_addr_get_mgid() at the rdmacm code). So to achieve interop with > > IPoIB all we need to do is remove the rdmacm signature bit and not to > > over-write the rdmacm qkey on the the qkey of the ipoib ipv4 broadcast > > group, are you ok with that? > I believe this would achieve interop with ipoib. However, overwriting the > qkey > may break any existing UD communication that the user may have. I just need > to > think about this more, and see what we can come up with. Basically, since there is only one QKEY per UD QP, a user IB UD app should either set a qp per qkey or take the same approach as IPoIB does, one qp, one qkey, all unicast/mcast traffic and mcast joins use the same qkey, it sounds fine to me. Or. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] multicast code/merge status
On Wed, 2007-01-10 at 13:47, Or Gerlitz wrote: > On 1/10/07, Sean Hefty <[EMAIL PROTECTED]> wrote: > >> How about adding sendonly param to the ABI and having the ucma kernel > >> code returning -EINVAL if someone tries to set it to true. Such code can > >> be pushed to 2.6.21 and when you have the time to complete the > >> implementation you can complete this? > > > I don't think adding this is a huge deal; I just haven't gotten to it yet. > > However, I'd like to make sure there's enough time once the change is made > > to > > verify that we have the right result before pushing it upstream. > > Compared to the amount of functionality (ib_sa ref mcast counting and > rdma cm u/k mcast API) provided by this code, i really think it can be > done in two phases, first push a code that does not support sendonly > and later once its implemented push a patch that adds this. > > Anyway, per the best of my knowledge you would not be able to fully > verify the "right" result , i understand the opensm does not really > support a sendonly join (*) in the sense of that it does not support > configuring "one-way" MFTs for a subset of the branches/leaves of the > multicast group spanning tree, but this is orthogonal to your code. Yes, this is orthogonal and only affects strict conformance vis a vis the reception of packets for this group which would be discarded anyhow. This is an optimization provided by the spec. > (*) there are some more issues here which need to be addressed, see > for example the "Some SMs don't support send-only yet" weird comment > at ipoib_mcast_sendonly_join() It's more likely an SA issue but I'm only guessing... It may also be historical... -- Hal > > Woody has also seen this issue. And of course, I can't reproduce it on my > > systems, but I'm actively looking into the problem. It looks like some > > sort of > > issue with ipoib trying to join a non-existent multicast group. > > mmm, weird, are there active rdmacm multicast consumers involved in > this settings? > > >> Looking on the code, i understand that if an multicast consumer attempts > >> to join a group for which another consumer is already joined then it > >> just gets the group params, that is the mgid is your discriminator (with > >> the exception of an all zeros mgid which has a different treatment) > >> which makes much sense to me. > > > Not exactly. The rdma_cm consumer gets the group parameters for the ipoib > > broadcast group. It uses this information as a template for joining new > > groups. > > OK, got you at last (sorry but i have somehow ignored the call to > ib_addr_get_mgid() at the rdmacm code). So to achieve interop with > IPoIB all we need to do is remove the rdmacm signature bit and not to > over-write the rdmacm qkey on the the qkey of the ipoib ipv4 broadcast > group, are you ok with that? > > > One issue is that an rdma_cm consumer can first allocate a UD QP to use > > with UD > > traffic. When it later joins a multicast group, the qkey must be the same. > > How > > does ipoib handle this? > > Well, ipoib first sets a zero qkey into its qp in ipoib_init_qp() and > later in ipoib_mcast_attach() does another qp modify providing > priv->qkey which is the ipv4 broadcast group one, the rdma_cm can > mimic this behaviour for qps created with rdma_create_qp and also for > those created by the user. > > >> Since for our apps needs we do intend to join the 224.0.0.1 group, > >> resolving a) above is fine for us --> we will join 224.0.0.1 above, > >> provide the qkey to the rdma cm and it will join to the other group (eg > >> 224.5.5.5) with this qkey. > > > I'm not completely following you on this yet. > > forget this, there is no need in such tricks since the rdmacm has the > ipv4 broadcast group qkey. > > >> can you remind me what the idea/trick here, aren't you supposed to > >> generate an mgid for this case? > > > This either returns an existing MCMemberRecord that this node has joined, > > or it > > fills out an MCMemberRecord that can be used to join a new group. If the > > mgid > > is zero, the SA will assign one. > > thanks > > Or. > > ___ > openib-general mailing list > openib-general@openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] multicast code/merge status
On 1/10/07, Michael S. Tsirkin <[EMAIL PROTECTED]> wrote: > > On 1/10/07, Michael S. Tsirkin <[EMAIL PROTECTED]> wrote: > > > > Quoting Or Gerlitz <[EMAIL PROTECTED]>: > > > > Subject: Re: multicast code/merge status > > > > > Since Voltaire requested this code to be in OFED 1.2, could you please > > > clarify > > > whether the code in Sean's branch is in the shape you want it to be in > > > for OFED 1.2? > > > If rdma_cm multicast can't interoperate with IPoIB, is it still useful > > > for you? > > > > Yes, we want this code in OFED 1.2 and we want it to be inter operable > > with IPoIB, there was no concrete objection for the interop > > requirement and the details seem quite simple, we have narrowed it > > now to the removal of the rdmacm signature bit and usage of the ipv4 > > broadcast group qkey by the rdmacm. > I am just trying to figure out when's the best time to import the multicast > branch into OFED. So, there will still be changes there? Pending on Sean's approval, basically my take is that the code should be queued to 2.6.21 and then integrated right away into OFED 1.2. So any ipoib testing would excersize the ipoib/ib_sa changes, the are very few little changes expected, mostly an "ipoib interop" patch of two lines removing from cma.c the setting on an "rdmacm signature bit" at the mgid and usage of its own qkey. Or. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] multicast code/merge status
> mmm, weird, are there active rdmacm multicast consumers involved in > this settings? I believe the issue is restricted to ipoib only. > OK, got you at last (sorry but i have somehow ignored the call to > ib_addr_get_mgid() at the rdmacm code). So to achieve interop with > IPoIB all we need to do is remove the rdmacm signature bit and not to > over-write the rdmacm qkey on the the qkey of the ipoib ipv4 broadcast > group, are you ok with that? I believe this would achieve interop with ipoib. However, overwriting the qkey may break any existing UD communication that the user may have. I just need to think about this more, and see what we can come up with. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] multicast code/merge status
On 1/10/07, Sean Hefty <[EMAIL PROTECTED]> wrote: >> How about adding sendonly param to the ABI and having the ucma kernel >> code returning -EINVAL if someone tries to set it to true. Such code can >> be pushed to 2.6.21 and when you have the time to complete the >> implementation you can complete this? > I don't think adding this is a huge deal; I just haven't gotten to it yet. > However, I'd like to make sure there's enough time once the change is made to > verify that we have the right result before pushing it upstream. Compared to the amount of functionality (ib_sa ref mcast counting and rdma cm u/k mcast API) provided by this code, i really think it can be done in two phases, first push a code that does not support sendonly and later once its implemented push a patch that adds this. Anyway, per the best of my knowledge you would not be able to fully verify the "right" result , i understand the opensm does not really support a sendonly join (*) in the sense of that it does not support configuring "one-way" MFTs for a subset of the branches/leaves of the multicast group spanning tree, but this is orthogonal to your code. (*) there are some more issues here which need to be addressed, see for example the "Some SMs don't support send-only yet" weird comment at ipoib_mcast_sendonly_join() > Woody has also seen this issue. And of course, I can't reproduce it on my > systems, but I'm actively looking into the problem. It looks like some sort > of > issue with ipoib trying to join a non-existent multicast group. mmm, weird, are there active rdmacm multicast consumers involved in this settings? >> Looking on the code, i understand that if an multicast consumer attempts >> to join a group for which another consumer is already joined then it >> just gets the group params, that is the mgid is your discriminator (with >> the exception of an all zeros mgid which has a different treatment) >> which makes much sense to me. > Not exactly. The rdma_cm consumer gets the group parameters for the ipoib > broadcast group. It uses this information as a template for joining new > groups. OK, got you at last (sorry but i have somehow ignored the call to ib_addr_get_mgid() at the rdmacm code). So to achieve interop with IPoIB all we need to do is remove the rdmacm signature bit and not to over-write the rdmacm qkey on the the qkey of the ipoib ipv4 broadcast group, are you ok with that? > One issue is that an rdma_cm consumer can first allocate a UD QP to use with > UD > traffic. When it later joins a multicast group, the qkey must be the same. > How > does ipoib handle this? Well, ipoib first sets a zero qkey into its qp in ipoib_init_qp() and later in ipoib_mcast_attach() does another qp modify providing priv->qkey which is the ipv4 broadcast group one, the rdma_cm can mimic this behaviour for qps created with rdma_create_qp and also for those created by the user. >> Since for our apps needs we do intend to join the 224.0.0.1 group, >> resolving a) above is fine for us --> we will join 224.0.0.1 above, >> provide the qkey to the rdma cm and it will join to the other group (eg >> 224.5.5.5) with this qkey. > I'm not completely following you on this yet. forget this, there is no need in such tricks since the rdmacm has the ipv4 broadcast group qkey. >> can you remind me what the idea/trick here, aren't you supposed to >> generate an mgid for this case? > This either returns an existing MCMemberRecord that this node has joined, or > it > fills out an MCMemberRecord that can be used to join a new group. If the mgid > is zero, the SA will assign one. thanks Or. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] multicast code/merge status
> On 1/10/07, Michael S. Tsirkin <[EMAIL PROTECTED]> wrote: > > > Quoting Or Gerlitz <[EMAIL PROTECTED]>: > > > Subject: Re: multicast code/merge status > > > Since Voltaire requested this code to be in OFED 1.2, could you please > > clarify > > whether the code in Sean's branch is in the shape you want it to be in for > > OFED 1.2? > > If rdma_cm multicast can't interoperate with IPoIB, is it still useful for > > you? > > Yes, we want this code in OFED 1.2 and we want it to be inter operable > with IPoIB, there was no concrete objection for the interop > requirement and the details seem quite simple, we have narrowed it > now to the removal of the rdmacm signature bit and usage of the ipv4 > broadcast group qkey by the rdmacm. I am just trying to figure out when's the best time to import the multicast branch into OFED. So, there will still be changes there? > The sendonly issue is not related to the interop and we can do well > without it being implemneted for OFED 1.2 OK. -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] multicast code/merge status
On 1/10/07, Michael S. Tsirkin <[EMAIL PROTECTED]> wrote: > > Quoting Or Gerlitz <[EMAIL PROTECTED]>: > > Subject: Re: multicast code/merge status > Since Voltaire requested this code to be in OFED 1.2, could you please clarify > whether the code in Sean's branch is in the shape you want it to be in for > OFED 1.2? > If rdma_cm multicast can't interoperate with IPoIB, is it still useful for > you? Yes, we want this code in OFED 1.2 and we want it to be inter operable with IPoIB, there was no concrete objection for the interop requirement and the details seem quite simple, we have narrowed it now to the removal of the rdmacm signature bit and usage of the ipv4 broadcast group qkey by the rdmacm. The sendonly issue is not related to the interop and we can do well without it being implemneted for OFED 1.2 Or. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] multicast code/merge status
> Quoting Or Gerlitz <[EMAIL PROTECTED]>: > Subject: Re: multicast code/merge status > > Sean Hefty wrote: > >> Other then that, as we discussed in SC06 there are some changes that > >> need to be integrated in the code to allow for interoperability > >> between a multicast rdma cm based app to IPoIB, specifically removing > >> the RDMA CM signature from the mgid which generated from the ip addr > >> and pkey, but not only. > > > ...I have not completed these changes yet. Specifically, I have not > > added a send only join parameter or changed the qkey. > > OK, I understand that adding a send only join param changes the > librdmacm/ucma ABI and further that you might be somehow busy to fully > implement the sendonly scheme at the multicast code for the 2.6.21 time > frame. > > How about adding sendonly param to the ABI and having the ucma kernel > code returning -EINVAL if someone tries to set it to true. Such code can > be pushed to 2.6.21 and when you have the time to complete the > implementation you can complete this? Since Voltaire requested this code to be in OFED 1.2, could you please clarify whether the code in Sean's branch is in the shape you want it to be in for OFED 1.2? If rdma_cm multicast can't interoperate with IPoIB, is it still useful for you? -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] multicast code/merge status
> OK, I understand that adding a send only join param changes the
> librdmacm/ucma ABI and further that you might be somehow busy to fully
> implement the sendonly scheme at the multicast code for the 2.6.21 time
> frame.
>
> How about adding sendonly param to the ABI and having the ucma kernel
> code returning -EINVAL if someone tries to set it to true. Such code can
> be pushed to 2.6.21 and when you have the time to complete the
> implementation you can complete this?
I don't think adding this is a huge deal; I just haven't gotten to it yet.
However, I'd like to make sure there's enough time once the change is made to
verify that we have the right result before pushing it upstream.
> Is it what Dotan has reported? i recall the test does not use librdmacm
> nor IPoIB, so how does it exercise the kernel ib_sa api at all ??? i
> guess it uses libibmad or libibumad to send the joins etc.
Woody has also seen this issue. And of course, I can't reproduce it on my
systems, but I'm actively looking into the problem. It looks like some sort of
issue with ipoib trying to join a non-existent multicast group.
> Looking on the code, i understand that if an multicast consumer attempts
> to join a group for which another consumer is already joined then it
> just gets the group params, that is the mgid is your discriminator (with
> the exception of an all zeros mgid which has a different treatment)
> which makes much sense to me.
Not exactly. The rdma_cm consumer gets the group parameters for the ipoib
broadcast group. It uses this information as a template for joining new groups.
> Going forward with this idea, a cma consumer that wants to use the ipv4
> broadcast group qkey can join the group and learn the qkey.
One issue is that an rdma_cm consumer can first allocate a UD QP to use with UD
traffic. When it later joins a multicast group, the qkey must be the same.
How
does ipoib handle this?
> Since for our apps needs we do intend to join the 224.0.0.1 group,
> resolving a) above is fine for us --> we will join 224.0.0.1 above,
> provide the qkey to the rdma cm and it will join to the other group (eg
> 224.5.5.5) with this qkey.
>
> what do you think?
I'm not completely following you on this yet.
>> } else {
>> memset(rec, 0, sizeof *rec);
>> ib_get_cached_gid(device, port_num, 0, &rec->port_gid);
>> rec->pkey = 0x;
>> get_random_bytes(&rec->qkey, sizeof rec->qkey);
>> rec->join_state = 1;
>
>
> can you remind me what the idea/trick here, aren't you supposed to
> generate an mgid for this case?
This either returns an existing MCMemberRecord that this node has joined, or it
fills out an MCMemberRecord that can be used to join a new group. If the mgid
is zero, the SA will assign one.
- Sean
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] multicast code/merge status
Sean Hefty wrote:
>> Other then that, as we discussed in SC06 there are some changes that
>> need to be integrated in the code to allow for interoperability
>> between a multicast rdma cm based app to IPoIB, specifically removing
>> the RDMA CM signature from the mgid which generated from the ip addr
>> and pkey, but not only.
> ...I have not completed these changes yet. Specifically, I have not
> added a send only join parameter or changed the qkey.
OK, I understand that adding a send only join param changes the
librdmacm/ucma ABI and further that you might be somehow busy to fully
implement the sendonly scheme at the multicast code for the 2.6.21 time
frame.
How about adding sendonly param to the ABI and having the ucma kernel
code returning -EINVAL if someone tries to set it to true. Such code can
be pushed to 2.6.21 and when you have the time to complete the
implementation you can complete this?
> I have also not full examined an issue where the SM log fills up with
> bad multicast join requests.
Is it what Dotan has reported? i recall the test does not use librdmacm
nor IPoIB, so how does it exercise the kernel ib_sa api at all ??? i
guess it uses libibmad or libibumad to send the joins etc.
>> The second change is related to the qkey, looking in the current code
>> of cma_join_ib_multicast() (at the multicast-sa_cache branch of the
>> rdma-dev git) i see that the qkey is the mc ip address, which is not
>> consistent with what librdmacm is assuming (0x1234567 etc).
>
> This is a bug in the kernel code. It should be using the standard qkey
> of 0x12345678 - for now anyway.
OK
>> Anyway, what we need here is to plug into the scheme of ipoib which
>> uses the qkey associated with the ipv4 broadcast multicast group. It
>> turns out that there is some twilight zone here which i am working to
>> understand better. You can see that for the ipv4 brd group ipoib lets
>> the SM to allocate the group and qkey (ie the create param of
>> ipoib_mcast_join is zero), i will give it some thought and let you
>> know how i think the rdma cm can plug into this scheme, will be happy
>> to get other ideas as well.
> The rdma_cm knows the qkey that ipoib uses before it joins a multicast
> group. See cma_join_ib_multicast() - call to ib_sa_get_mcmember_rec().
Looking on the code, i understand that if an multicast consumer attempts
to join a group for which another consumer is already joined then it
just gets the group params, that is the mgid is your discriminator (with
the exception of an all zeros mgid which has a different treatment)
which makes much sense to me.
Going forward with this idea, a cma consumer that wants to use the ipv4
broadcast group qkey can join the group and learn the qkey.
However, there are two problems with this approach
a) it can't provide the qkey to the rdma cm for another group it want to
join and assuming the --local-- ipoib is not joined on the other group,
we are back to the original problem.
b) assuming the above problem is solved, the cma consumer must stay
on-line (ie not leave) with the broadcast group and hence will get all
the ipv4 broadcast traffic of the cluster.
We can assume that at least some of the multicast traffic of the node is
routed to an ipoib subnet, we can further assume that the net stack
would cause ipoib to join to the mgroup related to the "all hosts" ipv4
address --> 224.0.0.1
Since for our apps needs we do intend to join the 224.0.0.1 group,
resolving a) above is fine for us --> we will join 224.0.0.1 above,
provide the qkey to the rdma cm and it will join to the other group (eg
224.5.5.5) with this qkey.
what do you think?
> int ib_sa_get_mcmember_rec(struct ib_device *device, u8 port_num,
> union ib_gid *mgid, struct ib_sa_mcmember_rec *rec)
> {
> struct mcast_device *dev;
> struct mcast_port *port;
> struct mcast_group *group;
> unsigned long flags;
> int ret = 0;
>
> dev = ib_get_client_data(device, &mcast_client);
> if (!dev)
> return -ENODEV;
>
> port = &dev->port[port_num - dev->start_port];
> if (mgid && memcmp(mgid, &mgid0, sizeof mgid0)) {
> spin_lock_irqsave(&port->lock, flags);
> group = mcast_find(port, mgid);
> if (group)
> *rec = group->rec;
> else
> ret = -EADDRNOTAVAIL;
> spin_unlock_irqrestore(&port->lock, flags);
> } else {
> memset(rec, 0, sizeof *rec);
> ib_get_cached_gid(device, port_num, 0, &rec->port_gid);
> rec->pkey = 0x;
> get_random_bytes(&rec->qkey, sizeof rec->qkey);
> rec->join_state = 1;
can you remind me what the idea/trick here, aren't you supposed to
generate an mgid for this case?
> }
>
> return ret;
> }
Or.
___
openib-gen
Re: [openib-general] multicast code/merge status
> My thought re this was that since the rest of the original patch > sequence specifically the rdma_cm UDP and ucma code are merged in > 2.6.20-rcX which is the code OFED 1.2 is based on, the easy path for you > would be to stage the multicast code for upstream push to 2.6.21 and > then push the code as to OFED 1.2, what do you think? I think shooting for 2.6.21 is fine, but... > Other then that, as we discussed in SC06 there are some changes that > need to be integrated in the code to allow for interoperability between > a multicast rdma cm based app to IPoIB, specifically removing the RDMA > CM signature from the mgid which generated from the ip addr and pkey, > but not only. ...I have not completed these changes yet. Specifically, I have not added a send only join parameter or changed the qkey. I have also not full examined an issue where the SM log fills up with bad multicast join requests. > The second change is related to the qkey, looking in the current code > of cma_join_ib_multicast() (at the multicast-sa_cache branch of the > rdma-dev git) i see that the qkey is the mc ip address, which is not > consistent with what librdmacm is assuming (0x1234567 etc). This is a bug in the kernel code. It should be using the standard qkey of 0x12345678 - for now anyway. > Anyway, what we need here is to plug into the scheme of ipoib which uses > the qkey associated with the ipv4 broadcast multicast group. It turns > out that there is some twilight zone here which i am working to > understand better. You can see that for the ipv4 brd group ipoib lets > the SM to allocate the group and qkey (ie the create param of > ipoib_mcast_join is zero), i will give it some thought and let you know > how i think the rdma cm can plug into this scheme, will be happy to get > other ideas as well. The rdma_cm knows the qkey that ipoib uses before it joins a multicast group. See cma_join_ib_multicast() - call to ib_sa_get_mcmember_rec(). - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] multicast code/merge status
Michael S. Tsirkin wrote: >>> I currently have these in separate branches. I know that Woody has them >>> combined in his tree, but I can create an 'ofed' branch with these together. >> I published a 'multicast-sa_cache' branch (which was a little clearer than >> 'ofed'). I can change the name if more patches need to be pulled in. > > Cool, thanks, I'll try to put these in ofed next week. Hi Sean, My thought re this was that since the rest of the original patch sequence specifically the rdma_cm UDP and ucma code are merged in 2.6.20-rcX which is the code OFED 1.2 is based on, the easy path for you would be to stage the multicast code for upstream push to 2.6.21 and then push the code as to OFED 1.2, what do you think? Other then that, as we discussed in SC06 there are some changes that need to be integrated in the code to allow for interoperability between a multicast rdma cm based app to IPoIB, specifically removing the RDMA CM signature from the mgid which generated from the ip addr and pkey, but not only. The second change is related to the qkey, looking in the current code of cma_join_ib_multicast() (at the multicast-sa_cache branch of the rdma-dev git) i see that the qkey is the mc ip address, which is not consistent with what librdmacm is assuming (0x1234567 etc). Anyway, what we need here is to plug into the scheme of ipoib which uses the qkey associated with the ipv4 broadcast multicast group. It turns out that there is some twilight zone here which i am working to understand better. You can see that for the ipv4 brd group ipoib lets the SM to allocate the group and qkey (ie the create param of ipoib_mcast_join is zero), i will give it some thought and let you know how i think the rdma cm can plug into this scheme, will be happy to get other ideas as well. Or. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
