Re: [openib-general] [Fwd: RE: IB mcast question]
Steve Wise wrote: > - UD QPs created via librdma will have the same qkey > - mcast groups created via librdma will use this same qkey as well > - qkey value will not have the high order bit set This is what I would do as well. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [Fwd: RE: IB mcast question]
On Tue, 2006-08-15 at 14:40 -0700, Sean Hefty wrote: > Then, as Roland said, in order to support a QP using multiple multicast > groups, > we need all QKeys to be the same (for the RDMA CM). If we want to do anything > with port numbers, we probably need to fold those into the MGID. > > - Sean Ok, (now that I've crawled back out of my qkey rathole ;-), what's the plan for fixing this? I suggest we get it working with a single qkey and not worry about trying to support the UDP/IP/mcast port concept: Proposal: - UD QPs created via librdma will have the same qkey - mcast groups created via librdma will use this same qkey as well - qkey value will not have the high order bit set Sound good? ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [Fwd: RE: IB mcast question]
On Tue, 2006-08-15 at 17:49, Steve Wise wrote: > > > According to the spec, C10-15 sez the qkey in the outgoing packet will > > > be the qkey from the QP context IF the high order bit of the qkey is > > > set. If the high order bit is _not_ set, then the outgoing packet will > > > contain the qkey from the WR. (why? why?) > > > > The high order bit is for controlled QKeys which are only supposed to be > > sent by "privileged" consumers (e.g. MAD layer for QP1 handing). > > I see. > > > Also, the C10-15 requirement does not speak to additional requirements > > on the Q_Keys themselves like the one I cited to Sean in a previous > > email in terms of multicast group addressing requirements. > > > > > Now, in my experiment, my mcast qkey was 0xe00a0a0a > > > > I don't think that should be allowed from user space. > > > > Ah. So rdma cm needs to not use these special qkeys... Yes, that's what I think. -- Hal > > > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [Fwd: RE: IB mcast question]
> > > > Where does it say that a mcast group is associated with any qkey at all? > > In order to create a multicast group, the Q_Key needs to be supplied. > See p. 912-3 o15-0.2.3 > Thanks Hal. I'm beginning to understand... :-) ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [Fwd: RE: IB mcast question]
> > According to the spec, C10-15 sez the qkey in the outgoing packet will > > be the qkey from the QP context IF the high order bit of the qkey is > > set. If the high order bit is _not_ set, then the outgoing packet will > > contain the qkey from the WR. (why? why?) > > The high order bit is for controlled QKeys which are only supposed to be > sent by "privileged" consumers (e.g. MAD layer for QP1 handing). I see. > Also, the C10-15 requirement does not speak to additional requirements > on the Q_Keys themselves like the one I cited to Sean in a previous > email in terms of multicast group addressing requirements. > > > Now, in my experiment, my mcast qkey was 0xe00a0a0a > > I don't think that should be allowed from user space. > Ah. So rdma cm needs to not use these special qkeys... ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [Fwd: RE: IB mcast question]
>> 1) Doesn't need to be in the group (as sean pointed out)
>
>Yes it does. There are send only ("non") members in the JoinState
>component of the MCMemberRecord. This can be used to optimize the MC
>routing (as that port never needs to receive but needs to be able to
>send on the MLID).
I believe that the local port needs to be at least a SendOnly member (for
routing purposes), but the QP does not need to be attached to the group.
- Sean
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [Fwd: RE: IB mcast question]
On Tue, 2006-08-15 at 17:41, Steve Wise wrote: > On Tue, 2006-08-15 at 17:29 -0400, Hal Rosenstock wrote: > > On Tue, 2006-08-15 at 17:18, Sean Hefty wrote: > > > > > App 1: > > > > > Creates QP with QKey=22 > > > > > Joins multicast group 1 with QKey=33 > > > > > > > > > > App 2: > > > > > Creates QP with QKey=44 > > > > > Joins multicast group 1 > > > > > Sends to multicast group but with QKey=22 > > > > > > > >I think that last send is technically an IBA spec violation. > > > > > > I'll admit that this definitely seems like a hack, but I haven't found > > > where it > > > is violation. (I'll keep looking.) > > > > The first occurrence that I find is the following: > > > > p.145 13) where it states that "Further, in addition to having the same > > MGID, all members of the multicast group must share the same P_Key and > > Q_Key. > > > > Where does it say that a mcast group is associated with any qkey at all? In order to create a multicast group, the Q_Key needs to be supplied. See p. 912-3 o15-0.2.3 -- Hal > 13) just sez that all qp's that want to exchange datagrams must have the > same pkey/qkey. That's the normal UD paradigm. > > > > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [Fwd: RE: IB mcast question]
On Tue, 2006-08-15 at 17:40, Sean Hefty wrote: > >p.145 13) where it states that "Further, in addition to having the same > >MGID, all members of the multicast group must share the same P_Key and > >Q_Key. > > > >That is compliance per C4-3 on p. 143. > > Thanks - my take is that the compliance breaks when the QPs are first created > with the wrong QKey. Yes, I think so too. > Then, as Roland said, in order to support a QP using multiple multicast > groups, > we need all QKeys to be the same (for the RDMA CM). Does 1 QP need to support multiple multicast groups ? If so, I agree. That was the point I missed before. > If we want to do anything > with port numbers, we probably need to fold those into the MGID. I'm not quite following you yet on this. -- Hal > - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [Fwd: RE: IB mcast question]
On Tue, 2006-08-15 at 17:26, Steve Wise wrote:
> On Tue, 2006-08-15 at 14:18 -0700, Sean Hefty wrote:
> > > > App 1:
> > > > Creates QP with QKey=22
> > > > Joins multicast group 1 with QKey=33
> > > >
> > > > App 2:
> > > > Creates QP with QKey=44
> > > > Joins multicast group 1
> > > > Sends to multicast group but with QKey=22
> > >
> > >I think that last send is technically an IBA spec violation.
> >
> > I'll admit that this definitely seems like a hack, but I haven't found
> > where it
> > is violation. (I'll keep looking.) From 10.5.2.1, a QP doesn't need to be
> > attached to a multicast group to initiate a send, so it seems like any
> > potential
> > violation is only that the QKey and MLID wouldn't match.
> >
> > - Sean
>
> And I can't find in the IBTA spec where they talk about the qkey field
> of a mcast group at all.
Not so fast... I know the requirements are spread out through the spec
and it's hard to piece them all together. I cited where this one is
located.
> qkeys are used to validate an incoming
> datagram. If the qkey in the packet doesn't match the qkey of the qp,
> then its dropped. This is an HCA ingress issue, not a mcast issue
> really.
>
> >From what I can gather, the sender:
>
> 1) Doesn't need to be in the group (as sean pointed out)
Yes it does. There are send only ("non") members in the JoinState
component of the MCMemberRecord. This can be used to optimize the MC
routing (as that port never needs to receive but needs to be able to
send on the MLID).
-- Hal
> 2) posts a SEND WR with an address handle for the the mcast group, the
> remote_qpn == 0xf, and the remote_qkey == to remote qp's qkey.
>
>
> Now why C10-15 is there, I dunno...but i'm sure somebody around here
> does...
>
> :)
>
>
>
>
>
>
>
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [Fwd: RE: IB mcast question]
On Tue, 2006-08-15 at 17:29 -0400, Hal Rosenstock wrote: > On Tue, 2006-08-15 at 17:18, Sean Hefty wrote: > > > > App 1: > > > > Creates QP with QKey=22 > > > > Joins multicast group 1 with QKey=33 > > > > > > > > App 2: > > > > Creates QP with QKey=44 > > > > Joins multicast group 1 > > > > Sends to multicast group but with QKey=22 > > > > > >I think that last send is technically an IBA spec violation. > > > > I'll admit that this definitely seems like a hack, but I haven't found > > where it > > is violation. (I'll keep looking.) > > The first occurrence that I find is the following: > > p.145 13) where it states that "Further, in addition to having the same > MGID, all members of the multicast group must share the same P_Key and > Q_Key. > Where does it say that a mcast group is associated with any qkey at all? 13) just sez that all qp's that want to exchange datagrams must have the same pkey/qkey. That's the normal UD paradigm. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [Fwd: RE: IB mcast question]
>p.145 13) where it states that "Further, in addition to having the same >MGID, all members of the multicast group must share the same P_Key and >Q_Key. > >That is compliance per C4-3 on p. 143. Thanks - my take is that the compliance breaks when the QPs are first created with the wrong QKey. Then, as Roland said, in order to support a QP using multiple multicast groups, we need all QKeys to be the same (for the RDMA CM). If we want to do anything with port numbers, we probably need to fold those into the MGID. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [Fwd: RE: IB mcast question]
On Tue, 2006-08-15 at 17:13, Steve Wise wrote: > On Tue, 2006-08-15 at 13:55 -0700, Roland Dreier wrote: > > > > App 1: > > > Creates QP with QKey=22 > > > Joins multicast group 1 with QKey=33 > > > > > > App 2: > > > Creates QP with QKey=44 > > > Joins multicast group 1 > > > Sends to multicast group but with QKey=22 > > > > I think that last send is technically an IBA spec violation. > > > > - R. > > You guys are confusing me. "sends to multicast group but with > QKey=22"... Does that mean you posted a SEND with the remote_qkey=22? > > According to the spec, C10-15 sez the qkey in the outgoing packet will > be the qkey from the QP context IF the high order bit of the qkey is > set. If the high order bit is _not_ set, then the outgoing packet will > contain the qkey from the WR. (why? why?) The high order bit is for controlled QKeys which are only supposed to be sent by "privileged" consumers (e.g. MAD layer for QP1 handing). Also, the C10-15 requirement does not speak to additional requirements on the Q_Keys themselves like the one I cited to Sean in a previous email in terms of multicast group addressing requirements. > Now, in my experiment, my mcast qkey was 0xe00a0a0a I don't think that should be allowed from user space. -- Hal > and the qp qkeys > were zero. So when the sender posted a SEND with remote_qkey=0xe00a0a0a, > the interface placed the qkey from sender's qp, which was zero, in the > packet. That's why it worked I guess... > > Now my head hurts... > > > > > ___ > openib-general mailing list > openib-general@openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [Fwd: RE: IB mcast question]
On Tue, 2006-08-15 at 17:18, Sean Hefty wrote: > > > App 1: > > > Creates QP with QKey=22 > > > Joins multicast group 1 with QKey=33 > > > > > > App 2: > > > Creates QP with QKey=44 > > > Joins multicast group 1 > > > Sends to multicast group but with QKey=22 > > > >I think that last send is technically an IBA spec violation. > > I'll admit that this definitely seems like a hack, but I haven't found where > it > is violation. (I'll keep looking.) The first occurrence that I find is the following: p.145 13) where it states that "Further, in addition to having the same MGID, all members of the multicast group must share the same P_Key and Q_Key. That is compliance per C4-3 on p. 143. There may be others. -- Hal > From 10.5.2.1, a QP doesn't need to be > attached to a multicast group to initiate a send, so it seems like any > potential > violation is only that the QKey and MLID wouldn't match. > > - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [Fwd: RE: IB mcast question]
On Tue, 2006-08-15 at 14:18 -0700, Sean Hefty wrote: > > > App 1: > > > Creates QP with QKey=22 > > > Joins multicast group 1 with QKey=33 > > > > > > App 2: > > > Creates QP with QKey=44 > > > Joins multicast group 1 > > > Sends to multicast group but with QKey=22 > > > >I think that last send is technically an IBA spec violation. > > I'll admit that this definitely seems like a hack, but I haven't found where > it > is violation. (I'll keep looking.) From 10.5.2.1, a QP doesn't need to be > attached to a multicast group to initiate a send, so it seems like any > potential > violation is only that the QKey and MLID wouldn't match. > > - Sean And I can't find in the IBTA spec where they talk about the qkey field of a mcast group at all. qkeys are used to validate an incoming datagram. If the qkey in the packet doesn't match the qkey of the qp, then its dropped. This is an HCA ingress issue, not a mcast issue really. >From what I can gather, the sender: 1) Doesn't need to be in the group (as sean pointed out) 2) posts a SEND WR with an address handle for the the mcast group, the remote_qpn == 0xf, and the remote_qkey == to remote qp's qkey. Now why C10-15 is there, I dunno...but i'm sure somebody around here does... :) ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [Fwd: RE: IB mcast question]
> > App 1: > > Creates QP with QKey=22 > > Joins multicast group 1 with QKey=33 > > > > App 2: > > Creates QP with QKey=44 > > Joins multicast group 1 > > Sends to multicast group but with QKey=22 > >I think that last send is technically an IBA spec violation. I'll admit that this definitely seems like a hack, but I haven't found where it is violation. (I'll keep looking.) From 10.5.2.1, a QP doesn't need to be attached to a multicast group to initiate a send, so it seems like any potential violation is only that the QKey and MLID wouldn't match. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [Fwd: RE: IB mcast question]
On Tue, 2006-08-15 at 13:55 -0700, Roland Dreier wrote: > > App 1: > > Creates QP with QKey=22 > > Joins multicast group 1 with QKey=33 > > > > App 2: > > Creates QP with QKey=44 > > Joins multicast group 1 > > Sends to multicast group but with QKey=22 > > I think that last send is technically an IBA spec violation. > > - R. You guys are confusing me. "sends to multicast group but with QKey=22"... Does that mean you posted a SEND with the remote_qkey=22? According to the spec, C10-15 sez the qkey in the outgoing packet will be the qkey from the QP context IF the high order bit of the qkey is set. If the high order bit is _not_ set, then the outgoing packet will contain the qkey from the WR. (why? why?) Now, in my experiment, my mcast qkey was 0xe00a0a0a and the qp qkeys were zero. So when the sender posted a SEND with remote_qkey=0xe00a0a0a, the interface placed the qkey from sender's qp, which was zero, in the packet. That's why it worked I guess... Now my head hurts... ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [Fwd: RE: IB mcast question]
> App 1: > Creates QP with QKey=22 > Joins multicast group 1 with QKey=33 > > App 2: > Creates QP with QKey=44 > Joins multicast group 1 > Sends to multicast group but with QKey=22 I think that last send is technically an IBA spec violation. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [Fwd: RE: IB mcast question]
>I don't see how this could work-- to mimic IP, an app has to be able >to multicast a datagram to multiple destination ports using the same >multicast group. Since IB multicast groups have a Q_Key associated, >the only way this could work is if all QPs and MCGs share one Q_Key. Hmm... I'd like to find a way to avoid using the same QKey for everything, but maybe it's not any worse than what we have with IP anyway. I guess if we know if a UD QP will not join a multicast group, we can use any QKey, but use a well known QKey if it will join a multicast group. What would happen in the following situation: App 1: Creates QP with QKey=22 Joins multicast group 1 with QKey=33 App 2: Creates QP with QKey=44 Joins multicast group 1 Sends to multicast group but with QKey=22 - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [Fwd: RE: IB mcast question]
On Tue, 2006-08-15 at 16:12 -0400, Hal Rosenstock wrote: > On Tue, 2006-08-15 at 15:59, Roland Dreier wrote: > > > This is what I was referring to. I'm really not familiar with IP > > multicast > > > beyond what I read in a book while coding the RDMA CM. It sounds like > > we might > > > be able to use the QKey as the port number for the QP to mimic the > > behavior. > > > > I don't see how this could work-- to mimic IP, an app has to be able > > to multicast a datagram to multiple destination ports using the same > > multicast group. Since IB multicast groups have a Q_Key associated, > > the only way this could work is if all QPs and MCGs share one Q_Key. > > Isn't it all QPs on the same MCG need to share one Q_Key ? > >From my experimentation, only the qkeys of the QPs need match. The MCG qkey is only needed for sending to the group. Am I confused? (often the case :) ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [Fwd: RE: IB mcast question]
On Tue, 2006-08-15 at 15:59, Roland Dreier wrote: > > This is what I was referring to. I'm really not familiar with IP multicast > > beyond what I read in a book while coding the RDMA CM. It sounds like we > might > > be able to use the QKey as the port number for the QP to mimic the > behavior. > > I don't see how this could work-- to mimic IP, an app has to be able > to multicast a datagram to multiple destination ports using the same > multicast group. Since IB multicast groups have a Q_Key associated, > the only way this could work is if all QPs and MCGs share one Q_Key. Isn't it all QPs on the same MCG need to share one Q_Key ? -- Hal > > ___ > openib-general mailing list > openib-general@openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [Fwd: RE: IB mcast question]
> This is what I was referring to. I'm really not familiar with IP multicast > beyond what I read in a book while coding the RDMA CM. It sounds like we > might > be able to use the QKey as the port number for the QP to mimic the behavior. I don't see how this could work-- to mimic IP, an app has to be able to multicast a datagram to multiple destination ports using the same multicast group. Since IB multicast groups have a Q_Key associated, the only way this could work is if all QPs and MCGs share one Q_Key. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] [Fwd: RE: IB mcast question]
accidentally dropped off the list... Forwarded Message From: Sean Hefty <[EMAIL PROTECTED]> To: 'Steve Wise' <[EMAIL PROTECTED]> Subject: RE: [openib-general] IB mcast question Date: Tue, 15 Aug 2006 11:59:12 -0700 FYI - your reply dropped off the list. >For type SOCK_DGRAM (UDP), the socket will receive packets from multiple >subscribed ip mcast groups iff the dst_port of the incoming packet >matches the port to which the socket is bound... This is what I was referring to. I'm really not familiar with IP multicast beyond what I read in a book while coding the RDMA CM. It sounds like we might be able to use the QKey as the port number for the QP to mimic the behavior. The RDMA CM sets the QKey for UD QPs to the port number, but sets the QKey of a multicast group to the IPv4 address. >NOTE: I'm just trying to understand how this works in IB. I'm not >necessarily advocating it should behave exactly like ip mcast/udp. Clients need to create an UD QP. When they join a multicast group, they get an MGID, MLID, and QKey. The UD QP needs to attach to the MGID / MLID, and have a matching QKey. Today, the RDMA CM assigns a QKey to a UD QP when it's created; it doesn't know if it will join a multicast group or not. >And if you want to support a single UD QP receiving from multiple >subscribed groups, you'll have to have the same Qkey for all the groups >and QPs. Right? I believe that this will be the case. A send can specify the remote QKey, so it may be that UD QP transfers are okay, and only multicast has an issue. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
