Re: [swg] Re: [openib-general] round 2 - proposal for socket based connectionmodel
Just to correct one comment: A ULP written to TCP/IP can use RDMA transport without change. An example is SDP not that the ULP must use what SDP uses. Also, please keep in mind that SDP on iWARP uses the port mapper protocol to obtain the IP address and port to target for the connection request. So, the TCP connection establishment is to the RDMA listen endpoint from the start and the SDP hello exchange then fills in the rest of the parameters required to determine whether the connection should proceed and what resources should be configured when the response is generated. I will also re-iterate what another person stated and that is to separate out the interface from the wire protocol. IBTA defines wire protocols / semantics while OpenIB is defining its API to communicate the wire protocol and associated semantics. I agree with that person on this point and their other point on the need for the IBTA to construct a solid spec for the wire protocol and associated semantics. OpenIB will then determine how best to implement but these are separate efforts and it would be more productive for all to table the discussion for now. The original request was whether something would break if the private data size was changed. It was noted that one cannot know what will or will not break thus the requirement is to provide a method for software to note the difference in the layout. How is for the IBTA to specify. Just a thought.. Mike At 03:43 PM 10/25/2005, Sean Hefty wrote: Kanevsky, Arkady wrote: What are you trying to achieve? I'm trying to define a connection *service* for Infiniband that uses TCP/IP addresses as its user interface. That service will have its own protocol, in much the same way that SDP, SRP, etc. do today. I am trying to define an IB REQ protocol extension that support IP connection 5-tuple exchange between connection requestor and responder. Why? What need is there for a protocol extension to the IB CM? To me, this is similar to setting a bit in the CM REQ to indicate that the private data format looks like SDP's private data. The format of the _private_ data shouldn't be known to the CM; that's why it's private data. And define mapping between IP 5-tuple and IB entities. No mapping between IP <-> IB addresses was defined in the proposal. Defining this mapping is required to make this work. Right now, the mapping is the responsibility of every user. That way ULP which was written to TCP/IP, UDP/IP, CSTP/IP (and so on) can use RDMA transport without change. A ULP written to TCP/IP can use an RDMA transport without change. They use SDP. However, an application that wants to take advantage of QP semantics must change. (And if they want to take full advantage of RDMA, they'll likely need to be re-architected as well.) The goal in that case becomes to permit them to establish connections using TCP/IP addresses. To meet this goal, we need to define how to map IP address to and from IB addresses. That mapping is part of the protocol, and is missing from the proposal. And if the application isn't going to know that they're running on Infiniband, then the mapping must also include mapping to a destination service ID. To modify ULP to know that it runs on top of IB vs. iWARP vs. (any other RDMA transport) is bad idea. It is one thing to choose proper port to connect. Completely different to ask ULP to parse private data in transport specific way. The same protocol must support both user level ULPs and kernel level ULPs. Defining an interface that allows a ULP to use either iWarp, IB, or some other random RDMA transport is an implementation issue. However, it requires something that maps IP to IB addresses (including service IDs). To be more concrete, you've gone from having source and destination TCP/IP addresses to including them in a CM REQ. What translated the source and destination IP addresses into GIDs and a PKey? Who converted those into IB routing information? How was the destination of the CM REQ determined? What service ID was selected? - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] round 2 - proposal for socket based connectionmodel
Sean Hefty wrote: > -Original Message- > From: Sean Hefty [mailto:[EMAIL PROTECTED] > Sent: Tuesday, October 25, 2005 6:44 PM > To: Kanevsky, Arkady > Cc: Sean Hefty; openib-general@openib.org; [EMAIL PROTECTED] > Subject: Re: [openib-general] round 2 - proposal for socket > based connectionmodel > > > Kanevsky, Arkady wrote: > > What are you trying to achieve? > > I'm trying to define a connection *service* for Infiniband > that uses TCP/IP > addresses as its user interface. That service will have its > own protocol, in > much the same way that SDP, SRP, etc. do today. > > > I am trying to define an IB REQ protocol extension that support IP > > connection 5-tuple exchange between connection requestor and > > responder. > > Why? What need is there for a protocol extension to the IB > CM? To me, this is > similar to setting a bit in the CM REQ to indicate that the > private data format > looks like SDP's private data. The format of the _private_ > data shouldn't be > known to the CM; that's why it's private data. There is no requirement that the remote side uses the same Linux CM. So in order to achieve interopability you need a protocol. SDP hello-world protocol is defined for SDP. We are defining an equivalent that is ULP independent. If CM is not involved then it is ULP that populate the 5-tuple info on requestor side and parses it on the remote side. Thus, make ULP CM IB specific. This is what we are trying to avoid. ULP should not change regardless whether or not it is running on IB, iWARP, VIA or any other RDMA transport. iWARP does not need private data to pass 5-tuple. > > > And define mapping between IP 5-tuple and IB entities. > > No mapping between IP <-> IB addresses was defined in the > proposal. Defining > this mapping is required to make this work. Right now, the > mapping is the > responsibility of every user. > > > That way ULP which was written to TCP/IP, UDP/IP, CSTP/IP > (and so on) > > can use RDMA transport without change. > > A ULP written to TCP/IP can use an RDMA transport without > change. They use SDP. > However, an application that wants to take advantage of QP > semantics must > change. (And if they want to take full advantage of RDMA, > they'll likely need > to be re-architected as well.) The goal in that case becomes > to permit them to > establish connections using TCP/IP addresses. > > To meet this goal, we need to define how to map IP address to > and from IB > addresses. That mapping is part of the protocol, and is > missing from the > proposal. And if the application isn't going to know that > they're running on > Infiniband, then the mapping must also include mapping to a > destination service ID. > > > To modify ULP to know that it runs on top of IB vs. iWARP > > vs. (any other RDMA transport) is bad idea. > > It is one thing to choose proper port to connect. > > Completely different to ask ULP to parse private data > > in transport specific way. > > The same protocol must support both user level ULPs > > and kernel level ULPs. > > Defining an interface that allows a ULP to use either iWarp, > IB, or some other > random RDMA transport is an implementation issue. However, > it requires > something that maps IP to IB addresses (including service IDs). > > To be more concrete, you've gone from having source and > destination TCP/IP > addresses to including them in a CM REQ. What translated the > source and > destination IP addresses into GIDs and a PKey? Who converted > those into IB > routing information? How was the destination of the CM REQ > determined? What > service ID was selected? IPoIB defines IP -> GID Port -> IB Service ID (part of this proposal) Pkey is configuration setup done by administrator. Ditto for VLAN. > > - Sean > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] round 2 - proposal for socket basedconnection model
No. iWARP does not have to pass this info. The info is needed for IB because ZB and SI were introduced in IBTA 1.2 specs as optional functionality. So if ULP wants to use that functionality it need to find out whether remote side can support it. This is needed for backwards compatibility. For example iSER protocol defines the use of remote invalidate but obviously can not be done if remote side can not support it. I do not recall right now whether iWARP defined that functionality as required or optional. Arkady Kanevsky email: [EMAIL PROTECTED] Network Appliance phone: 781-768-5395 375 Totten Pond Rd. Fax: 781-895-1195 Waltham, MA 02451-2010 central phone: 781-768-5300 > -Original Message- > From: Tom Tucker [mailto:[EMAIL PROTECTED] > Sent: Tuesday, October 25, 2005 5:56 PM > To: Kanevsky, Arkady > Cc: [EMAIL PROTECTED]; openib-general@openib.org > Subject: RE: [openib-general] round 2 - proposal for socket > basedconnection model > > > Arkady: > > I may actually have a constructive comment about the protocol > (private data format). One thing I noticed is that *almost* > everything in the private data header is available in the > native iWARP protocol header except the ZB and SI bits. If > these bits become part of the canonical private data header, > then does that require an iWARP transport to use the header > too even though only two bits are useful? > > Sorry if this is a dumb question, > > Tom > > On Tue, 2005-10-25 at 16:40 -0500, Tom Tucker wrote: > > Arkady: > > > > I don't think anyone disagrees with your goals. Unfortunately > > additional requirements on the implementation were coupled with the > > specification of the private data format (protocol). This > peripheral > > discussion derailed any attempt to discuss the protocol. > > > > Attempts to separate the protocol discussion from the > implementation > > failed. And so here we are... > > > > > > On Tue, 2005-10-25 at 15:38 -0400, Kanevsky, Arkady wrote: > > > What are you trying to achieve? > > > > > > I am trying to define an IB REQ protocol extension that > support IP > > > connection 5-tuple exchange between connection requestor and > > > responder. And define mapping between IP 5-tuple and IB entities. > > > > > > That way ULP which was written to TCP/IP, UDP/IP, CSTP/IP (and so > > > on) can use RDMA transport without change. To modify ULP to know > > > that it runs on top of IB vs. iWARP vs. (any other RDMA > transport) > > > is bad idea. It is one thing to choose proper port to connect. > > > Completely different to ask ULP to parse private data > > > in transport specific way. > > > > > > The same protocol must support both user level ULPs > > > and kernel level ULPs. > > > Arkady > > > > > > Arkady Kanevsky email: [EMAIL PROTECTED] > > > Network Appliance phone: 781-768-5395 > > > 375 Totten Pond Rd. Fax: 781-895-1195 > > > Waltham, MA 02451-2010 central phone: 781-768-5300 > > > > > > > > > > > > > -Original Message- > > > > From: Sean Hefty [mailto:[EMAIL PROTECTED] > > > > Sent: Tuesday, October 25, 2005 3:22 PM > > > > To: Kanevsky, Arkady > > > > Cc: Sean Hefty; openib-general@openib.org; [EMAIL PROTECTED] > > > > Subject: Re: [openib-general] round 2 - proposal for socket > > > > based connectionmodel > > > > > > > > > > > > Kanevsky, Arkady wrote: > > > > > Sean, > > > > > answers in-line. > > > > > Arkady > > > > > > > > At this point, I'm just going to disagree with this approach > > > > and move on with > > > > the current implementation of the CMA. What's needed is a > > > > service that provides > > > > IB connections using TCP/IP addressing. I don't believe this > > > > proposal meets > > > > this goal. > > > > > > > > To meet the requirement of connecting over IB using TCP/IP > > > > addressing, I believe > > > > that we need a service with a reserved service > identifier or range of > > > > identifiers, a mechanism for mapping between IP and IB > > > > addresses, and a > > > > mechanism for reversing the mapping. > > > > > > > > I don't see where the proposal addresses the bulk of the work > > > > that's required, > > > > nor do I think that it will present an API to the user that > > > > does not expose IB > > > > related addressing (such as service IDs). > > > > > > > > - Sean > > > > > > > ___ > > > openib-general mailing list > > > openib-general@openib.org > > > http://openib.org/mailman/listinfo/openib-general > > > > > > To unsubscribe, please visit > > > http://openib.org/mailman/listinfo/openib-general > > ___ > > openib-general mailing list > > openib-general@openib.org > > http://openib.org/mailman/listinfo/openib-general > > > > To unsubscribe, please visit > > http://openib.org/mailman/listinf
RE: [swg] Re: [openib-general] TCP/IP connection service over IB
DAPL also strip this private data header and present to Consumer IP addresses and ports as separate items from Consumer private data. Arkady Kanevsky email: [EMAIL PROTECTED] Network Appliance phone: 781-768-5395 375 Totten Pond Rd. Fax: 781-895-1195 Waltham, MA 02451-2010 central phone: 781-768-5300 > -Original Message- > From: Tom Tucker [mailto:[EMAIL PROTECTED] > Sent: Tuesday, October 25, 2005 5:52 PM > To: Ted H. Kim > Cc: [EMAIL PROTECTED]; openib-general > Subject: Re: [swg] Re: [openib-general] TCP/IP connection > service over IB > > > On Tue, 2005-10-25 at 13:16 -0700, Ted H. Kim wrote: > > Tom, > > > > Some comments inline ... > > > > > > Tom Tucker wrote: > > > I think it's relevant, so let's make sure my assumptions are > > > correct: > > > > > > - The ITAPI will be a "ULP" on OpenIB > > > > ITAPI is like uDAPL, so if uDAPL is a "ULP" then the answer is yes. > > The point is that for uDAPL you have the actual "app" running over > > uDAPL. So I guess it's a matter of terminology whether > uDAPL is a ULP > > or is it some sort of middleware with the app being the "ULP". > > > > Yeah, you're right the terminology is probably a little > goofy. The reason for the goofosity is that some of the "ulp" > really are protocols (ISER, IPoIB), and some are API (DAPL, > MPI). All use the same interface > to register with OpenIB. > > But that said, yes, ITAPI is like uDAPL. > > > > > > - The ITAPI will create the IRD/ORD headers in its > private data and > > > submit this as part of its connection establishment. > > > - The ITAPI consumer at the remote peer will use this data to > > > configure it's local QP before accepting the connection > > > > > > Over IB, the IRD/ORD private data will be prepended with > a "private > > > data header" that contains the source and destination IP > addresses, > > > source port, etc... The remote peer will not see this > data as part > > > of the private data, but rather will see it in the CMA > event in the > > > upcall. > > > > Over IB, the IRD/ORD data is already built in to the > standard CM stuff > > (i.e. the "responder resources" and "initiator depth" fields of REQ > > and REP). So no additional demands are made on private data > for IB in > > ITAPI for the IOH purpose. Of course the ITAPI app (like a > uDAPL app) > > can also use private data for app specific/ULP reasons. > > ok -- bad example. Sorry. This is a weird one. On iWARP, you > need the private data header to pass this stuff along and on > IB, you don't. What I was trying to say is that "whatever the > private data", on IB it will get a private data header > prepended and on iWARP, it won't. > > > > > > > > Over iWARP/MPA, there will be nothing else in the private data > > > except what was provided by the consumer (ITAPI in this > case). The > > > reason being that this extra information (IP addressing > info) is in > > > the protocol header proper. > > > > Just to restate for clarity, ITAPI for iWARP will use the first 16 > > bytes of MPA private date for the IOH (IRD/ORD header). The rest is > > usable for app/ULP reasons. > > Yessir. And in fact, the ITAPI CM will strip this stuff > before presenting it to the app. > > > > > > > I should point out that there was once a proposal of doing > a RDDP IETF > > draft which would have sub-divided the MPA private data into a > > "middleware" section and an "app" section. The idea was to be sure > > that the app/ULP and middleware (e.g. the IOH) uses of private data > > would not step on each other. I think this idea did not progress, > > mostly because the author (John Carrier, formerly of > Adaptec) changed > > jobs and was no longer working on iWARP stuff. > > > > While not directly proposed, this idea could have been > carried over to > > IB. Some of the ideas on this thread are already implicitly > doing this > > middleware (for IP addressing purpose) vs ULP/app split. > > > > I think we are grappling with a lot of these layering issues > now. We are also grappling with protocol vs. implementation issues. > > Keep it coming, because this is exactly the kind of feedback > I think we need. > > > -ted > > > ___ > openib-general mailing list > openib-general@openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] round 2 - proposal for socketbased connectionmodel
Sean wrote: >Kanevsky, Arkady wrote: >> What are you trying to achieve? >I'm trying to define a connection *service* for Infiniband that uses TCP/IP >addresses as its user interface. That service will have its own protocol, in >much the same way that SDP, SRP, etc. do today. Seems like we have two proposals, at a high level, I see no substantial benefit of one method over the other, they would both facilitate connecting two machines using IP address. However, one of the 2 proposals has code that is in the trunk that people can start to code to today, the other is a paper tiger that has no code. I suggest that rather than recommend an entirely new model (with no code), why not instead comment on the code that exists and send patches if it needs to be improved. Isn't that the way that open source is suppose to work ? my 2 cents, woody ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] RFC userspace CMA
Sean Hefty wrote: - The kernel CMA will expose a new call, rdma_init_qp_attr() to initialize QP attributes used to modify the state of the QP. The call will be similar to the infiniband CM routine. Use of this call is optional. The CMA will automatically transition QPs created by rdma_create_qp(). The changes are more involved than this. To handle the QP transitions in userspace, the kernel CMA needs to generate another event: CONNECT_RESPONSE. It will also need an additional API: rdma_establish(). (We can overload rdma_accept() in place of rdma_establish().) Basically, the 3-way handshake used by IB needs to be exposed. Use of either of these can be limited to those users that do not associate a QP with their rdma_cm_id. Alternatively, the uCMA kernel component can be integrated with the kernel CMA and make use of private interfaces. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] round 2 - proposal for socket based connectionmodel
Kanevsky, Arkady wrote: What are you trying to achieve? I'm trying to define a connection *service* for Infiniband that uses TCP/IP addresses as its user interface. That service will have its own protocol, in much the same way that SDP, SRP, etc. do today. I am trying to define an IB REQ protocol extension that support IP connection 5-tuple exchange between connection requestor and responder. Why? What need is there for a protocol extension to the IB CM? To me, this is similar to setting a bit in the CM REQ to indicate that the private data format looks like SDP's private data. The format of the _private_ data shouldn't be known to the CM; that's why it's private data. And define mapping between IP 5-tuple and IB entities. No mapping between IP <-> IB addresses was defined in the proposal. Defining this mapping is required to make this work. Right now, the mapping is the responsibility of every user. That way ULP which was written to TCP/IP, UDP/IP, CSTP/IP (and so on) can use RDMA transport without change. A ULP written to TCP/IP can use an RDMA transport without change. They use SDP. However, an application that wants to take advantage of QP semantics must change. (And if they want to take full advantage of RDMA, they'll likely need to be re-architected as well.) The goal in that case becomes to permit them to establish connections using TCP/IP addresses. To meet this goal, we need to define how to map IP address to and from IB addresses. That mapping is part of the protocol, and is missing from the proposal. And if the application isn't going to know that they're running on Infiniband, then the mapping must also include mapping to a destination service ID. To modify ULP to know that it runs on top of IB vs. iWARP vs. (any other RDMA transport) is bad idea. It is one thing to choose proper port to connect. Completely different to ask ULP to parse private data in transport specific way. The same protocol must support both user level ULPs and kernel level ULPs. Defining an interface that allows a ULP to use either iWarp, IB, or some other random RDMA transport is an implementation issue. However, it requires something that maps IP to IB addresses (including service IDs). To be more concrete, you've gone from having source and destination TCP/IP addresses to including them in a CM REQ. What translated the source and destination IP addresses into GIDs and a PKey? Who converted those into IB routing information? How was the destination of the CM REQ determined? What service ID was selected? - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] round 2 - proposal for socket based connection model
> -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Tom Tucker > Sent: Tuesday, October 25, 2005 2:56 PM > To: Kanevsky, Arkady > Cc: [EMAIL PROTECTED]; openib-general@openib.org > Subject: RE: [openib-general] round 2 - proposal for socket > based connection model > > Arkady: > > I may actually have a constructive comment about the protocol > (private data format). One thing I noticed is that *almost* > everything in the private data header is available in the > native iWARP protocol header except the ZB and SI bits. If > these bits become part of the canonical private data header, > then does that require an iWARP transport to use the header > too even though only two bits are useful? > > Sorry if this is a dumb question, > I'm not sure I followed why these were needed myself. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] round 2 - proposal for socket basedconnection model
Why does an application care whether the remote implementation supports ZB? Whether memory regions can be described with zero based rkeys or not doesn't matter on an end-to-end level. Its only a local issue. So ZB shouldn't be there IMO. - Original Message - From: "Tom Tucker" <[EMAIL PROTECTED]> To: "Kanevsky, Arkady" <[EMAIL PROTECTED]> Cc: <[EMAIL PROTECTED]>; Sent: Tuesday, October 25, 2005 4:55 PM Subject: RE: [openib-general] round 2 - proposal for socket basedconnection model Arkady: I may actually have a constructive comment about the protocol (private data format). One thing I noticed is that *almost* everything in the private data header is available in the native iWARP protocol header except the ZB and SI bits. If these bits become part of the canonical private data header, then does that require an iWARP transport to use the header too even though only two bits are useful? Sorry if this is a dumb question, Tom On Tue, 2005-10-25 at 16:40 -0500, Tom Tucker wrote: Arkady: I don't think anyone disagrees with your goals. Unfortunately additional requirements on the implementation were coupled with the specification of the private data format (protocol). This peripheral discussion derailed any attempt to discuss the protocol. Attempts to separate the protocol discussion from the implementation failed. And so here we are... On Tue, 2005-10-25 at 15:38 -0400, Kanevsky, Arkady wrote: > What are you trying to achieve? > > I am trying to define an IB REQ protocol extension that > support IP connection 5-tuple exchange between connection > requestor and responder. > And define mapping between IP 5-tuple and IB entities. > > That way ULP which was written to TCP/IP, UDP/IP, CSTP/IP (and so > on) > can use RDMA transport without change. > To modify ULP to know that it runs on top of IB vs. iWARP > vs. (any other RDMA transport) is bad idea. > It is one thing to choose proper port to connect. > Completely different to ask ULP to parse private data > in transport specific way. > > The same protocol must support both user level ULPs > and kernel level ULPs. > Arkady > > Arkady Kanevsky email: [EMAIL PROTECTED] > Network Appliance phone: 781-768-5395 > 375 Totten Pond Rd. Fax: 781-895-1195 > Waltham, MA 02451-2010 central phone: 781-768-5300 > > > > > -Original Message- > > From: Sean Hefty [mailto:[EMAIL PROTECTED] > > Sent: Tuesday, October 25, 2005 3:22 PM > > To: Kanevsky, Arkady > > Cc: Sean Hefty; openib-general@openib.org; [EMAIL PROTECTED] > > Subject: Re: [openib-general] round 2 - proposal for socket > > based connectionmodel > > > > > > Kanevsky, Arkady wrote: > > > Sean, > > > answers in-line. > > > Arkady > > > > At this point, I'm just going to disagree with this approach > > and move on with > > the current implementation of the CMA. What's needed is a > > service that provides > > IB connections using TCP/IP addressing. I don't believe this > > proposal meets > > this goal. > > > > To meet the requirement of connecting over IB using TCP/IP > > addressing, I believe > > that we need a service with a reserved service identifier or > > range of > > identifiers, a mechanism for mapping between IP and IB > > addresses, and a > > mechanism for reversing the mapping. > > > > I don't see where the proposal addresses the bulk of the work > > that's required, > > nor do I think that it will present an API to the user that > > does not expose IB > > related addressing (such as service IDs). > > > > - Sean > > > ___ > openib-general mailing list > openib-general@openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Automated userspace build error
Nishanth> Hrm, well, I'm testing the latest svn (3865), did the Nishanth> patch just get checked in? Yeah, I only noticed it and fixed it after your original email. I just meant that I had already checked it in before sending my reply. Sorry for the confusion... - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Automated userspace build error
On 25.10.2005 [15:09:42 -0700], Roland Dreier wrote: > >checking dynamic linker charactericonfigure: error: > >ibv_get_devices() not found. libibcm requires libibcm. > > The last error seeming like a circular dependency is just a typo, > fixed by the following patch (already checked in). Hrm, well, I'm testing the latest svn (3865), did the patch just get checked in? > As for why your build is failing, it seems that the libibcm configure > is not finding an install of libibverbs. Without knowing what your > setup is like, it's hard to speculate why that might be. True true; let me do some digging and figure out what's going on. Thanks, Nish ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: [PATCH] Minor mad_rmpp.c cleanup
Roland Dreier wrote: This changes alloc_response_msg() in mad_rmpp.c to return the struct it allocates directly (or an error code a la ERR_PTR), rather than returning a status and passing the struct back in a pointer param. This simplifies the code and gets rid of warnings like drivers/infiniband/core/mad_rmpp.c: In function nack_recv: drivers/infiniband/core/mad_rmpp.c:192: warning: msg may be used uninitialized in this function with newer versions of gcc. Thanks! Committed. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Automated userspace build error
> checking dynamic linker charactericonfigure: error: > ibv_get_devices() not found. libibcm requires libibcm. The last error seeming like a circular dependency is just a typo, fixed by the following patch (already checked in). As for why your build is failing, it seems that the libibcm configure is not finding an install of libibverbs. Without knowing what your setup is like, it's hard to speculate why that might be. - R. --- libibcm/configure.in(revision 3861) +++ libibcm/configure.in(working copy) @@ -26,7 +26,7 @@ dnl Checks for libraries if test "$disable_libcheck" != "yes" then AC_CHECK_LIB(ibverbs, ibv_get_devices, [], -AC_MSG_ERROR([ibv_get_devices() not found. libibcm requires libibcm.])) +AC_MSG_ERROR([ibv_get_devices() not found. libibcm requires libibverbs.])) AC_CHECK_LIB(ibat, ib_at_route_by_ip, [], AC_MSG_ERROR([ib_at_route_by_ip() not found. libibcm requires libat.])) fi ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Automated userspace build error
Hi all, I'm trying to add at least the build portion of userspace testing to my daily kernel build tests, but am running into the following failure in libibcm: checking dynamic linker charactericonfigure: error: ibv_get_devices() not found. libibcm requires libibcm. Not sure, why the log got overrun, but obviously the latter part is concerning. Is there a circular dependence somewhere? Thanks, Nish ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] round 2 - proposal for socket based connection model
Arkady: I may actually have a constructive comment about the protocol (private data format). One thing I noticed is that *almost* everything in the private data header is available in the native iWARP protocol header except the ZB and SI bits. If these bits become part of the canonical private data header, then does that require an iWARP transport to use the header too even though only two bits are useful? Sorry if this is a dumb question, Tom On Tue, 2005-10-25 at 16:40 -0500, Tom Tucker wrote: > Arkady: > > I don't think anyone disagrees with your goals. Unfortunately additional > requirements on the implementation were coupled with the specification > of the private data format (protocol). This peripheral discussion > derailed any attempt to discuss the protocol. > > Attempts to separate the protocol discussion from the implementation > failed. And so here we are... > > > On Tue, 2005-10-25 at 15:38 -0400, Kanevsky, Arkady wrote: > > What are you trying to achieve? > > > > I am trying to define an IB REQ protocol extension that > > support IP connection 5-tuple exchange between connection > > requestor and responder. > > And define mapping between IP 5-tuple and IB entities. > > > > That way ULP which was written to TCP/IP, UDP/IP, CSTP/IP (and so on) > > can use RDMA transport without change. > > To modify ULP to know that it runs on top of IB vs. iWARP > > vs. (any other RDMA transport) is bad idea. > > It is one thing to choose proper port to connect. > > Completely different to ask ULP to parse private data > > in transport specific way. > > > > The same protocol must support both user level ULPs > > and kernel level ULPs. > > Arkady > > > > Arkady Kanevsky email: [EMAIL PROTECTED] > > Network Appliance phone: 781-768-5395 > > 375 Totten Pond Rd. Fax: 781-895-1195 > > Waltham, MA 02451-2010 central phone: 781-768-5300 > > > > > > > > > -Original Message- > > > From: Sean Hefty [mailto:[EMAIL PROTECTED] > > > Sent: Tuesday, October 25, 2005 3:22 PM > > > To: Kanevsky, Arkady > > > Cc: Sean Hefty; openib-general@openib.org; [EMAIL PROTECTED] > > > Subject: Re: [openib-general] round 2 - proposal for socket > > > based connectionmodel > > > > > > > > > Kanevsky, Arkady wrote: > > > > Sean, > > > > answers in-line. > > > > Arkady > > > > > > At this point, I'm just going to disagree with this approach > > > and move on with > > > the current implementation of the CMA. What's needed is a > > > service that provides > > > IB connections using TCP/IP addressing. I don't believe this > > > proposal meets > > > this goal. > > > > > > To meet the requirement of connecting over IB using TCP/IP > > > addressing, I believe > > > that we need a service with a reserved service identifier or range of > > > identifiers, a mechanism for mapping between IP and IB > > > addresses, and a > > > mechanism for reversing the mapping. > > > > > > I don't see where the proposal addresses the bulk of the work > > > that's required, > > > nor do I think that it will present an API to the user that > > > does not expose IB > > > related addressing (such as service IDs). > > > > > > - Sean > > > > > ___ > > openib-general mailing list > > openib-general@openib.org > > http://openib.org/mailman/listinfo/openib-general > > > > To unsubscribe, please visit > > http://openib.org/mailman/listinfo/openib-general > ___ > openib-general mailing list > openib-general@openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [swg] Re: [openib-general] TCP/IP connection service over IB
On Tue, 2005-10-25 at 13:16 -0700, Ted H. Kim wrote: > Tom, > > Some comments inline ... > > > Tom Tucker wrote: > > I think it's relevant, so let's make sure my assumptions are correct: > > > > - The ITAPI will be a "ULP" on OpenIB > > ITAPI is like uDAPL, so if uDAPL is a "ULP" then the answer is yes. > The point is that for uDAPL you have the actual "app" running over > uDAPL. So I guess it's a matter of terminology whether uDAPL is > a ULP or is it some sort of middleware with the app being the "ULP". > Yeah, you're right the terminology is probably a little goofy. The reason for the goofosity is that some of the "ulp" really are protocols (ISER, IPoIB), and some are API (DAPL, MPI). All use the same interface to register with OpenIB. But that said, yes, ITAPI is like uDAPL. > > > - The ITAPI will create the IRD/ORD headers in its private data and > > submit this as part of its connection establishment. > > - The ITAPI consumer at the remote peer will use this data to configure > > it's local QP before accepting the connection > > > > Over IB, the IRD/ORD private data will be prepended with a "private data > > header" that contains the source and destination IP addresses, source > > port, etc... The remote peer will not see this data as part of the > > private data, but rather will see it in the CMA event in the upcall. > > Over IB, the IRD/ORD data is already built in to the standard CM > stuff (i.e. the "responder resources" and "initiator depth" fields of > REQ and REP). So no additional demands are made on private data for IB > in ITAPI for the IOH purpose. Of course the ITAPI app (like a uDAPL app) > can also use private data for app specific/ULP reasons. ok -- bad example. Sorry. This is a weird one. On iWARP, you need the private data header to pass this stuff along and on IB, you don't. What I was trying to say is that "whatever the private data", on IB it will get a private data header prepended and on iWARP, it won't. > > > > Over iWARP/MPA, there will be nothing else in the private data except > > what was provided by the consumer (ITAPI in this case). The reason being > > that this extra information (IP addressing info) is in the protocol > > header proper. > > Just to restate for clarity, ITAPI for iWARP will use the first 16 bytes of > MPA private date for the IOH (IRD/ORD header). The rest is usable for > app/ULP reasons. Yessir. And in fact, the ITAPI CM will strip this stuff before presenting it to the app. > > > I should point out that there was once a proposal of doing a RDDP IETF > draft which would have sub-divided the MPA private data into a > "middleware" section and an "app" section. The idea was to be sure that > the app/ULP and middleware (e.g. the IOH) uses of private data would not > step on each other. I think this idea did not progress, mostly because > the author (John Carrier, formerly of Adaptec) changed jobs and was no > longer working on iWARP stuff. > > While not directly proposed, this idea could have been carried over to IB. > Some of the ideas on this thread are already implicitly > doing this middleware (for IP addressing purpose) vs ULP/app split. > I think we are grappling with a lot of these layering issues now. We are also grappling with protocol vs. implementation issues. Keep it coming, because this is exactly the kind of feedback I think we need. > -ted > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] round 2 - proposal for socket based connection model
Arkady: I don't think anyone disagrees with your goals. Unfortunately additional requirements on the implementation were coupled with the specification of the private data format (protocol). This peripheral discussion derailed any attempt to discuss the protocol. Attempts to separate the protocol discussion from the implementation failed. And so here we are... On Tue, 2005-10-25 at 15:38 -0400, Kanevsky, Arkady wrote: > What are you trying to achieve? > > I am trying to define an IB REQ protocol extension that > support IP connection 5-tuple exchange between connection > requestor and responder. > And define mapping between IP 5-tuple and IB entities. > > That way ULP which was written to TCP/IP, UDP/IP, CSTP/IP (and so on) > can use RDMA transport without change. > To modify ULP to know that it runs on top of IB vs. iWARP > vs. (any other RDMA transport) is bad idea. > It is one thing to choose proper port to connect. > Completely different to ask ULP to parse private data > in transport specific way. > > The same protocol must support both user level ULPs > and kernel level ULPs. > Arkady > > Arkady Kanevsky email: [EMAIL PROTECTED] > Network Appliance phone: 781-768-5395 > 375 Totten Pond Rd. Fax: 781-895-1195 > Waltham, MA 02451-2010 central phone: 781-768-5300 > > > > > -Original Message- > > From: Sean Hefty [mailto:[EMAIL PROTECTED] > > Sent: Tuesday, October 25, 2005 3:22 PM > > To: Kanevsky, Arkady > > Cc: Sean Hefty; openib-general@openib.org; [EMAIL PROTECTED] > > Subject: Re: [openib-general] round 2 - proposal for socket > > based connectionmodel > > > > > > Kanevsky, Arkady wrote: > > > Sean, > > > answers in-line. > > > Arkady > > > > At this point, I'm just going to disagree with this approach > > and move on with > > the current implementation of the CMA. What's needed is a > > service that provides > > IB connections using TCP/IP addressing. I don't believe this > > proposal meets > > this goal. > > > > To meet the requirement of connecting over IB using TCP/IP > > addressing, I believe > > that we need a service with a reserved service identifier or range of > > identifiers, a mechanism for mapping between IP and IB > > addresses, and a > > mechanism for reversing the mapping. > > > > I don't see where the proposal addresses the bulk of the work > > that's required, > > nor do I think that it will present an API to the user that > > does not expose IB > > related addressing (such as service IDs). > > > > - Sean > > > ___ > openib-general mailing list > openib-general@openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] [PATCH] Minor mad_rmpp.c cleanup
This changes alloc_response_msg() in mad_rmpp.c to return the struct
it allocates directly (or an error code a la ERR_PTR), rather than
returning a status and passing the struct back in a pointer param.
This simplifies the code and gets rid of warnings like
drivers/infiniband/core/mad_rmpp.c: In function nack_recv:
drivers/infiniband/core/mad_rmpp.c:192: warning: msg may be used
uninitialized in this function
with newer versions of gcc.
Signed-off-by: Roland Dreier <[EMAIL PROTECTED]>
--- infiniband/core/mad_rmpp.c (revision 3865)
+++ infiniband/core/mad_rmpp.c (working copy)
@@ -151,28 +151,27 @@ static void ack_recv(struct mad_rmpp_rec
ib_free_send_mad(msg);
}
-static int alloc_response_msg(struct ib_mad_agent *agent,
- struct ib_mad_recv_wc *recv_wc,
- struct ib_mad_send_buf **msg)
+static struct ib_mad_send_buf *alloc_response_msg(struct ib_mad_agent *agent,
+ struct ib_mad_recv_wc
*recv_wc)
{
- struct ib_mad_send_buf *m;
+ struct ib_mad_send_buf *msg;
struct ib_ah *ah;
ah = ib_create_ah_from_wc(agent->qp->pd, recv_wc->wc,
recv_wc->recv_buf.grh, agent->port_num);
if (IS_ERR(ah))
- return PTR_ERR(ah);
+ return (void *) ah;
- m = ib_create_send_mad(agent, recv_wc->wc->src_qp,
- recv_wc->wc->pkey_index, 1,
- IB_MGMT_RMPP_HDR, IB_MGMT_RMPP_DATA, GFP_KERNEL);
- if (IS_ERR(m)) {
+ msg = ib_create_send_mad(agent, recv_wc->wc->src_qp,
+recv_wc->wc->pkey_index, 1,
+IB_MGMT_RMPP_HDR, IB_MGMT_RMPP_DATA,
+GFP_KERNEL);
+ if (IS_ERR(msg))
ib_destroy_ah(ah);
- return PTR_ERR(m);
- }
- m->ah = ah;
- *msg = m;
- return 0;
+ else
+ msg->ah = ah;
+
+ return msg;
}
void ib_rmpp_send_handler(struct ib_mad_send_wc *mad_send_wc)
@@ -191,8 +190,8 @@ static void nack_recv(struct ib_mad_agen
struct ib_rmpp_mad *rmpp_mad;
int ret;
- ret = alloc_response_msg(&agent->agent, recv_wc, &msg);
- if (ret)
+ msg = alloc_response_msg(&agent->agent, recv_wc);
+ if (IS_ERR(msg))
return;
rmpp_mad = msg->mad;
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] uDAPL open HCA problem
* On Oct,10 James Lentini<[EMAIL PROTECTED]> wrote : > > > On Fri, 21 Oct 2005, LEI CHAI wrote: > > > ips_by_gid: RET 0 at_rec 0x7fa8d380 -> id 4627 > > dapli_at_event_cb() > > ip_comp_handler: rec 0x7fa8d380 ->id 4627 id 4627 num -22 3c66c000 > > ip_comp_handler: resolution err -22 retry 1 > > ip_comp_handler: ips_by_gid 0 rec 0x7fa8d380->id 4628 > > dapli_at_event_cb() > > ip_comp_handler: rec 0x7fa8d380 ->id 4628 id 4628 num -22 0 > > ip_comp_handler: resolution err -22 retry 2 > > [rdma_udapl_priv.c:640] error(262144): Cannot open IA > > ip_comp_handler: ips_by_gid 0 rec 0x7fa8d380->id 4629 > > dapli_at_event_cb() > > ip_comp_handler: rec 0x7fa8d380 ->id 4629 id 4629 num -22 0 > > ip_comp_handler: resolution err -22 retry 3 > > ip_comp_handler: ips_by_gid 0 rec 0x7fa8d380->id 4630 > > dapli_at_event_cb() > > ip_comp_handler: rec 0x7fa8d380 ->id 4630 id 4630 num -22 0 > > ip_comp_handler: resolution err -22 retry 4 > > ip_comp_handler: ERR: at_rec 0x7fa8d380, id 4630 num -22 > > open_hca: ERR ib_at_ips_by_gid for mthca0 > > ib_at_ips_by_gid is failing again. Have you setup an IPoIB address? Sorry for the late reply :-( Yes, we have IPoIB setup. This happens intermittently. As suggested by Woody, we will also try out the scm version. Thanks, Sayantan. = ib0 Link encap:UNSPEC HWaddr 00-00-04-04-FE-80-00-00-00-00-00-00-00-00-00-00 inet addr:150.1.110.4 Bcast:150.1.255.255 Mask:255.255.0.0 inet6 addr: fe80::202:c902:40:315/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:2044 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:128 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) lsmod | grep ^ib [EMAIL PROTECTED]:~] lsmod | grep ^ib ib_ipoib 48008 0 ib_uat 14840 0 ib_at 25696 1 ib_uat ib_sa 17804 2 ib_ipoib,ib_at ib_ucm 22280 0 ib_cm 37744 1 ib_ucm ib_uverbs 35992 0 ib_umad18208 0 ib_mthca 122656 0 ib_mad 44072 4 ib_sa,ib_cm,ib_umad,ib_mthca ib_core56192 8 ib_ipoib,ib_sa,ib_ucm,ib_cm,ib_uverbs,ib_umad,ib_mthca,ib_mad Error: [EMAIL PROTECTED]:osu_benchmarks] ../bin/mpiexec -n 2 ./a.out DAPL: NOT Setting Loopback dapl_ib_init: ib_thread_init(7629) dapl_ia_open (ib0, 8, 0x7fe20a18, 0x5ae728) open_hca: mthca0 - 0x5c4390 ib_thread(7629,0x40200960): ENTER: pipe 8 at 4 open_hca: Found dev mthca0 0002c90200400314 open_hca: GID subnet fe80 id 0002c90200400315 ips_by_gid: RET 0 at_rec 0x7fe20780 -> id 37 dapli_at_event_cb() ip_comp_handler: rec 0x7fe20780 ->id 37 id 37 num -22 3c77c000 ip_comp_handler: resolution err -22 retry 1 ip_comp_handler: ips_by_gid 0 rec 0x7fe20780->id 38 dapli_at_event_cb() ip_comp_handler: rec 0x7fe20780 ->id 38 id 38 num -22 0 ip_comp_handler: resolution err -22 retry 2 ip_comp_handler: ips_by_gid 0 rec 0x7fe20780->id 39 dapli_at_event_cb() ip_comp_handler: rec 0x7fe20780 ->id 39 id 39 num -22 0 ip_comp_handler: resolution err -22 retry 3 ip_comp_handler: ips_by_gid 0 rec 0x7fe20780->id 40 dapli_at_event_cb() ip_comp_handler: rec 0x7fe20780 ->id 40 id 40 num -22 0 ip_comp_handler: resolution err -22 retry 4 ip_comp_handler: ERR: at_rec 0x7fe20780, id 40 num -22 open_hca: ERR ib_at_ips_by_gid for mthca0 dapls_ib_open_hca failed 4 dapl_ia_open () returns 0x4 DAPL: Stopped (dapl_fini) dapl_ib_release: ib_thread_destroy(7629) ib_thread_destroy: waiting for ib_thread ib_thread(7629) EXIT [rdma_udapl_priv.c:640] error(262144): Cannot open IA DAPL: NOT Setting Loopback dapl_ib_init: ib_thread_init(7630) dapl_ia_open (ib0, 8, 0x7fe55578, 0x5ae728) open_hca: mthca0 - 0x5c4390 ib_thread(7630,0x40200960): ENTER: pipe 8 at 4 open_hca: Found dev mthca0 0002c90200400314 open_hca: GID subnet fe80 id 0002c90200400315 ips_by_gid: RET 0 at_rec 0x7fe552e0 -> id 41 dapli_at_event_cb() ip_comp_handler: rec 0x7fe552e0 ->id 41 id 41 num -22 3c77c000 ip_comp_handler: resolution err -22 retry 1 ip_comp_handler: ips_by_gid 0 rec 0x7fe552e0->id 42 dapli_at_event_cb() ip_comp_handler: rec 0x7fe552e0 ->id 42 id 42 num -22 0 ip_comp_handler: resolution err -22 retry 2 ip_comp_handler: ips_by_gid 0 rec 0x7fe552e0->id 43 dapli_at_event_cb() ip_comp_handler: rec 0x7fe552e0 ->id 43 id 43 num -22 0 ip_comp_handler: resolution err -22 retry 3 ip_comp_handler: ips_by_gid 0 rec 0x7fe552e0->id 44 dapli_at_event_cb() ip_comp_handler: rec 0x7fe552e0 ->id 44 id 44 num -22 0 ip_comp_handler: resolution err -22 retry 4 ip_comp_handler: ERR: at_rec 0x7fe552e0, id 44 num -22 [rdma_udapl_priv.c:640] error(262144): Cannot open IA open_hca: ERR ib_at_ips_by_gid f
Re: [openib-general] [PATCH] new uDAPL openIB provider using socket CM
Hi Woody, * On Oct,2 Woodruff, Robert J<[EMAIL PROTECTED]> wrote : > Arlin wrote, > >James, > > >Here is a patch to add an optional openIB uDAPL provider that uses the > socket CM >for anyone having > >problems scaling out with the uCM/uAT version. To build the new > provider, simply >"make > >VERBS=openib_scm". This version does not require IPoIB, uCM, or uAT. > > >-arlin > > I have been using this DAPL provider with my testing of MPI and > it seems stable. It has been tested up to 128 nodes. > The uAT version on the other hand seems to still have problems > so I would be glad to see this version be put into the tree also. > > Sayantan, you may also want to try this socket based CM > uDAPL provide for your work. It seems more stable than the uAT version > in my testing thus far. Thanks for this information! We will try this out in our lab. Sincerely, Sayantan. -- http://www.cse.ohio-state.edu/~surs ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [swg] Re: [openib-general] TCP/IP connection service over IB
> -Original Message From Ted Kim - > > > I should point out that there was once a proposal of doing a > RDDP IETF draft which would have sub-divided the MPA private > data into a "middleware" section and an "app" section. The > idea was to be sure that the app/ULP and middleware (e.g. the > IOH) uses of private data would not step on each other. I > think this idea did not progress, mostly because the author > (John Carrier, formerly of Adaptec) changed jobs and was no > longer working on iWARP stuff. > > While not directly proposed, this idea could have been > carried over to IB. > Some of the ideas on this thread are already implicitly doing > this middleware (for IP addressing purpose) vs ULP/app split. > > > -ted > >From a spec-minimalist viewpoint there is no real benefit in having the wire protocol distinquish between payload provided by a non-privileged middleware library and a non-privileged application. It might be a really nice and convenient thing for it to do, but there is no real harm in having the middleware do the marking. The real issue here is that data that had been in a privileged header that had been implicitly validated by successful routing in both directions is being replaced by data that is traveling opaque directly from non-privileged code on the peer. On an IP network you cannot successfully establish a connnection where the remote IP address has no correlation with reality and is totally unreviewable by network administrators. The network administrator can also always block connections from certain addresses as a matter of policy and the application cannot override that. Typically the system administrator can as well. Passing IP addresses in non-privileged private data is an entirely different issue. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [swg] Re: [openib-general] TCP/IP connection service over IB
Tom, Some comments inline ... Tom Tucker wrote: I think it's relevant, so let's make sure my assumptions are correct: - The ITAPI will be a "ULP" on OpenIB ITAPI is like uDAPL, so if uDAPL is a "ULP" then the answer is yes. The point is that for uDAPL you have the actual "app" running over uDAPL. So I guess it's a matter of terminology whether uDAPL is a ULP or is it some sort of middleware with the app being the "ULP". - The ITAPI will create the IRD/ORD headers in its private data and submit this as part of its connection establishment. - The ITAPI consumer at the remote peer will use this data to configure it's local QP before accepting the connection Over IB, the IRD/ORD private data will be prepended with a "private data header" that contains the source and destination IP addresses, source port, etc... The remote peer will not see this data as part of the private data, but rather will see it in the CMA event in the upcall. Over IB, the IRD/ORD data is already built in to the standard CM stuff (i.e. the "responder resources" and "initiator depth" fields of REQ and REP). So no additional demands are made on private data for IB in ITAPI for the IOH purpose. Of course the ITAPI app (like a uDAPL app) can also use private data for app specific/ULP reasons. Over iWARP/MPA, there will be nothing else in the private data except what was provided by the consumer (ITAPI in this case). The reason being that this extra information (IP addressing info) is in the protocol header proper. Just to restate for clarity, ITAPI for iWARP will use the first 16 bytes of MPA private date for the IOH (IRD/ORD header). The rest is usable for app/ULP reasons. I should point out that there was once a proposal of doing a RDDP IETF draft which would have sub-divided the MPA private data into a "middleware" section and an "app" section. The idea was to be sure that the app/ULP and middleware (e.g. the IOH) uses of private data would not step on each other. I think this idea did not progress, mostly because the author (John Carrier, formerly of Adaptec) changed jobs and was no longer working on iWARP stuff. While not directly proposed, this idea could have been carried over to IB. Some of the ideas on this thread are already implicitly doing this middleware (for IP addressing purpose) vs ULP/app split. -ted ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] [PATCH] new uDAPL openIB provider using socket CM
Arlin wrote, >James, >Here is a patch to add an optional openIB uDAPL provider that uses the socket CM >for anyone having >problems scaling out with the uCM/uAT version. To build the new provider, simply >"make >VERBS=openib_scm". This version does not require IPoIB, uCM, or uAT. >-arlin I have been using this DAPL provider with my testing of MPI and it seems stable. It has been tested up to 128 nodes. The uAT version on the other hand seems to still have problems so I would be glad to see this version be put into the tree also. Sayantan, you may also want to try this socket based CM uDAPL provide for your work. It seems more stable than the uAT version in my testing thus far. woody ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] round 2 - proposal for socket based connectionmodel
What are you trying to achieve? I am trying to define an IB REQ protocol extension that support IP connection 5-tuple exchange between connection requestor and responder. And define mapping between IP 5-tuple and IB entities. That way ULP which was written to TCP/IP, UDP/IP, CSTP/IP (and so on) can use RDMA transport without change. To modify ULP to know that it runs on top of IB vs. iWARP vs. (any other RDMA transport) is bad idea. It is one thing to choose proper port to connect. Completely different to ask ULP to parse private data in transport specific way. The same protocol must support both user level ULPs and kernel level ULPs. Arkady Arkady Kanevsky email: [EMAIL PROTECTED] Network Appliance phone: 781-768-5395 375 Totten Pond Rd. Fax: 781-895-1195 Waltham, MA 02451-2010 central phone: 781-768-5300 > -Original Message- > From: Sean Hefty [mailto:[EMAIL PROTECTED] > Sent: Tuesday, October 25, 2005 3:22 PM > To: Kanevsky, Arkady > Cc: Sean Hefty; openib-general@openib.org; [EMAIL PROTECTED] > Subject: Re: [openib-general] round 2 - proposal for socket > based connectionmodel > > > Kanevsky, Arkady wrote: > > Sean, > > answers in-line. > > Arkady > > At this point, I'm just going to disagree with this approach > and move on with > the current implementation of the CMA. What's needed is a > service that provides > IB connections using TCP/IP addressing. I don't believe this > proposal meets > this goal. > > To meet the requirement of connecting over IB using TCP/IP > addressing, I believe > that we need a service with a reserved service identifier or range of > identifiers, a mechanism for mapping between IP and IB > addresses, and a > mechanism for reversing the mapping. > > I don't see where the proposal addresses the bulk of the work > that's required, > nor do I think that it will present an API to the user that > does not expose IB > related addressing (such as service IDs). > > - Sean > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] round 2 - proposal for socket based connectionmodel
Kanevsky, Arkady wrote: Sean, answers in-line. Arkady At this point, I'm just going to disagree with this approach and move on with the current implementation of the CMA. What's needed is a service that provides IB connections using TCP/IP addressing. I don't believe this proposal meets this goal. To meet the requirement of connecting over IB using TCP/IP addressing, I believe that we need a service with a reserved service identifier or range of identifiers, a mechanism for mapping between IP and IB addresses, and a mechanism for reversing the mapping. I don't see where the proposal addresses the bulk of the work that's required, nor do I think that it will present an API to the user that does not expose IB related addressing (such as service IDs). - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] round 2 - proposal for socket based connectionmodel
Title: Message Sean, answers in-line. Arkady Arkady Kanevsky email: [EMAIL PROTECTED] Network Appliance phone: 781-768-5395 375 Totten Pond Rd. Fax: 781-895-1195 Waltham, MA 02451-2010 central phone: 781-768-5300 -Original Message-From: Sean Hefty [mailto:[EMAIL PROTECTED] Sent: Tuesday, October 25, 2005 1:05 PMTo: Kanevsky, Arkady; openib-general@openib.org; [EMAIL PROTECTED]Subject: RE: [openib-general] round 2 - proposal for socket based connectionmodel Dear OpenIB, SWG and DAT members, enclosed is teh second version of the proposal. There are really 2 proposals that are related. The first one is encoding IP 5-tuple into REQ private data with small additional info for versioning and IB capabilities. The second is just a couple of ideas, not a real proposal, on maping of IP ports to IB Service IDs. Comments on the private data format: Combine major/minor version into a single field. There’s no advantage to have two fields, so keep it simple.[AK] agree Remove ZB and SI bits. These are unrelated to socket addressing.[AK] That is true these are unrelated to socket addressing. But since several ULPs over IB need this info it can be added to the generic CM extensions for IB. I will rename the proposal to deal with it. I prefer a single private data formating proposal rather then several layered on top of each other. If IBTA think this is generic enough and want to redefine some reserved fields for it - good. This is captured in discussion slides. If the destination port number is encoded in a service ID, then it can be removed from the private data.[AK] This is dependent on how port mapping to Service ID is done. But if SDP will incorporate this into hello-wold protocol this may still be needed. With 64-bytes Consumer private data requirement relaxed saving 2 bytes will not make much difference. The transport protocol number could also be encoded in the service ID and removed from the private data. Actually, the version, IP version, and source port could all be encoded in the service ID, limiting the private data to just 32 bytes of IP addresses.[AK] Encoding IP version into Service ID sounds strange. Service ID is a pprt equivalent. Sure it is much larger than IP ports but why does CM extensions should encode more than port into it? Even with this Consumer private data is still only 60 bytes (not old 64-bytes requirement). - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] RE: [dat-discussions] round 2 - proposal for socket based connection model
> -Original Message- > From: Sean Hefty [mailto:[EMAIL PROTECTED] > Sent: Tuesday, October 25, 2005 11:21 AM > To: Caitlin Bestler > Cc: Kanevsky, Arkady; openib-general@openib.org; [EMAIL PROTECTED] > Subject: Re: [openib-general] RE: [dat-discussions] round 2 - > proposal for socket based connection model > > Caitlin Bestler wrote: > > What you are proposing is an API that purports to have the > semantics > > of TCP/IP connection establishment that can be implemented under > > non-IP transports such as InfiniBand. > > > > However, as proposed the mapping of this API to InfiniBand > does *not* > > implement the semantics of TCP/IP connection establishment > in that the > > remote address presented to the listener has been subject to *no* > > authentication. > > > > That is a change in the API that has an impact on the > application. It > > is creating a requiremet for the application to validate the remote > > identity greater than it would face for TCP/IP connection > > establishment. > > What API proposal are you referring to? > > If you're referring to the CMA, there's only a kernel > (privileged) component in existence. It sets the IP address > in the private data. What is the issue that you're referring to? > > - Sean > The remote peer will be able to use an existing CM to send a forged IP address. There is nothing the receiving CMA, or consumer (no matter how privileged) can do to detect this without the cooperation of privileged components on the remote end. They cannot know that the cooperation they are receiving from the remote end is from a privileged entity unless it comes from a privileged QP and is not part of the existing pass-through data. And playing the "I'm only in kernel" ostrich game doesn't help. Any connection establishment protocol has to make sense from both user and kernel modes and needs to be symmetric. But it needs to include clear controls on who is trusted to provide what information, and what information MUST come from a privileged entity. A source IP address that can come from a non-privileged entity is NOT consistent with IP network connection establishment semantics. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] RE: [dat-discussions] round 2 - proposal for socket based connection model
> -Original Message- > From: Tom Tucker [mailto:[EMAIL PROTECTED] > Sent: Tuesday, October 25, 2005 11:13 AM > To: Caitlin Bestler > Cc: Sean Hefty; Kanevsky, Arkady; [EMAIL PROTECTED]; DAT > Collaborative; openib-general@openib.org > Subject: RE: [openib-general] RE: [dat-discussions] round 2 - > proposal for socket based connection model > > On Tue, 2005-10-25 at 10:51 -0700, Caitlin Bestler wrote: > > > > > > > > > > I believe that the assurances you are talking about are > peculiar to > > > an implementation, not to the network. > > > > > > > I disagree. Anytime you send an IP datagram on an IP > network you are > > expected to provide an authentic source address. Any intermediate > > network device MAY enforce that rule and drop packets with invalid > > source addresses. > > > > I don't see anything in the protocol specs (RFC 791, RFC 793, > ...) that talks about this, so we just have to agree to disagree. :-) > Joe Touch's current draft on spoofing prevention covers this well in Section 3.2 (draft-ietf-tcpm-tcp-antispoof-02). IP networks can prevent address spoofing at the network layer using IPSec or by having border routers/filters validate the source address of incoming packets against routing rules. The latter is covered in RFC 2827 "Ingress Filtering for Multihomed Networks" and RFC 2267 "Network Ingress Ingress Filtering: Defeating Denial of Service Attacks which employ IP Address Spoofing" And more generally, in a TCP network a non-privileged client is NOT allowed to bind to any address and is NOT allowed to send raw Ethernet to bypass the host stack. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] RE: [dat-discussions] round 2 - proposal for socket based connection model
Caitlin Bestler wrote: What you are proposing is an API that purports to have the semantics of TCP/IP connection establishment that can be implemented under non-IP transports such as InfiniBand. However, as proposed the mapping of this API to InfiniBand does *not* implement the semantics of TCP/IP connection establishment in that the remote address presented to the listener has been subject to *no* authentication. That is a change in the API that has an impact on the application. It is creating a requiremet for the application to validate the remote identity greater than it would face for TCP/IP connection establishment. What API proposal are you referring to? If you're referring to the CMA, there's only a kernel (privileged) component in existence. It sets the IP address in the private data. What is the issue that you're referring to? - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] RE: [dat-discussions] round 2 - proposal for socket based connection model
> -Original Message- > From: Sean Hefty [mailto:[EMAIL PROTECTED] > Sent: Tuesday, October 25, 2005 11:10 AM > To: Kanevsky, Arkady > Cc: Caitlin Bestler; openib-general@openib.org; [EMAIL PROTECTED] > Subject: Re: [openib-general] RE: [dat-discussions] round 2 - > proposal for socket based connection model > > Kanevsky, Arkady wrote: > > It is APIs not ULPs that are concern. > > Yes - and an application that wants to use IP addressing > instead of IB addressing should use a different API than that > of the IB CM. Trying to define the IB CM to use anybody's > favorite transport/network address is the wrong solution to > the problem. That is a service level issue best left to the > service that's trying to perform the mapping. > What you are proposing is an API that purports to have the semantics of TCP/IP connection establishment that can be implemented under non-IP transports such as InfiniBand. However, as proposed the mapping of this API to InfiniBand does *not* implement the semantics of TCP/IP connection establishment in that the remote address presented to the listener has been subject to *no* authentication. That is a change in the API that has an impact on the application. It is creating a requiremet for the application to validate the remote identity greater than it would face for TCP/IP connection establishment. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] RE: [dat-discussions] round 2 - proposal for socket based connection model
On Tue, 2005-10-25 at 10:51 -0700, Caitlin Bestler wrote: > > > > > > I believe that the assurances you are talking about are > > peculiar to an implementation, not to the network. > > > > I disagree. Anytime you send an IP datagram on an IP network > you are expected to provide an authentic source address. Any > intermediate network device MAY enforce that rule and drop > packets with invalid source addresses. > I don't see anything in the protocol specs (RFC 791, RFC 793, ...) that talks about this, so we just have to agree to disagree. :-) > IP Addresses stored in private data, by contrast, are guaranteed > to pass all middleboxes unmolested without review of validation. > This is not a spoofer taking advantage of a lazy network admin, > this is a spoofer being given a "get out of jail free" card that > says the network admin is not even allowed to do spot checks. > > > The CMA is what is preparing the private data header, not the > > app. WRT a IB CM app, it could very easily pretend to be a > > "CMA App" and build it's own private data that spoofed the > > address. How would the local CM know that it is supposed to > > verify this? Where is the service id/private data format > > mapping database? > > > > In short, I think we are mixing many different things together here. > > > > > > For the very same reasons that a userspace consumer is not allowed > to pretend to be the CM itself, it should not be allowed to just > make up Source IP Addresses. If it's going to lie it needs to be > a privileged liar. > > Preserving the existing CM infrastructure is fine, but not if it > forces us to take something that should be authenticated by privileged > software and simply trust that userspace code will fill it in correctly. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] RE: [dat-discussions] round 2 - proposal for socket based connection model
Kanevsky, Arkady wrote: It is APIs not ULPs that are concern. Yes - and an application that wants to use IP addressing instead of IB addressing should use a different API than that of the IB CM. Trying to define the IB CM to use anybody's favorite transport/network address is the wrong solution to the problem. That is a service level issue best left to the service that's trying to perform the mapping. Each ULP can define its own protocol. Each ULP does define its own protocol - connection or otherwise. SDP cannot talk to IPoIB which cannot talk to SRP. If ULP uses API, it does the parsing. APIs are merely an interface. What needs to be defined is a service that can do the parsing. For Linux, the API to that service should be defined using the standard open source method. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Support for UC connections using the CM API?
OK, I checked in the mthca QP transition table fix. By the way, I think we probably need to consolidate the checking of required/optional QP modify attributes so that mthca, ipath and ehca don't each have their own copy of the code (and their own bugs). - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] RE: [dat-discussions] round 2 - proposal for socket based connection model
It is APIs not ULPs that are concern. Each ULP can define its own protocol. But APIs can not. But defining a protocol for each ULP is also bad. This proposal defines it for all ULPs. If ULP uses API, it does the parsing. If ULP uses verbs it can do the parsing and encoding itself. But in the later case it will have to have a different ULP CM for each transport. Bad idea. Arkady Arkady Kanevsky email: [EMAIL PROTECTED] Network Appliance phone: 781-768-5395 375 Totten Pond Rd. Fax: 781-895-1195 Waltham, MA 02451-2010 central phone: 781-768-5300 > -Original Message- > From: Sean Hefty [mailto:[EMAIL PROTECTED] > Sent: Tuesday, October 25, 2005 1:52 PM > To: Kanevsky, Arkady > Cc: Caitlin Bestler; openib-general@openib.org; [EMAIL PROTECTED] > Subject: Re: [openib-general] RE: [dat-discussions] round 2 - > proposal for socket based connection model > > > Kanevsky, Arkady wrote: > > Sean, > > The reason IBTA is interested to address IP address issue > > is because of multiple UPLs and APIs want to support > > socket based connection model. Sure each one of them > > can define its own protocol (for private data). > > But this will not ensure interoperability. > > There's no interoperability between different ULPs anyway. > Each does define its > own protocol. Trying to standardize part of the CM REQ > private data doesn't > help in this regard. > > - Sean > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] RE: [dat-discussions] round 2 - proposal for socket based connection model
> > I believe that the assurances you are talking about are > peculiar to an implementation, not to the network. > I disagree. Anytime you send an IP datagram on an IP network you are expected to provide an authentic source address. Any intermediate network device MAY enforce that rule and drop packets with invalid source addresses. IP Addresses stored in private data, by contrast, are guaranteed to pass all middleboxes unmolested without review of validation. This is not a spoofer taking advantage of a lazy network admin, this is a spoofer being given a "get out of jail free" card that says the network admin is not even allowed to do spot checks. > The CMA is what is preparing the private data header, not the > app. WRT a IB CM app, it could very easily pretend to be a > "CMA App" and build it's own private data that spoofed the > address. How would the local CM know that it is supposed to > verify this? Where is the service id/private data format > mapping database? > > In short, I think we are mixing many different things together here. > > For the very same reasons that a userspace consumer is not allowed to pretend to be the CM itself, it should not be allowed to just make up Source IP Addresses. If it's going to lie it needs to be a privileged liar. Preserving the existing CM infrastructure is fine, but not if it forces us to take something that should be authenticated by privileged software and simply trust that userspace code will fill it in correctly. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] RE: [dat-discussions] round 2 - proposal for socket based connection model
Kanevsky, Arkady wrote: Sean, The reason IBTA is interested to address IP address issue is because of multiple UPLs and APIs want to support socket based connection model. Sure each one of them can define its own protocol (for private data). But this will not ensure interoperability. There's no interoperability between different ULPs anyway. Each does define its own protocol. Trying to standardize part of the CM REQ private data doesn't help in this regard. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] RE: [dat-discussions] round 2 - proposal for socket based connection model
Sean, The reason IBTA is interested to address IP address issue is because of multiple UPLs and APIs want to support socket based connection model. Sure each one of them can define its own protocol (for private data). But this will not ensure interoperability. Arkady Arkady Kanevsky email: [EMAIL PROTECTED] Network Appliance phone: 781-768-5395 375 Totten Pond Rd. Fax: 781-895-1195 Waltham, MA 02451-2010 central phone: 781-768-5300 > -Original Message- > From: Sean Hefty [mailto:[EMAIL PROTECTED] > Sent: Tuesday, October 25, 2005 1:34 PM > To: Kanevsky, Arkady > Cc: Caitlin Bestler; openib-general@openib.org; [EMAIL PROTECTED] > Subject: Re: [openib-general] RE: [dat-discussions] round 2 - > proposal for socket based connection model > > > Kanevsky, Arkady wrote: > > Think of a single API that supports iWARP and IB (transport > > independent API). > > The CMA implements this today and did not require any changes > to the IB CM. > > > To a connection listener it provides the IP 5-tuple + private data. > > For IB it means that CM parses REQ and extracts IP 5-tuple > as separate > > fields from private data. > > Why push this down into the CM? The CM should operate on IB > addresses, not IP > addresses. The mapping of IP addresses to IB addresses is > done at a higher level. > > > Listener does not parse the private data encoding of the proposal. > > The listener is the one who cares about the IP addressing. > > - Sean > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] Fix for MAD layer DMA mappings
BTW, I just tried this on my PPC 4xx system, and the MAD layer works fine now. The port makes it to active and IPoIB works as well. Thanks, Roland ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] RE: [dat-discussions] round 2 - proposal for socket based connection model
On Tue, 2005-10-25 at 10:21 -0700, Caitlin Bestler wrote: > > > -Original Message- > > From: Sean Hefty [mailto:[EMAIL PROTECTED] > > Sent: Tuesday, October 25, 2005 10:08 AM > > To: Kanevsky, Arkady > > Cc: Caitlin Bestler; [EMAIL PROTECTED]; > > openib-general@openib.org; [EMAIL PROTECTED] > > Subject: Re: [openib-general] RE: [dat-discussions] round 2 - > > proposal for socket based connection model > > > > Kanevsky, Arkady wrote: > > > Correct. > > > But this does bring the question how responder CM knows > > that it need > > > to parse the private data. I suspect this will be done via > > new version > > > of CM. > > > But a suage of some of the CM REQ reserved fields are also possible. > > > Anotherwords the current CM version assumes that CM only > > supports one > > > version and there is no need to support more than 1 version. > > > > The responder knows how to parse the private data based on > > the service ID that they're listening on. This is how it's > > done today, and how it will still need to be done. What is > > the motivation to change it? > > > > What data is beyond the addressing? How does the responder > > know how to interpret that? > > > > I agree, the listener is responsible for knowing what format > the Private Data is supposed to be in. Therefore it knows in > advance what portions of it are relevant to the CM (the IP > address information and/or the ITAPI IRD/ORD pre-header). > So the listen request can specify the required CM parsing. > > But that does not prevent a non-privileged application from > forging the IP address information. These connection requests > are being presented to daemons as though they had the same > degree of authentication as address headers in an IP network > could have. The latter can be quite high when switches and > routers validate source addresses versus arriving ports. I believe that the assurances you are talking about are peculiar to an implementation, not to the network. The CMA is what is preparing the private data header, not the app. WRT a IB CM app, it could very easily pretend to be a "CMA App" and build it's own private data that spoofed the address. How would the local CM know that it is supposed to verify this? Where is the service id/private data format mapping database? In short, I think we are mixing many different things together here. > ___ > openib-general mailing list > openib-general@openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] RE: [dat-discussions] round 2 - proposal for socket based connection model
Kanevsky, Arkady wrote: Think of a single API that supports iWARP and IB (transport independent API). The CMA implements this today and did not require any changes to the IB CM. To a connection listener it provides the IP 5-tuple + private data. For IB it means that CM parses REQ and extracts IP 5-tuple as separate fields from private data. Why push this down into the CM? The CM should operate on IB addresses, not IP addresses. The mapping of IP addresses to IB addresses is done at a higher level. Listener does not parse the private data encoding of the proposal. The listener is the one who cares about the IP addressing. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] RE: [dat-discussions] round 2 - proposal for socket based connection model
> -Original Message- > From: Tom Tucker [mailto:[EMAIL PROTECTED] > Sent: Tuesday, October 25, 2005 10:24 AM > To: Caitlin Bestler > Cc: DAT Collaborative; openib-general@openib.org; [EMAIL PROTECTED] > Subject: Re: [openib-general] RE: [dat-discussions] round 2 - > proposal for socket based connection model > > What does this have to do with the protocol? > It's a whopping big security vulnerability. The application is left with an expectation that the address is more validated than it is. Admittedly even on an IP network it is not perfectly authenticated, but with this protocol the remote address information is far less authenticated and trivially spoofed. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] RE: [dat-discussions] round 2 - proposal for socket based connection model
Caitlin Bestler wrote: Is that because you do not agree that there is a problem? Or is it that you think the gap betweeen this and existing IP connection semantics is small enough that it is better to cover it with a disclosure than by changing the CM protocol? I would define the problem as: applications want to connect over IB using IP addressing. Defining the CM REQ private data solves is only a small part of the solution (reverse lookup). On an IP network the remote IP Address/port was vouched for by the remote kernel at the minimum, and MAY have been authenticated by each routing element along the way. Private data supplied through the existing CM protocol has neither of those safeguards. I think that security is a separate issue outside of this. I have no idea what OS is running on a remote system, let alone how it may have verified an address. That said, the kernel CMA would set this data based on information that it collects. But only users of the CMA would have this additional protection. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] RE: [dat-discussions] round 2 - proposal for socket based connection model
Think of a single API that supports iWARP and IB (transport independent API). To a connection listener it provides the IP 5-tuple + private data. For IB it means that CM parses REQ and extracts IP 5-tuple as separate fields from private data. Listener does not parse the private data encoding of the proposal. So CM need to know if it need to encode IP 5-tuple on requestor side and if need to parse on responder side. Arkady Arkady Kanevsky email: [EMAIL PROTECTED] Network Appliance phone: 781-768-5395 375 Totten Pond Rd. Fax: 781-895-1195 Waltham, MA 02451-2010 central phone: 781-768-5300 > -Original Message- > From: Sean Hefty [mailto:[EMAIL PROTECTED] > Sent: Tuesday, October 25, 2005 1:08 PM > To: Kanevsky, Arkady > Cc: Caitlin Bestler; [EMAIL PROTECTED]; > openib-general@openib.org; [EMAIL PROTECTED] > Subject: Re: [openib-general] RE: [dat-discussions] round 2 - > proposal for socket based connection model > > > Kanevsky, Arkady wrote: > > Correct. > > But this does bring the question how responder CM knows > that it need > > to parse the private data. I suspect this will be done via > new version > > of CM. But a suage of some of the CM REQ reserved fields are also > > possible. Anotherwords the current CM version assumes that CM only > > supports one version and there is no need to support more than 1 > > version. > > The responder knows how to parse the private data based on > the service ID that > they're listening on. This is how it's done today, and how > it will still need > to be done. What is the motivation to change it? > > What data is beyond the addressing? How does the responder > know how to > interpret that? > > - Sean > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] RE: [dat-discussions] round 2 - proposal for socket based connection model
What does this have to do with the protocol? On Tue, 2005-10-25 at 09:35 -0700, Caitlin Bestler wrote: > On an IP network, a non-privileged user is generally not capable of > forging > a source IP address and is typically prevented from using certain > source ports. > > I would propose that the CM [MAY|SHOULD|MUST] enforce that a non- > privileged > user can only use a Source IP Address and Port that they would have > been > able to use following the normal stack path (or what it would have > been in the > case that there is no conventional IP stack associated with this > path). > > So if IPoIB is installed, you would not be able to use any address > that > you would have been blocked from using over IPoIB. Or at least you > would not be guaranteed that you could. > > I think that MUST is the correct level of enforcement, but it needs to > be > clear that the CM and OS *MAY* do this checking and that a userspace > IB application cannot use the IB stack to perform IP spoofing. > > > __ > From: [EMAIL PROTECTED] [mailto:dat- > [EMAIL PROTECTED] On Behalf Of Kanevsky, Arkady > Sent: Tuesday, October 25, 2005 9:00 AM > To: openib-general@openib.org; dat- > [EMAIL PROTECTED]; [EMAIL PROTECTED] > Subject: [dat-discussions] round 2 - proposal for socket based > connection model > > > Dear OpenIB, SWG and DAT members, > enclosed is teh second version of the proposal. > There are really 2 proposals that are related. > The first one is encoding IP 5-tuple into REQ private data > with small additional info for versioning and IB capabilities. > The second is just a couple of ideas, not a real proposal, > on maping of IP ports > to IB Service IDs. > > Thanks everybody for tons of feedback and deep discussions. > I appologize if I had missed something. > > Happy reading, > Arkady > > > Arkady Kanevsky email: [EMAIL PROTECTED] > > Network Appliance phone: 781-768-5395 > > 375 Totten Pond Rd. Fax: 781-895-1195 > > Waltham, MA 02451-2010 central phone: 781-768-5300 > > > > > > > > __ > YAHOO! GROUPS LINKS > > 1. Visit your group "dat-discussions" on the web. > > 2. To unsubscribe from this group, send an email to: > [EMAIL PROTECTED] > > 3. Your use of Yahoo! Groups is subject to the Yahoo! > Terms of Service. > > > __ > > ___ > openib-general mailing list > openib-general@openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] RE: [dat-discussions] round 2 - proposal for socket based connection model
> -Original Message- > From: Sean Hefty [mailto:[EMAIL PROTECTED] > Sent: Tuesday, October 25, 2005 10:08 AM > To: Kanevsky, Arkady > Cc: Caitlin Bestler; [EMAIL PROTECTED]; > openib-general@openib.org; [EMAIL PROTECTED] > Subject: Re: [openib-general] RE: [dat-discussions] round 2 - > proposal for socket based connection model > > Kanevsky, Arkady wrote: > > Correct. > > But this does bring the question how responder CM knows > that it need > > to parse the private data. I suspect this will be done via > new version > > of CM. > > But a suage of some of the CM REQ reserved fields are also possible. > > Anotherwords the current CM version assumes that CM only > supports one > > version and there is no need to support more than 1 version. > > The responder knows how to parse the private data based on > the service ID that they're listening on. This is how it's > done today, and how it will still need to be done. What is > the motivation to change it? > > What data is beyond the addressing? How does the responder > know how to interpret that? > I agree, the listener is responsible for knowing what format the Private Data is supposed to be in. Therefore it knows in advance what portions of it are relevant to the CM (the IP address information and/or the ITAPI IRD/ORD pre-header). So the listen request can specify the required CM parsing. But that does not prevent a non-privileged application from forging the IP address information. These connection requests are being presented to daemons as though they had the same degree of authentication as address headers in an IP network could have. The latter can be quite high when switches and routers validate source addresses versus arriving ports. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] RE: [dat-discussions] round 2 - proposal for socket based connection model
Kanevsky, Arkady wrote: Correct. But this does bring the question how responder CM knows that it need to parse the private data. I suspect this will be done via new version of CM. But a suage of some of the CM REQ reserved fields are also possible. Anotherwords the current CM version assumes that CM only supports one version and there is no need to support more than 1 version. The responder knows how to parse the private data based on the service ID that they're listening on. This is how it's done today, and how it will still need to be done. What is the motivation to change it? What data is beyond the addressing? How does the responder know how to interpret that? - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] round 2 - proposal for socket based connectionmodel
Title: Message Dear OpenIB, SWG and DAT members, enclosed is teh second version of the proposal. There are really 2 proposals that are related. The first one is encoding IP 5-tuple into REQ private data with small additional info for versioning and IB capabilities. The second is just a couple of ideas, not a real proposal, on maping of IP ports to IB Service IDs. Comments on the private data format: Combine major/minor version into a single field. There’s no advantage to have two fields, so keep it simple. Remove ZB and SI bits. These are unrelated to socket addressing. If the destination port number is encoded in a service ID, then it can be removed from the private data. The transport protocol number could also be encoded in the service ID and removed from the private data. Actually, the version, IP version, and source port could all be encoded in the service ID, limiting the private data to just 32 bytes of IP addresses. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] RE: [dat-discussions] round 2 - proposal for socket based connection model
> -Original Message- > From: Sean Hefty [mailto:[EMAIL PROTECTED] > Sent: Tuesday, October 25, 2005 9:56 AM > To: Caitlin Bestler > Cc: Kanevsky, Arkady; [EMAIL PROTECTED]; > openib-general@openib.org; [EMAIL PROTECTED] > Subject: Re: [openib-general] RE: [dat-discussions] round 2 - > proposal for socket based connection model > > Caitlin Bestler wrote: > > I believe it requires a CM protocol version change, or a > "IP Address > > Header present" bit. > > > > Basically, userspace consumers can supply *any* 72 bytes of private > > data currently. > > To maintain backwards compatability you need an authenticator that > > says "this IP header data vouched for by privileged > components on this > > end", and that authenticator cannot be within the private data. > > I believe that the solution is keep the CM protocol as is. > The CM private data should be completely controlled by the > service. The IB CM does not care if an IP address is in the > private data or not. > > My reading of the proposal is that it defines a private data > format that a particular service may or may not use. > Is that because you do not agree that there is a problem? Or is it that you think the gap betweeen this and existing IP connection semantics is small enough that it is better to cover it with a disclosure than by changing the CM protocol? How would advise an application that uses the remote address to check an Access Control List (such as an NFS daemon) to treat this data? On an IP network the remote IP Address/port was vouched for by the remote kernel at the minimum, and MAY have been authenticated by each routing element along the way. Private data supplied through the existing CM protocol has neither of those safeguards. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] RE: [dat-discussions] round 2 - proposal for socket based connection model
Correct. But this does bring the question how responder CM knows that it need to parse the private data. I suspect this will be done via new version of CM. But a suage of some of the CM REQ reserved fields are also possible. Anotherwords the current CM version assumes that CM only supports one version and there is no need to support more than 1 version. This proposal may change this assumption. Arkady Arkady Kanevsky email: [EMAIL PROTECTED] Network Appliance phone: 781-768-5395 375 Totten Pond Rd. Fax: 781-895-1195 Waltham, MA 02451-2010 central phone: 781-768-5300 > -Original Message- > From: Sean Hefty [mailto:[EMAIL PROTECTED] > Sent: Tuesday, October 25, 2005 12:56 PM > To: Caitlin Bestler > Cc: Kanevsky, Arkady; [EMAIL PROTECTED]; > openib-general@openib.org; [EMAIL PROTECTED] > Subject: Re: [openib-general] RE: [dat-discussions] round 2 - > proposal for socket based connection model > > > Caitlin Bestler wrote: > > I believe it requires a CM protocol version change, or a "IP Address > > Header present" bit. > > > > Basically, userspace consumers can supply *any* 72 bytes of private > > data > > currently. > > To maintain backwards compatability you need an > authenticator that says > > "this IP > > header data vouched for by privileged components on this > end", and that > > authenticator > > cannot be within the private data. > > I believe that the solution is keep the CM protocol as is. > The CM private data > should be completely controlled by the service. The IB CM > does not care if an > IP address is in the private data or not. > > My reading of the proposal is that it defines a private data > format that a > particular service may or may not use. > > - Sean > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] RE: [dat-discussions] round 2 - proposal for socket based connection model
Caitlin Bestler wrote: I believe it requires a CM protocol version change, or a "IP Address Header present" bit. Basically, userspace consumers can supply *any* 72 bytes of private data currently. To maintain backwards compatability you need an authenticator that says "this IP header data vouched for by privileged components on this end", and that authenticator cannot be within the private data. I believe that the solution is keep the CM protocol as is. The CM private data should be completely controlled by the service. The IB CM does not care if an IP address is in the private data or not. My reading of the proposal is that it defines a private data format that a particular service may or may not use. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] RE: [dat-discussions] round 2 - proposal for socket based connection model
Title: Message I believe it requires a CM protocol version change, or a "IP Address Header present" bit. Basically, userspace consumers can supply *any* 72 bytes of private data currently. To maintain backwards compatability you need an authenticator that says "this IP header data vouched for by privileged components on this end", and that authenticator cannot be within the private data. The equivalent guarantee is provided on IP networks by the fact that raw sockets are not accessible by non-privileged applications. From: Kanevsky, Arkady [mailto:[EMAIL PROTECTED] Sent: Tuesday, October 25, 2005 9:39 AMTo: Caitlin Bestler; [EMAIL PROTECTED]; openib-general@openib.org; [EMAIL PROTECTED]Subject: RE: [openib-general] RE: [dat-discussions] round 2 - proposal for socket based connection model Caitlin, how does it change the proposed protocol? Arkady Arkady Kanevsky email: [EMAIL PROTECTED] Network Appliance phone: 781-768-5395 375 Totten Pond Rd. Fax: 781-895-1195 Waltham, MA 02451-2010 central phone: 781-768-5300 -Original Message-From: Caitlin Bestler [mailto:[EMAIL PROTECTED] Sent: Tuesday, October 25, 2005 12:36 PMTo: [EMAIL PROTECTED]; openib-general@openib.org; [EMAIL PROTECTED]Subject: [openib-general] RE: [dat-discussions] round 2 - proposal for socket based connection model On an IP network, a non-privileged user is generally not capable of forging a source IP address and is typically prevented from using certain source ports. I would propose that the CM [MAY|SHOULD|MUST] enforce that a non-privileged user can only use a Source IP Address and Port that they would have been able to use following the normal stack path (or what it would have been in the case that there is no conventional IP stack associated with this path). So if IPoIB is installed, you would not be able to use any address that you would have been blocked from using over IPoIB. Or at least you would not be guaranteed that you could. I think that MUST is the correct level of enforcement, but it needs to be clear that the CM and OS *MAY* do this checking and that a userspace IB application cannot use the IB stack to perform IP spoofing. From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Kanevsky, ArkadySent: Tuesday, October 25, 2005 9:00 AMTo: openib-general@openib.org; [EMAIL PROTECTED]; [EMAIL PROTECTED]Subject: [dat-discussions] round 2 - proposal for socket based connection model Dear OpenIB, SWG and DAT members, enclosed is teh second version of the proposal. There are really 2 proposals that are related. The first one is encoding IP 5-tuple into REQ private data with small additional info for versioning and IB capabilities. The second is just a couple of ideas, not a real proposal, on maping of IP ports to IB Service IDs. Thanks everybody for tons of feedback and deep discussions. I appologize if I had missed something. Happy reading, Arkady Arkady Kanevsky email: [EMAIL PROTECTED] Network Appliance phone: 781-768-5395 375 Totten Pond Rd. Fax: 781-895-1195 Waltham, MA 02451-2010 central phone: 781-768-5300 YAHOO! GROUPS LINKS Visit your group "dat-discussions" on the web. To unsubscribe from this group, send an email to: [EMAIL PROTECTED] Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] RE: [dat-discussions] round 2 - proposal for socket based connection model
Title: Message Caitlin, how does it change the proposed protocol? Arkady Arkady Kanevsky email: [EMAIL PROTECTED] Network Appliance phone: 781-768-5395 375 Totten Pond Rd. Fax: 781-895-1195 Waltham, MA 02451-2010 central phone: 781-768-5300 -Original Message-From: Caitlin Bestler [mailto:[EMAIL PROTECTED] Sent: Tuesday, October 25, 2005 12:36 PMTo: [EMAIL PROTECTED]; openib-general@openib.org; [EMAIL PROTECTED]Subject: [openib-general] RE: [dat-discussions] round 2 - proposal for socket based connection model On an IP network, a non-privileged user is generally not capable of forging a source IP address and is typically prevented from using certain source ports. I would propose that the CM [MAY|SHOULD|MUST] enforce that a non-privileged user can only use a Source IP Address and Port that they would have been able to use following the normal stack path (or what it would have been in the case that there is no conventional IP stack associated with this path). So if IPoIB is installed, you would not be able to use any address that you would have been blocked from using over IPoIB. Or at least you would not be guaranteed that you could. I think that MUST is the correct level of enforcement, but it needs to be clear that the CM and OS *MAY* do this checking and that a userspace IB application cannot use the IB stack to perform IP spoofing. From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Kanevsky, ArkadySent: Tuesday, October 25, 2005 9:00 AMTo: openib-general@openib.org; [EMAIL PROTECTED]; [EMAIL PROTECTED]Subject: [dat-discussions] round 2 - proposal for socket based connection model Dear OpenIB, SWG and DAT members, enclosed is teh second version of the proposal. There are really 2 proposals that are related. The first one is encoding IP 5-tuple into REQ private data with small additional info for versioning and IB capabilities. The second is just a couple of ideas, not a real proposal, on maping of IP ports to IB Service IDs. Thanks everybody for tons of feedback and deep discussions. I appologize if I had missed something. Happy reading, Arkady Arkady Kanevsky email: [EMAIL PROTECTED] Network Appliance phone: 781-768-5395 375 Totten Pond Rd. Fax: 781-895-1195 Waltham, MA 02451-2010 central phone: 781-768-5300 YAHOO! GROUPS LINKS Visit your group "dat-discussions" on the web. To unsubscribe from this group, send an email to: [EMAIL PROTECTED] Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] RE: [dat-discussions] round 2 - proposal for socket based connection model
Title: Message On an IP network, a non-privileged user is generally not capable of forging a source IP address and is typically prevented from using certain source ports. I would propose that the CM [MAY|SHOULD|MUST] enforce that a non-privileged user can only use a Source IP Address and Port that they would have been able to use following the normal stack path (or what it would have been in the case that there is no conventional IP stack associated with this path). So if IPoIB is installed, you would not be able to use any address that you would have been blocked from using over IPoIB. Or at least you would not be guaranteed that you could. I think that MUST is the correct level of enforcement, but it needs to be clear that the CM and OS *MAY* do this checking and that a userspace IB application cannot use the IB stack to perform IP spoofing. From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Kanevsky, ArkadySent: Tuesday, October 25, 2005 9:00 AMTo: openib-general@openib.org; [EMAIL PROTECTED]; [EMAIL PROTECTED]Subject: [dat-discussions] round 2 - proposal for socket based connection model Dear OpenIB, SWG and DAT members, enclosed is teh second version of the proposal. There are really 2 proposals that are related. The first one is encoding IP 5-tuple into REQ private data with small additional info for versioning and IB capabilities. The second is just a couple of ideas, not a real proposal, on maping of IP ports to IB Service IDs. Thanks everybody for tons of feedback and deep discussions. I appologize if I had missed something. Happy reading, Arkady Arkady Kanevsky email: [EMAIL PROTECTED] Network Appliance phone: 781-768-5395 375 Totten Pond Rd. Fax: 781-895-1195 Waltham, MA 02451-2010 central phone: 781-768-5300 YAHOO! GROUPS LINKS Visit your group "dat-discussions" on the web. To unsubscribe from this group, send an email to: [EMAIL PROTECTED] Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] question about poll_cq()
Joerg> hi, i hope it's ok to ask this question here. i just want Joerg> to know more about poll_cq(). the standard seems to define Joerg> only the input/ouput-params. after reading the code i Joerg> figured out that there is some kind of "mapping" from Joerg> libibverbs to the device specific "plugin" (mthca). so it Joerg> looks like that the ibv_poll_cq() call from the userspace Joerg> (e.g. ibv_xx_pingpong) finally ends in a call of: Joerg> mthca_poll_cq() in mthca_cq.c Yes, if the underlying device is a Mellanox HCA. PathScale HCAs will end up in a function in libipathverbs, and IBM eHCAs will end up in libehca. Joerg> so my question is: can someone give me a brief summary what Joerg> mthca_poll_cq (or mthca_poll_one()) is really doing? i Joerg> want to know how polling the completion queue really works, Joerg> there must be some kind of low-level acknowledge/response- Joerg> message if a new entry enters the cq? where is the Joerg> completion queue really located (on a rdma operation) - Joerg> local or remote? A very brief sketch of what happens is that the device-specific implementation of CQs for Mellanox HCAs allocates a circular buffer in memory and passes the address to the hardware. The buffer is divided into fixed-size chunks, each of which represents one completion entry. Initially the buffer is cleared out, and every time the hardware adds an entry onto the completion queue, it sets a bit in that chunk to show that the entry is now valid. The driver polls the CQ by looking to see if the next chunk has the bit set. If it does, then the driver translates the entry from hardware format into standard struct ibv_wc format; if it doesn't, then the driver returns status indicating that the CQ is empty. Completion queues are always located in local system memory. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] RFC userspace CMA
Caitlin Bestler wrote: - The kernel CMA will be modified to remove the requirement to use rdma_create_qp(). Users that want to allocate and manage their own QP states will be able to specify QP attributes (qpn, qp_type, srq) through the rdma_conn_param structure. Why? If the userspace CMA talks to the kernel CMA, then the kernel CMA cannot transition the QP. There's not even a valid handle. The alternative is to have the userspace CMA talk to userspace IB CM, SQ query, address translation modules. A user of the kernel CMA can still call rdma_create_qp() and have the kernel CMA transition it for them. The same is true for userspace applications. Why does the uCMA need to open HCAs? Why does it have to be anything other than a front-end to the kCMA? The kernel CMA abstracts device addition/removal from the user. To accomplish the same goals with the userspace CMA, the uCMA needs to open/close the device. If the user opens and closes the device, then API changes are necessary. I don't see any benefit for the user to open the device, since it requires users to search for devices based on some sort of identifier. I can see where a user-mode daemon might get a connection request that could only be answered with a QP on an device that it had not previolusly opened, and that opening that device based on information in the Connection Request would be useful -- but that still doesn't have the uCMA opening the device. More importantly, the problem can be solved just as easily by having the listener listen only on rdma devices that it has already opened. This results in the listen call operating on an RDMA device, rather than on an IP address, which is the intent of the API. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] RFC userspace CMA
> -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Sean Hefty > Sent: Monday, October 24, 2005 5:06 PM > To: openib > Subject: [openib-general] RFC userspace CMA > > I'm soliciting any comments that anyone might have on the > general design for the userspace CMA before I get too far > into the implementation. > > - The API will match the kernel API for the most part. The > exception is that event handling will match other userspace > libraries (get/ack event). > > - There will be a single CMA device exported through > /sys/class/infiniband. > > - The kernel CMA will be modified to remove the requirement > to use rdma_create_qp(). Users that want to allocate and > manage their own QP states will be able to specify QP > attributes (qpn, qp_type, srq) through the rdma_conn_param structure. > Why? Every CM interface I've dealt with has had the Consumer create and configure the QP on each end. On the active side the QP is supplied with the connect request. On the passive side it is supplied with the accept. State modifications and other configuration changes were done by the CM based on the Consumer having passed in the handle. > - The kernel CMA will expose a new call, rdma_init_qp_attr() > to initialize QP attributes used to modify the state of the > QP. The call will be similar to the infiniband CM routine. > Use of this call is optional. The CMA will automatically > transition QPs created by rdma_create_qp(). > > - The uCMA will open devices for users and return them the > device context with related events. The uCMA will close the > device if there are no rdma_cma_id's associated with it. > > - To support device add, the uCMA will need a new verb's call: > ibv_open_device_by_guid(). If a connection request occurs > for a device that is not yet known by the uCMA, it will open > the device. > Why does the uCMA need to open HCAs? Why does it have to be anything other than a front-end to the kCMA? I can see where a user-mode daemon might get a connection request that could only be answered with a QP on an device that it had not previolusly opened, and that opening that device based on information in the Connection Request would be useful -- but that still doesn't have the uCMA opening the device. More importantly, the problem can be solved just as easily by having the listener listen only on rdma devices that it has already opened. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] Fix for MAD layer DMA mappings
Sean Hefty wrote: The following patch should fix the MAD layer's DMA mapping issue. This patch includes all related patches that were previously posted. The fix involved changing the MAD layer API. All callers must now use the MAD layer to allocate and free send MADs. DMA mappings are done by the MAD layer. These changes have been committed. Let me know if I broke anything. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] question about poll_cq()
> -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Joerg Zinke > Sent: Tuesday, October 25, 2005 2:40 AM > To: openib-general@openib.org > Subject: [openib-general] question about poll_cq() > > hi, > > i hope it's ok to ask this question here. > i just want to know more about poll_cq(). > the standard seems to define only the input/ouput-params. > after reading the code i figured out that there is some kind > of "mapping" from libibverbs to the device specific "plugin" (mthca). > so it looks like that the ibv_poll_cq() call from the userspace (e.g. > ibv_xx_pingpong) finally ends in a call of: mthca_poll_cq() > in mthca_cq.c > > so my question is: can someone give me a brief summary what > mthca_poll_cq (or mthca_poll_one()) is really doing? > i want to know how polling the completion queue really works, > there must be some kind of low-level acknowledge/response- > message if a new entry enters the cq? > where is the completion queue really located (on a rdma > operation) - local or remote? > Understanding how a given device implements poll_cq is legitimate if the purpose is debugging and/or understanding memory/bus utilization. However it is one of those that you SHOULD NOT know when righting code that uses poll_cq. What will vary over time and model is where the work completions are stored (in device and/or host memory) how they are formatted and whether they are self contained or reference other data (such as the original work request). The work completion that you get from a successful poll_cq may never have existed as that sequence of bytes until you made the call to poll_cq. If these things were not deliberately undefined then there would be no need for both ibv_poll_cq and device specific methods. For a discussion on what you can assume about a CQ across all devices I'd suggest reviewing the IBTA and/or RDMAC verbs. But basically, about all that is guaranteed is that it is a reliable ordered queued that the Consumer MUST provision adequately so as to avoid overflows. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] $BD>%"%I65$($^$9!#(B
$B!V$O$8$a$^$7$F!#%^%j$C$F$^$9!#$$$-$J$j$N%a!<%k$4$a$s$J(B $B$5$$!#(B $BAjCL$K>h$C$FM_$7$/$F!"%a!<%k$7$F$_$^$7$?!#CK$N?M$G$9$h$M!)(B $B!!%a!<%k=P$7$?;~$+$iF,$NCf$O$3$N=P2q$$$N;v$G0l?'@w$^$C$A$c(B $B$C$F$$$^$9!#$=$A$i$O;d$N;v$I$&;W$$$^$9$+!)(B($B6[D%46!*!)!K$:(B $B$C$HG:$s$G$$$^$7$?!#7k6I<+J,$+$i$3$&$7$F%a!<%kAw$i$J$$$H0l(B $BJb$b?J$^$J$/>!%"%I65$($^$9(B $B!#7h$7$F5?$C$F$O$$$^$;$s$,!"=c?h$J=P2q$$$K$7$?$/$F!"$A$g$C(B $B$H?5=E$K$J$C$F$$$^$9!#(B $B!!2q$C$F$/$l$k$J$i!";[EMAIL PROTECTED];W$$$^$9$N$G!"(B $B46http://www.otakkujp.net?adress$B!V%^%j!W$GEPO?(B $B$7$F$$$^$9!#$"$/$^$G$b%$%?%:%iKI;_$N0Y$K%5%$%H7PM3$r$H$C$F(B $B$$$k$N$GM}2r$7$F$/[EMAIL PROTECTED])$=$l$G$O!"$h$m$7$/$M!#!W(B $B!LE:IU!M!'@5D>[EMAIL PROTECTED]"=c?h$J=P2q$$$rCg2p$7$F$*$j$^$9(B $B!#%W%i%$%P%7!pJs$N3NG'$OL5NAEPO?$+$i"-(B http://www.otakkujp.net?imasugu [EMAIL PROTECTED]@$1$G$J$/!"L5NA$G2q0w$K$J$i$l$?J}$X$N%"%I8x3+(B [EMAIL PROTECTED]>R2p$bKhF|9T$C$F$*$j$^$9$N$G!"Hs>o$KJXMx$G$9!#(B $B5qH](B [EMAIL PROTECTED] ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] $BD>%"%I65$($^$9!#(B
$B!V$O$8$a$^$7$F!#%^%j$C$F$^$9!#$$$-$J$j$N%a!<%k$4$a$s$J(B $B$5$$!#(B $BAjCL$K>h$C$FM_$7$/$F!"%a!<%k$7$F$_$^$7$?!#CK$N?M$G$9$h$M!)(B $B!!%a!<%k=P$7$?;~$+$iF,$NCf$O$3$N=P2q$$$N;v$G0l?'@w$^$C$A$c(B $B$C$F$$$^$9!#$=$A$i$O;d$N;v$I$&;W$$$^$9$+!)(B($B6[D%46!*!)!K$:(B $B$C$HG:$s$G$$$^$7$?!#7k6I<+J,$+$i$3$&$7$F%a!<%kAw$i$J$$$H0l(B $BJb$b?J$^$J$/>!%"%I65$($^$9(B $B!#7h$7$F5?$C$F$O$$$^$;$s$,!"=c?h$J=P2q$$$K$7$?$/$F!"$A$g$C(B $B$H?5=E$K$J$C$F$$$^$9!#(B $B!!2q$C$F$/$l$k$J$i!";[EMAIL PROTECTED];W$$$^$9$N$G!"(B $B46http://www.otakkujp.net?adress$B!V%^%j!W$GEPO?(B $B$7$F$$$^$9!#$"$/$^$G$b%$%?%:%iKI;_$N0Y$K%5%$%H7PM3$r$H$C$F(B $B$$$k$N$GM}2r$7$F$/[EMAIL PROTECTED])$=$l$G$O!"$h$m$7$/$M!#!W(B $B!LE:IU!M!'@5D>[EMAIL PROTECTED]"=c?h$J=P2q$$$rCg2p$7$F$*$j$^$9(B $B!#%W%i%$%P%7!pJs$N3NG'$OL5NAEPO?$+$i"-(B http://www.otakkujp.net?imasugu [EMAIL PROTECTED]@$1$G$J$/!"L5NA$G2q0w$K$J$i$l$?J}$X$N%"%I8x3+(B [EMAIL PROTECTED]>R2p$bKhF|9T$C$F$*$j$^$9$N$G!"Hs>o$KJXMx$G$9!#(B $B5qH](B [EMAIL PROTECTED] ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] question about poll_cq()
hi, i hope it's ok to ask this question here. i just want to know more about poll_cq(). the standard seems to define only the input/ouput-params. after reading the code i figured out that there is some kind of "mapping" from libibverbs to the device specific "plugin" (mthca). so it looks like that the ibv_poll_cq() call from the userspace (e.g. ibv_xx_pingpong) finally ends in a call of: mthca_poll_cq() in mthca_cq.c so my question is: can someone give me a brief summary what mthca_poll_cq (or mthca_poll_one()) is really doing? i want to know how polling the completion queue really works, there must be some kind of low-level acknowledge/response- message if a new entry enters the cq? where is the completion queue really located (on a rdma operation) - local or remote? or maybe give me some hints/links to docs where i can read more about it. thanks. regards, joerg zinke ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: [RFC] OpenSM Interactive Console
Hal Rosenstock wrote: On Mon, 2005-10-24 at 14:24, Eitan Zahavi wrote: How do you get the old versions of this ? It is in the main trunk ... https://openib.org/svn/gen2/trunk/src/userspace/management/osm/doc/OpenS M_UM.pdf That's older than 1.7.1 1.7.0 manuals you had mentioned. Maybe nothing has changed in osmsh between the version of the manual in the trunk and currently. Yes correct no API changed -- Hal ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [RFC] OpenSM Interactive Console
Hal Rosenstock wrote:
On Mon, 2005-10-24 at 14:38, Eitan Zahavi wrote:
Hal Rosenstock wrote:
On Mon, 2005-10-24 at 03:08, Eitan Zahavi wrote:
I would suggest to use SNMP for the tasks below. IETF IPoIB group
has
defined an SNMP MIB that can support the required functionality
below.
The IETF SNMP MIBs are one way of presenting the information to the
outside world. There are other possible management interfaces. The
SNMP
MIB instrumentation would need to use lower layer APIs to get this
information out of the SM.
Yes but the IETF SM MIB is the only one that is close to a standard
way.
It does not require low level interface if it will integrate into the
OpenSM code.
One way to do it is buy extending OpenSM with an AgentX interface.
IMO one clear advantage of using SNMP for SM integration is that the
code will work with any SM that is IETF compliant.
Also if you want to write a "client server" type of application on top
of an SM you
can either stick to sending MADs which translate into SA client based
application or
you better stay with some known protocol for management (like SNMP)
and not develop yet another protocol for
doing exactly the same things as SNMP already supports.
There are limitations in the SNMP MIBs. One is that they are RO so they
are more for monitoring. Also, many environments do not use SNMP. It is
unclear how much of a requirement it is to manage any SM or how many
other SMs support the SM MIB. (There are other IB associated MIBs too).
SNMP MIBs are certainly not just RO a simple example from the SM MIB:
ibSmPortInfoLMC OBJECT-TYPE
SYNTAX Unsigned32(0..7)
MAX-ACCESS read-write
STATUS current
DESCRIPTION
"LID mask for multipath support. User should take extra caution
when setting this value, since any change will effect packet
routing."
::= { ibSmPortInfoEntry 19 }
I agree that it is possible that currently no SM is supporting the SM MIB.
But it does make sense to have ALL of the them support it. Such that they can
be activated/deactivated and configured in the manner.
Most unix distributions and windows box have standard SNMP agent and client
included in them
So it does not take more then simple bash or C code to interact with the SM if
it supports SNMP.
Everything but the dynamic partitioning (OpenSM does not have
partition manager to this moment)
What Troy meant by partitioning is not necessarily IB partitioning.
How are you sure about that? Troy - please comment.
I think you missed an email on this.
and forwarding of Performance
Monitoring traps (which are generated by the PM) can be done through
osmsh or through SA client today.
What PerfMgr are you referring to ?
No specific one. But the specification does not require the SM too.
Huh ? What spec ? An SM is required in a subnet. There is no subnet
without this. There is a subnet without a PerfMgr.
Yes its a typo I meant PM. SM is a requirement. You know I did not mean that.
For various reasons (like load) it might make more sense to have the
PM distributed.
Sure. Also, the PerfMgr need not be colocated with the SM anyhow.
Anyway, my point is that the SM is not the owner of PM trap reporting.
It is the PM that
should support Reporting (I.e InformInfo registration and Trap
forwarding) for PM traps.
But the spec does not define such traps anyway.
My point was that the PerfMgr is beyond the IBA spec. It is only the PMA
that is defined and has no traps so these will all need synthesis by the
PerfMgr.
Agree.
-- Hal
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] $BD>%"%I65$($^$9!#(B
$B!V$O$8$a$^$7$F!#%^%j$C$F$^$9!#$$$-$J$j$N%a!<%k$4$a$s$J(B $B$5$$!#(B $BAjCL$K>h$C$FM_$7$/$F!"%a!<%k$7$F$_$^$7$?!#CK$N?M$G$9$h$M!)(B $B!!%a!<%k=P$7$?;~$+$iF,$NCf$O$3$N=P2q$$$N;v$G0l?'@w$^$C$A$c(B $B$C$F$$$^$9!#$=$A$i$O;d$N;v$I$&;W$$$^$9$+!)(B($B6[D%46!*!)!K$:(B $B$C$HG:$s$G$$$^$7$?!#7k6I<+J,$+$i$3$&$7$F%a!<%kAw$i$J$$$H0l(B $BJb$b?J$^$J$/>!%"%I65$($^$9(B $B!#7h$7$F5?$C$F$O$$$^$;$s$,!"=c?h$J=P2q$$$K$7$?$/$F!"$A$g$C(B $B$H?5=E$K$J$C$F$$$^$9!#(B $B!!2q$C$F$/$l$k$J$i!";[EMAIL PROTECTED];W$$$^$9$N$G!"(B $B46http://www.otakkujp.net?adress$B!V%^%j!W$GEPO?(B $B$7$F$$$^$9!#$"$/$^$G$b%$%?%:%iKI;_$N0Y$K%5%$%H7PM3$r$H$C$F(B $B$$$k$N$GM}2r$7$F$/[EMAIL PROTECTED])$=$l$G$O!"$h$m$7$/$M!#!W(B $B!LE:IU!M!'@5D>[EMAIL PROTECTED]"=c?h$J=P2q$$$rCg2p$7$F$*$j$^$9(B $B!#%W%i%$%P%7!pJs$N3NG'$OL5NAEPO?$+$i"-(B http://www.otakkujp.net?imasugu [EMAIL PROTECTED]@$1$G$J$/!"L5NA$G2q0w$K$J$i$l$?J}$X$N%"%I8x3+(B [EMAIL PROTECTED]>R2p$bKhF|9T$C$F$*$j$^$9$N$G!"Hs>o$KJXMx$G$9!#(B $B5qH](B [EMAIL PROTECTED] ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
