RE: [dat-discussions] RE: [openib-general] Re: iWARP emulation protocol
Arkady, Intel MPI (real consumer of uDAPL) has no problem with this change. -arlin From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Kanevsky, Arkady Sent: Wednesday, October 19, 2005 6:40 AM To: Grant Grundler; Caitlin Bestler Cc: Roland Dreier; [EMAIL PROTECTED]; [EMAIL PROTECTED]; openib-general@openib.org Subject: [dat-discussions] RE: [openib-general] Re: iWARP emulation protocol Grant, The developers of the application(s) in questions are aware of the discussion. I will leave it to them to respond. I bring the discussion point at the weekly DAT Collaborative meeting which we have every Wednesday. I appologize that the DAT Collaborative charter does not allow to submit contribution without joining DAT Collaborative. But this is no different from Linux not accepting any contrubutions without proper license. Byt be rest assure that as a Chair I bring the concerns and suggestions stated in email discussion at the DAT meetings. Arkady Arkady Kanevsky email: [EMAIL PROTECTED] Network Appliance phone: 781-768-5395 375 Totten Pond Rd. Fax: 781-895-1195 Waltham, MA 02451-2010 central phone: 781-768-5300 -Original Message- From: Grant Grundler [mailto:[EMAIL PROTECTED] Sent: Tuesday, October 18, 2005 8:02 PM To: Caitlin Bestler Cc: Grant Grundler; Roland Dreier; Kanevsky, Arkady; [EMAIL PROTECTED]; [EMAIL PROTECTED]; openib-general@openib.org Subject: Re: [openib-general] Re: iWARP emulation protocol On Tue, Oct 18, 2005 at 04:40:54PM -0700, Caitlin Bestler wrote: Roland (and the rest of us) would like to see someone name a real consumer of the proposed interface. ie who depends on this change? Then the dependency for that use/user can be discussed and appropriate tradeoffs made. Make sense? Unfortunately not every application that is under development, or even deployed, can be discussed in a google-searchable public forum. That especially applies to user-mode development. Well, this is open source. While I don't want to preclude closed source developement, it's usually necessary to have an open source consumer that any open source developer can test with. So I could have actually tested such applications and still not be free to cite them here. Understood. I'm not asking *you* to cite one unless you happen to own one of the consumers. With any luck some of them are following the discussion and will jump in on their own. Unfortunately, since they are developing to uDAPL they are unlikely to be following this discussion. It doesn't help that the DAT yahoo-groups.com mailing list is rejecting my replies. It would be helpful if someone following this forum could share Roland's question with DAT mailing list if it didn't make it there already and possibly explain why naming a consumer is necessary. hth, grant YAHOO! GROUPS LINKS Visit your group dat-discussions on the web. To unsubscribe from this group, send an email to: [EMAIL PROTECTED] Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] Re: iWARP emulation protocol
-Original Message- From: Roland Dreier Sent: Tuesday, October 18, 2005 11:41 AM To: Kanevsky, Arkady Subject: [openib-general] Re: iWARP emulation protocol Arkady uDAPL users. 2) Are there real users or is this a generic uDAPL API thing? uDAPL vs. kDAPL is irrelevant here. The user or Kernel Consumer making the connection does not know whether their peer is running in user or kernel, nor should they. Every discussion of reducing the guaranteed private data size in DAPL has produced adverse reactions from application developers. They're either very good actors or were working on actual applications. An additional space preserving option that Arkady did not mention is limiting the IP alias service to IPv4 addresses. Anyone who really wants IPv6 addresses can get their SM to assign IPv6 compatible GIDs. Of course the flat IPv6 option is far simpler, and probably should be used unless a specific application is identified where those extra 96 bits makes the difference between making the private data be rewritten or left as is. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] Re: iWARP emulation protocol
An additional space preserving option that Arkady did not mention is limiting the IP alias service to IPv4 addresses. Anyone who really wants IPv6 addresses can get their SM to assign IPv6 compatible GIDs. Of course the flat IPv6 option is far simpler, and probably should be used unless a specific application is identified where those extra 96 bits makes the difference between making the private data be rewritten or left as is. This can be an extension to proposal 3 of last page. Arkady Kanevsky email: [EMAIL PROTECTED] Network Appliance phone: 781-768-5395 375 Totten Pond Rd. Fax: 781-895-1195 Waltham, MA 02451-2010 central phone: 781-768-5300 -Original Message- From: Caitlin Bestler [mailto:[EMAIL PROTECTED] Sent: Tuesday, October 18, 2005 3:16 PM To: Roland Dreier; Kanevsky, Arkady Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED]; openib-general@openib.org Subject: RE: [openib-general] Re: iWARP emulation protocol -Original Message- From: Roland Dreier Sent: Tuesday, October 18, 2005 11:41 AM To: Kanevsky, Arkady Subject: [openib-general] Re: iWARP emulation protocol Arkady uDAPL users. 2) Are there real users or is this a generic uDAPL API thing? uDAPL vs. kDAPL is irrelevant here. The user or Kernel Consumer making the connection does not know whether their peer is running in user or kernel, nor should they. Every discussion of reducing the guaranteed private data size in DAPL has produced adverse reactions from application developers. They're either very good actors or were working on actual applications. An additional space preserving option that Arkady did not mention is limiting the IP alias service to IPv4 addresses. Anyone who really wants IPv6 addresses can get their SM to assign IPv6 compatible GIDs. Of course the flat IPv6 option is far simpler, and probably should be used unless a specific application is identified where those extra 96 bits makes the difference between making the private data be rewritten or left as is. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] RE: iWARP emulation protocol
Kanevsky, Arkady wrote: uDAPL users. I'm not sure how much we should care about higher level abstractions for this discussion. We should do what's right for IB. Abstractions that want to use IP addresses can either use the standard protocol defined by the IBTA or define their own private data. To me, it seems that the most flexible solution is to pass the source and destination IP address in the CM REQ. We can then define a standard mapping from TCP port numbers to IB service records, or change the CM version to read into the private data. What's wrong with this approach? - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] RE: iWARP emulation protocol
Sean wrote: I'm not sure how much we should care about higher level abstractions for this discussion. We should do what's right for IB. Abstractions that want to use IP addresses can either use the standard protocol defined by the IBTA or define their own private data. Correct. But we should define standard protocol suited for most apps to avoid creations of multiple apps specific protocols. To me, it seems that the most flexible solution is to pass the source and destination IP address in the CM REQ. I agree. This is the cleanest and most simple to define. But it impacts some existing apps. That is why DAT has 64 bytes private data req. We do not loose too many users by the time we define the complete solution stack. We can then define a standard mapping from TCP port numbers to IB service records, or change the CM version to read into the private data. What's wrong with this approach? It is the standard mapping which we just spend 1 hour discussing at SWG. What is that standard mapping if it is native IB? IPoIB as intermediate layer? SDP as intermediate layer? What is the standard TCP port for iSER (pick your ULP) native over RDMA vs. the same ULP over IPoIB? This have to be defined. But is it part of the IP address and TCP port info sharing between 2 sides of the connection proposal or a separate proposal? I think it is separate proposal but both will have to be in place to support iWARP emulation. Arkady Arkady Kanevsky email: [EMAIL PROTECTED] Network Appliance phone: 781-768-5395 375 Totten Pond Rd. Fax: 781-895-1195 Waltham, MA 02451-2010 central phone: 781-768-5300 ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Re: iWARP emulation protocol
Hello, I had an email exchange with the iSER folks earlier this summer where I submitted the attached proposal for consideration. It does not contain the local port number because nobody seems to set it to a meaningful value. More food for thought. -David Roland Dreier wrote: Roland I have a few minor quibbles with this proposal. I think Roland it would be better to have only the IP version, source and Roland destination IPs and local in the CM private data. err-- ...and local PORT NUMBER in the CM private data... - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ---BeginMessage--- I'd like to propose a standard IBTA Hello/HelloAck message carried in the CM private data area and IANA port number to IB service ID mapping for use by IETF RDMA protocols mapped onto IB. I'm hoping that this extension will enable iSER, NFS, and perhaps future iWARP ULPs to transmit IP address information that is needed for authentication and advertise well-known service locations. The proposal consists of the following changes to version 1.2 of volume 1 IBTA specification: 1) When the Service ID is in the IBTA assigned range as described in section A3.2.3.1 where byte 6 is set to 0x02, the CM REQ packet will contain the following Hello message starting at offset 140 (start of private data area): bit 0 1 2 3 byte 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 0-3 | MajV | MinV | IPV | Cap | Reserved | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 4-7 | Src IP (127-96) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 8-11 | Src IP ( 95-64) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 12-15 | Src IP ( 63-32) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 16-19 | Src IP ( 31-00) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 20-23 | Dst IP (127-96) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 24-27 | Dst IP ( 95-64) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 28-31 | Dst IP ( 63-32) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 32-35 | Dst IP ( 31-00) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ where: MajV (4 bits) - The Major Protocol version number. The value must be set to 1. The accepting peer shall reject the connection if MajV in the Hello message does not match its local value. MinV (4 bits) - The Minor Protocol version number. The value must be set to 1. The accepting peer shall not reject the connection, solely on the basis that MinV of the Hello message does not match its local value. This enables future protocol extensions which are upwardly compatible. IPV (4 bits) - The Internet Protocol version number of the end-point address field (fields Src IP and Dst IP). If IPV = 0x4, the IP addresses are IP version 4 format (32 bit addresses). If IPV = 0x6, then both IP addresses are IP version 6 format (128 bit addresses). All other IPV values are reserved. The accepting peer shall reject the connection if IPV of the Hello message has a value other than 0x4 or 0x6. Cap (4 bits) - The Capabilities field conveys the connecting peers capabilities. The following capabilities are defined: Bit Position Capability Description 0 INVALIDATE_CAP Supports incoming Send w/Invalidate opcode 1 VABTO_CAP Supports incoming ZBVA Messages 2-3 reserved transmitted as zero and not checked at receiver where: The connecting peer shall set the INVALIDATE_CAP bit in the Cap field in Hello Message only if it can support incoming invalidate messages. If the accepting peer does not support Send w/Invalidate, it will ignore the Invalidate Capability advertisement. The connecting peer shall set the ZBVA_CAP bit in the Cap field in Hello Message only if it can support Zero based Virtual Addresses (ZBVA). If the accepting peer does not support ZBVA, it will ignore the ZBVA Capability advertisement. Internet Protocol Address (128 bits) - The Internet Protocol (IP) address for the local interface (Src IP) and remote interface (Dst IP). These can be either IPv4 or IPv6 addresses, as specified by the IPV field. If Src IP and Dst IP of the Hello Message are IPv4 addresses, as specified by the IPV field, then Src IP (31-0) and Dst IP (31-0) shall be used to transmit the source and destination IP addresses, respectively, and bits Src IP (127-32) and Dst IP (127-32) shall be set to zero. and the CM REP
