RE: [dat-discussions] RE: [openib-general] Re: iWARP emulation protocol
Arkady, Intel MPI (real consumer of uDAPL) has no problem with this change. -arlin From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Kanevsky, Arkady Sent: Wednesday, October 19, 2005 6:40 AM To: Grant Grundler; Caitlin Bestler Cc: Roland Dreier; [EMAIL PROTECTED]; [EMAIL PROTECTED]; openib-general@openib.org Subject: [dat-discussions] RE: [openib-general] Re: iWARP emulation protocol Grant, The developers of the application(s) in questions are aware of the discussion. I will leave it to them to respond. I bring the discussion point at the weekly DAT Collaborative meeting which we have every Wednesday. I appologize that the DAT Collaborative charter does not allow to submit contribution without joining DAT Collaborative. But this is no different from Linux not accepting any contrubutions without proper license. Byt be rest assure that as a Chair I bring the concerns and suggestions stated in email discussion at the DAT meetings. Arkady Arkady Kanevsky email: [EMAIL PROTECTED] Network Appliance phone: 781-768-5395 375 Totten Pond Rd. Fax: 781-895-1195 Waltham, MA 02451-2010 central phone: 781-768-5300 > -Original Message- > From: Grant Grundler [mailto:[EMAIL PROTECTED] > Sent: Tuesday, October 18, 2005 8:02 PM > To: Caitlin Bestler > Cc: Grant Grundler; Roland Dreier; Kanevsky, Arkady; > [EMAIL PROTECTED]; [EMAIL PROTECTED]; > openib-general@openib.org > Subject: Re: [openib-general] Re: iWARP emulation protocol > > > On Tue, Oct 18, 2005 at 04:40:54PM -0700, Caitlin Bestler wrote: > > > Roland (and the rest of us) would like to see someone name a > > > real consumer of the proposed interface. ie who depends on > > > this change? > > > Then the dependency for that use/user can be discussed and > > > appropriate tradeoffs made. Make sense? > > > > Unfortunately not every application that is under > development, or even > > deployed, can be discussed in a google-searchable public > forum. That > > especially applies to user-mode development. > > Well, this is open source. While I don't want to preclude > closed source developement, it's usually necessary to have an > open source consumer that any open source developer can test with. > > > So I could have actually tested such applications and still not be > > free to cite them here. > > Understood. I'm not asking *you* to cite one unless you > happen to own one of the consumers. > > > With any luck some of them > > are following the discussion and will jump in on their own. > > Unfortunately, since they are developing to uDAPL they are > unlikely to > > be following this discussion. > > It doesn't help that the DAT yahoo-groups.com mailing list is > rejecting my replies. It would be helpful if someone > following this forum could share Roland's question with DAT > mailing list if it didn't make it there already and possibly > explain why naming a consumer is necessary. > > hth, > grant > YAHOO! GROUPS LINKS Visit your group "dat-discussions" on the web. To unsubscribe from this group, send an email to: [EMAIL PROTECTED] Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] Re: iWARP emulation protocol
Grant, The developers of the application(s) in questions are aware of the discussion. I will leave it to them to respond. I bring the discussion point at the weekly DAT Collaborative meeting which we have every Wednesday. I appologize that the DAT Collaborative charter does not allow to submit contribution without joining DAT Collaborative. But this is no different from Linux not accepting any contrubutions without proper license. Byt be rest assure that as a Chair I bring the concerns and suggestions stated in email discussion at the DAT meetings. Arkady Arkady Kanevsky email: [EMAIL PROTECTED] Network Appliance phone: 781-768-5395 375 Totten Pond Rd. Fax: 781-895-1195 Waltham, MA 02451-2010 central phone: 781-768-5300 > -Original Message- > From: Grant Grundler [mailto:[EMAIL PROTECTED] > Sent: Tuesday, October 18, 2005 8:02 PM > To: Caitlin Bestler > Cc: Grant Grundler; Roland Dreier; Kanevsky, Arkady; > [EMAIL PROTECTED]; [EMAIL PROTECTED]; > openib-general@openib.org > Subject: Re: [openib-general] Re: iWARP emulation protocol > > > On Tue, Oct 18, 2005 at 04:40:54PM -0700, Caitlin Bestler wrote: > > > Roland (and the rest of us) would like to see someone name a > > > real consumer of the proposed interface. ie who depends on > > > this change? > > > Then the dependency for that use/user can be discussed and > > > appropriate tradeoffs made. Make sense? > > > > Unfortunately not every application that is under > development, or even > > deployed, can be discussed in a google-searchable public > forum. That > > especially applies to user-mode development. > > Well, this is open source. While I don't want to preclude > closed source developement, it's usually necessary to have an > open source consumer that any open source developer can test with. > > > So I could have actually tested such applications and still not be > > free to cite them here. > > Understood. I'm not asking *you* to cite one unless you > happen to own one of the consumers. > > > With any luck some of them > > are following the discussion and will jump in on their own. > > Unfortunately, since they are developing to uDAPL they are > unlikely to > > be following this discussion. > > It doesn't help that the DAT yahoo-groups.com mailing list is > rejecting my replies. It would be helpful if someone > following this forum could share Roland's question with DAT > mailing list if it didn't make it there already and possibly > explain why naming a consumer is necessary. > > hth, > grant > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Re: iWARP emulation protocol
On Tue, Oct 18, 2005 at 04:40:54PM -0700, Caitlin Bestler wrote: > > Roland (and the rest of us) would like to see someone name a > > real consumer of the proposed interface. ie who depends on > > this change? > > Then the dependency for that use/user can be discussed and > > appropriate tradeoffs made. Make sense? > > Unfortunately not every application that is under development, > or even deployed, can be discussed in a google-searchable > public forum. That especially applies to user-mode development. Well, this is open source. While I don't want to preclude closed source developement, it's usually necessary to have an open source consumer that any open source developer can test with. > So I could have actually tested such applications and still > not be free to cite them here. Understood. I'm not asking *you* to cite one unless you happen to own one of the consumers. > With any luck some of them > are following the discussion and will jump in on their own. > Unfortunately, since they are developing to uDAPL they are > unlikely to be following this discussion. It doesn't help that the DAT yahoo-groups.com mailing list is rejecting my replies. It would be helpful if someone following this forum could share Roland's question with DAT mailing list if it didn't make it there already and possibly explain why naming a consumer is necessary. hth, grant ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] Re: iWARP emulation protocol
> > Roland (and the rest of us) would like to see someone name a > real consumer of the proposed interface. ie who depends on > this change? > Then the dependency for that use/user can be discussed and > appropriate tradeoffs made. Make sense? > Unfortunately not every application that is under development, or even deployed, can be discussed in a google-searchable public forum. That especially applies to user-mode development. So I could have actually tested such applications and still not be free to cite them here. With any luck some of them are following the discussion and will jump in on their own. Unfortunately, since they are developing to uDAPL they are unlikely to be following this discussion. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Re: iWARP emulation protocol
On Tue, Oct 18, 2005 at 12:16:25PM -0700, Caitlin Bestler wrote: > > 2) Are there real users or is this a generic uDAPL API thing? > > uDAPL vs. kDAPL is irrelevant here. The user or Kernel Consumer > making the connection does not know whether their peer is running > in user or kernel, nor should they. Caitlin, I didn't see an answer to Roland's question in your reply. There is no kDAPL in linux. So yes, I agree uDAPL v kDAPL is irrelevant. > Every discussion of reducing the guaranteed private data size > in DAPL has produced adverse reactions from application developers. > They're either very good actors or were working on actual applications. Roland (and the rest of us) would like to see someone name a real consumer of the proposed interface. ie who depends on this change? Then the dependency for that use/user can be discussed and appropriate tradeoffs made. Make sense? hth, grant ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] RE: iWARP emulation protocol
Sean wrote: > I'm not sure how much we should care about higher level > abstractions for this > discussion. We should do what's right for IB. Abstractions > that want to use IP > addresses can either use the standard protocol defined by the > IBTA or define > their own private data. Correct. But we should define standard protocol suited for most apps to avoid creations of multiple apps specific protocols. > > To me, it seems that the most flexible solution is to pass > the source and > destination IP address in the CM REQ. I agree. This is the cleanest and most simple to define. But it impacts some existing apps. That is why DAT has 64 bytes private data req. We do not loose too many users by the time we define the complete solution stack. > We can then define a > standard mapping > from TCP port numbers to IB service records, or change the CM > version to read > into the private data. What's wrong with this approach? It is the "standard" mapping which we just spend 1 hour discussing at SWG. What is that standard mapping if it is native IB? IPoIB as intermediate layer? SDP as intermediate layer? What is the standard TCP port for iSER (pick your ULP) native over RDMA vs. the same ULP over IPoIB? This have to be defined. But is it part of the IP address and TCP port info sharing between 2 sides of the connection proposal or a separate proposal? I think it is separate proposal but both will have to be in place to support iWARP emulation. Arkady Arkady Kanevsky email: [EMAIL PROTECTED] Network Appliance phone: 781-768-5395 375 Totten Pond Rd. Fax: 781-895-1195 Waltham, MA 02451-2010 central phone: 781-768-5300 ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] RE: iWARP emulation protocol
Kanevsky, Arkady wrote: uDAPL users. I'm not sure how much we should care about higher level abstractions for this discussion. We should do what's right for IB. Abstractions that want to use IP addresses can either use the standard protocol defined by the IBTA or define their own private data. To me, it seems that the most flexible solution is to pass the source and destination IP address in the CM REQ. We can then define a standard mapping from TCP port numbers to IB service records, or change the CM version to read into the private data. What's wrong with this approach? - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] Re: iWARP emulation protocol
> > An additional space preserving option that Arkady did not > mention is limiting the IP alias service to IPv4 addresses. > Anyone who really wants IPv6 addresses can get their SM to > assign IPv6 compatible GIDs. Of course the flat IPv6 option > is far simpler, and probably should be used unless a specific > application is identified where those extra 96 bits makes the > difference between making the private data be rewritten or left as is. > This can be an extension to proposal 3 of last page. Arkady Kanevsky email: [EMAIL PROTECTED] Network Appliance phone: 781-768-5395 375 Totten Pond Rd. Fax: 781-895-1195 Waltham, MA 02451-2010 central phone: 781-768-5300 > -Original Message- > From: Caitlin Bestler [mailto:[EMAIL PROTECTED] > Sent: Tuesday, October 18, 2005 3:16 PM > To: Roland Dreier; Kanevsky, Arkady > Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED]; > openib-general@openib.org > Subject: RE: [openib-general] Re: iWARP emulation protocol > > > > > > -Original Message- > > From: Roland Dreier > > Sent: Tuesday, October 18, 2005 11:41 AM > > To: Kanevsky, Arkady > > Subject: [openib-general] Re: iWARP emulation protocol > > > > Arkady> uDAPL users. > > > > > > 2) Are there real users or is this a generic uDAPL API thing? > > > > uDAPL vs. kDAPL is irrelevant here. The user or Kernel > Consumer making the connection does not know whether their > peer is running in user or kernel, nor should they. > > Every discussion of reducing the guaranteed private data size > in DAPL has produced adverse reactions from application > developers. They're either very good actors or were working > on actual applications. > > An additional space preserving option that Arkady did not > mention is limiting the IP alias service to IPv4 addresses. > Anyone who really wants IPv6 addresses can get their SM to > assign IPv6 compatible GIDs. Of course the flat IPv6 option > is far simpler, and probably should be used unless a specific > application is identified where those extra 96 bits makes the > difference between making the private data be rewritten or left as is. > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] Re: iWARP emulation protocol
> -Original Message- > From: Roland Dreier > Sent: Tuesday, October 18, 2005 11:41 AM > To: Kanevsky, Arkady > Subject: [openib-general] Re: iWARP emulation protocol > > Arkady> uDAPL users. > > > 2) Are there real users or is this a generic uDAPL API thing? > uDAPL vs. kDAPL is irrelevant here. The user or Kernel Consumer making the connection does not know whether their peer is running in user or kernel, nor should they. Every discussion of reducing the guaranteed private data size in DAPL has produced adverse reactions from application developers. They're either very good actors or were working on actual applications. An additional space preserving option that Arkady did not mention is limiting the IP alias service to IPv4 addresses. Anyone who really wants IPv6 addresses can get their SM to assign IPv6 compatible GIDs. Of course the flat IPv6 option is far simpler, and probably should be used unless a specific application is identified where those extra 96 bits makes the difference between making the private data be rewritten or left as is. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] RE: iWARP emulation protocol
> -Original Message- > From: Roland Dreier [mailto:[EMAIL PROTECTED] > Sent: Tuesday, October 18, 2005 2:53 PM > To: Kanevsky, Arkady > Cc: Roland Dreier; Yaron Haviv; openib-general@openib.org; > [EMAIL PROTECTED] > Subject: Re: iWARP emulation protocol > > > The proposal doesn't talk about mapping from TCP port numbers into a > 16-bit range of IB service IDs. I think this is necessary. > I agree, that's part of the other proposals > Also, putting the destination address in the REP message doesn't make > sense to me. The destination IP and port number is something that the > initiator of the connection is sending to the destination, not the > other way around. The passive side of the connection (receiver of the > REQ) needs the destination IP as part of the REQ so that it can decide > whether to accept the connection; the active side (sender of the REQ) > knows who it is trying to talk to, so having the address information > in the REP is not useful. Also Agree, REP just needs few fields (ver, capabilities) Yaron ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: iWARP emulation protocol
[closed dat-discussions list snipped from Cc list] I have some comments about the proposal. Unfortunately I can't quote from a PDF file but I'll try to make it clear what I'm talking about. The proposal doesn't talk about mapping from TCP port numbers into a 16-bit range of IB service IDs. I think this is necessary. Also, putting the destination address in the REP message doesn't make sense to me. The destination IP and port number is something that the initiator of the connection is sending to the destination, not the other way around. The passive side of the connection (receiver of the REQ) needs the destination IP as part of the REQ so that it can decide whether to accept the connection; the active side (sender of the REQ) knows who it is trying to talk to, so having the address information in the REP is not useful. As I said above I believe the destination port should be encoded in the service ID, but the destination IP address should be in the REQ message. This consumes 16 more bytes of private data, but I would still like to understand whether there are real applications using 64 bytes of private data, or if this is just a uDAPL spec issue. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: iWARP emulation protocol
Arkady> uDAPL users. 1) http://www.zip.com.au/~akpm/linux/patches/stuff/top-posting.txt 2) Are there real users or is this a generic uDAPL API thing? - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] RE: iWARP emulation protocol
uDAPL users. Arkady Kanevsky email: [EMAIL PROTECTED] Network Appliance phone: 781-768-5395 375 Totten Pond Rd. Fax: 781-895-1195 Waltham, MA 02451-2010 central phone: 781-768-5300 > -Original Message- > From: Roland Dreier [mailto:[EMAIL PROTECTED] > Sent: Tuesday, October 18, 2005 2:19 PM > To: Kanevsky, Arkady > Cc: Yaron Haviv; openib-general@openib.org; > [EMAIL PROTECTED]; [EMAIL PROTECTED] > Subject: Re: iWARP emulation protocol > > > Arkady> The proposed protocol will be used by both kernel and user > Arkady> space Consumers. There are existing Consumers that rely > Arkady> on 64 bytes of private data. > > Which consumers are these? > > - R. > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: iWARP emulation protocol
Arkady> The proposed protocol will be used by both kernel and user Arkady> space Consumers. There are existing Consumers that rely Arkady> on 64 bytes of private data. Which consumers are these? - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Re: iWARP emulation protocol
Hello, I had an email exchange with the iSER folks earlier this summer where I submitted the attached proposal for consideration. It does not contain the local port number because nobody seems to set it to a meaningful value. More food for thought. -David Roland Dreier wrote: Roland> I have a few minor quibbles with this proposal. I think Roland> it would be better to have only the IP version, source and Roland> destination IPs and local in the CM private data. err-- "...and local PORT NUMBER in the CM private data..." - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general --- Begin Message --- I'd like to propose a standard IBTA Hello/HelloAck message carried in the CM private data area and IANA port number to IB service ID mapping for use by IETF RDMA protocols mapped onto IB. I'm hoping that this extension will enable iSER, NFS, and perhaps future iWARP ULPs to transmit IP address information that is needed for authentication and advertise well-known service locations. The proposal consists of the following changes to version 1.2 of volume 1 IBTA specification: 1) When the Service ID is in the IBTA assigned range as described in section A3.2.3.1 where byte 6 is set to 0x02, the CM REQ packet will contain the following Hello message starting at offset 140 (start of private data area): bit 0 1 2 3 byte 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 0-3 | MajV | MinV | IPV | Cap | Reserved | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 4-7 | Src IP (127-96) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 8-11 | Src IP ( 95-64) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 12-15 | Src IP ( 63-32) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 16-19 | Src IP ( 31-00) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 20-23 | Dst IP (127-96) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 24-27 | Dst IP ( 95-64) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 28-31 | Dst IP ( 63-32) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 32-35 | Dst IP ( 31-00) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ where: MajV (4 bits) - The Major Protocol version number. The value must be set to 1. The accepting peer shall reject the connection if MajV in the Hello message does not match its local value. MinV (4 bits) - The Minor Protocol version number. The value must be set to 1. The accepting peer shall not reject the connection, solely on the basis that MinV of the Hello message does not match its local value. This enables future protocol extensions which are upwardly compatible. IPV (4 bits) - The Internet Protocol version number of the end-point address field (fields Src IP and Dst IP). If IPV = 0x4, the IP addresses are IP version 4 format (32 bit addresses). If IPV = 0x6, then both IP addresses are IP version 6 format (128 bit addresses). All other IPV values are reserved. The accepting peer shall reject the connection if IPV of the Hello message has a value other than 0x4 or 0x6. Cap (4 bits) - The Capabilities field conveys the connecting peer’s capabilities. The following capabilities are defined: Bit Position Capability Description 0 INVALIDATE_CAP Supports incoming Send w/Invalidate opcode 1 VABTO_CAP Supports incoming ZBVA Messages 2-3 reserved transmitted as zero and not checked at receiver where: The connecting peer shall set the INVALIDATE_CAP bit in the Cap field in Hello Message only if it can support incoming invalidate messages. If the accepting peer does not support Send w/Invalidate, it will ignore the Invalidate Capability advertisement. The connecting peer shall set the ZBVA_CAP bit in the Cap field in Hello Message only if it can support Zero based Virtual Addresses (ZBVA). If the accepting peer does not support ZB
[openib-general] Re: iWARP emulation protocol
Roland> I have a few minor quibbles with this proposal. I think Roland> it would be better to have only the IP version, source and Roland> destination IPs and local in the CM private data. err-- "...and local PORT NUMBER in the CM private data..." - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
