Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux
At 10:25 AM 5/27/2005, Sukanta ganguly wrote: Mike, I am not sure I do understand what your were trying to communicate. Let me try and decode this. My basic point was to respond to Venkata's response about complete offload without any interaction with the host system. I disagree with that in its totality as I think there are dependencies and that needs to be specified in a formal manner, i.e. in the specs, so that we do not have multiple proprietary interafaces which change and application users have to change this consumption based on individual implementations. And you just brought up the reasons why what I was saying seemed to be justfied. Did I read that correctly? You understand correctly. We need defined interface points where other subsystems can choose to interact with the RDMA infrastructure. Further, there is a set of issues that have yet to be addressed, e.g. that go to how a customer uses / manages the attached fabrics and how their tool chain is impacted or not. Mike Thanks SG --- Michael Krause <[EMAIL PROTECTED]> wrote: > At 06:40 AM 5/27/2005, Sukanta ganguly wrote: > >Venkata, > > How will that work? If the RNIC offloads RDMA > and > >TCP completely from the Operating System and does > not > >share any state information then the application > >running on the host will never be in the position > to > >utilize the socket interface to use the > communication > >logic to send and receive data between the remote > node > >and itself. Some information needs to be shared. > How > >much of it and what exactly needs to be shared is > the > >question. > > Ok. It all depends upon what level of integration / > interaction a TOE and > thus a RNIC will have with the host network stack. > For example, if a > customer wants to have TCP and IP stats kept for the > off-loaded stack even > if it is just being using for RDMA, then there needs > to be a method defined > to consolidate these stats back into the host > network stack tool > chain. Similarly, if one wants to maintain a single > routing table to > manage, etc. on the host, then the RNIC needs to > access / update that > information accordingly. One can progress through > other aspects of > integration, e.g. connection management, security > interactions (e.g. DOS > protection), and so forth. What is exposed again > depends upon the level of > integration and how customers want to manage their > services. This problem > also exists for IB but most people have not thought > about this from a > customer perspective and how to integrate the IB > semantics into the way > customers manage their infrastructures, do billing, > etc. For some > environments, they simply do not care but if IB is > to be used in the > enterprise space, then some thought will be required > here since most IT > don't see anything as being "free" or self-managed. > > Again, Sockets is an application API and not how one > communicates to a TOE > or RDMA component. The RNIC PI has been proposed as > an interface to the > RDMA functionality. The PI supports all of the > iWARP and IB v 1.2 verbs. > > Mike > > > >Thanks > >SG > > > >--- Venkata Jagana <[EMAIL PROTECTED]> wrote: > > > > > > > > > > > > > > > > > > > > > [EMAIL PROTECTED] > wrote on > > > 05/25/2005 09:47:00 > > > PM: > > > > > > > Venkata, > > > > Interesting coincidence: I was talking with > > > someone (at HP) today > > > > who knows substantially more than I do about > > > RNICs. > > > > They indicated RNICs need to manage TCP state > on > > > the card from userspace. > > > > I suspect that's only possible through a > private > > > interface > > > > (e.g. ioctl() or /proc) or the non-existant > (in > > > kernel.org) > > > > TOE implementation. Is this correct? > > > > > > > > > > Not correct. > > > > > > Since RNICs are offloaded adapters with RDMA > > > protocols layered on > > > top of TCP stack, they do maintain the TCP state > > > internally but > > > it does not expose to the host. RNIC expose only > > > RNIC Verbs interface > > > to the host bot not TOE interface. > > > > > > Thanks > > > Venkat > > > > > > > > > > > hth, > > > > grant > > > > > > > > > > > > > > > > >--- > > > > SF.Net email is sponsored by: GoToMeeting - > the > > > easiest way to > > > collaborate > > > > online with coworkers and clients while > avoiding > > > the high cost of travel > > > and > > > > communications. There is no equipment to buy > and > > > you can meet as often as > > > > you want. Try it > > > > >free. http://ads.osdn.com/?ad_id=7402&alloc_id=16135&op=click > > > > > ___ > > > > Rdma-developers mailing list > > > > [EMAIL PROTECTED] > > > > > https://lists.sourceforge.net/lists/listinfo/rdma-developers > > > >__ > >Do You Yahoo!? > >Tired of spam? Yahoo! Mail has the best spam > protection around > > http://mail.yahoo.com > > > >
RE: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux
[EMAIL PROTECTED] wrote on 05/27/2005 10:40:20 AM: > > On Fri, May 27, 2005 at 09:58:44AM -0700, Caitlin Bestler wrote: > > > > I understand most of what you wrote but am still missing one bit: > > > > How is the RNIC told what the peer IP is it should > > communicate with? > > > > > > The TCP Connection is already established before the RNIC takes > > > control of the connection. Therefore the entire TCP state > > is already > > > established (source/dest IP Address/Port, negotiated options, EMSS, > > > local options, etc.). > > > > Would "shares the connection" be a better description? > > > > No, it does not "share the connection". After conversion the > RDMA stack is in full control of the TCP layer. The host cannot > modify anything about the TCP layer, or use its services. > For example the host cannot enable Nagle on the TCP connection > after it has been turned over to the RNIC. It definetly cannot > send or receive streaming data. Because of the nature of the > MPA protocol, a TCP connection that has been enabled for MPA > can *never* revert to non-MPA mode. It can only be used in MPA > mode or terminated. > Agree that it is never being shared but once the connection has been turned over to the RNIC, would it ever be possible to hand it back to the host or host taking control from the RNIC adapter? Might be helpful for failover situations. Thanks Venkat > It's the L2 and L3 layer data that is shared (or synchronized) > between the host and the RNIC. > > > I think I get the concepts at least now...this follows what I > > was suggesting earlier about having an RNIC support both NAPI > > (New API for NICs) (code under drivers/net) and openib.org > > Gen2 stack (code in drivers/infiniband). > > > > thanks, > > grant > > > > > --- > This SF.Net email is sponsored by Yahoo. > Introducing Yahoo! Search Developer Network - Create apps using Yahoo! > Search APIs Find out how you can build Yahoo! directly into your own > Applications - visit http://developer.yahoo.net/?fr=offad-ysdn-ostg-q22005 > ___ > Rdma-developers mailing list > [EMAIL PROTECTED] > https://lists.sourceforge.net/lists/listinfo/rdma-developers ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux
Venkata, I understand what you were saying now. Thanks SG --- Venkata Jagana <[EMAIL PROTECTED]> wrote: > > > > > > [EMAIL PROTECTED] wrote on > 05/27/2005 10:25:36 > AM: > > > Mike, > >I am not sure I do understand what your were > trying > > to communicate. Let me try and decode this. My > basic > > point was to respond to Venkata's response about > > complete offload without any interaction with the > host > > system. I disagree with that in its totality as I > > think there are dependencies and that needs to be > > specified in a formal manner, i.e. in the specs, > so > > that we do not have multiple proprietary > interafaces > > which change and application users have to change > this > > consumption based on individual implementations. > >And you just brought up the reasons why what I > was > > saying seemed to be justfied. Did I read that > > correctly? > > > > Thanks > > SG > > > > Hello Sukanta, > > Yes, there will be interactions with the host system > for an RNIC > but through an interface something similar to > RNIC-PI. > > The clarification that I was trying to make is that > from an RNIC > perspective, it > doesn't have to expose any ToE interface other than > the fact that adapters > might be providing > multiple services including ToE, RDMA, iSCSI but > that's outside the scope > of an > RNIC interface. > > Thanks > Venkat __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux
[EMAIL PROTECTED] wrote on 05/27/2005 10:25:36 AM: > Mike, > I am not sure I do understand what your were trying > to communicate. Let me try and decode this. My basic > point was to respond to Venkata's response about > complete offload without any interaction with the host > system. I disagree with that in its totality as I > think there are dependencies and that needs to be > specified in a formal manner, i.e. in the specs, so > that we do not have multiple proprietary interafaces > which change and application users have to change this > consumption based on individual implementations. > And you just brought up the reasons why what I was > saying seemed to be justfied. Did I read that > correctly? > > Thanks > SG > Hello Sukanta, Yes, there will be interactions with the host system for an RNIC but through an interface something similar to RNIC-PI. The clarification that I was trying to make is that from an RNIC perspective, it doesn't have to expose any ToE interface other than the fact that adapters might be providing multiple services including ToE, RDMA, iSCSI but that's outside the scope of an RNIC interface. Thanks Venkat___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux
> -Original Message- > From: Grant Grundler [mailto:[EMAIL PROTECTED] > Sent: Friday, May 27, 2005 10:10 AM > To: Caitlin Bestler > Cc: Sukanta ganguly; [EMAIL PROTECTED]; > openib-general@openib.org > Subject: Re: [Rdma-developers] Re: [openib-general] OpenIB > and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux > > On Fri, May 27, 2005 at 09:58:44AM -0700, Caitlin Bestler wrote: > > > I understand most of what you wrote but am still missing one bit: > > > How is the RNIC told what the peer IP is it should > communicate with? > > > > The TCP Connection is already established before the RNIC takes > > control of the connection. Therefore the entire TCP state > is already > > established (source/dest IP Address/Port, negotiated options, EMSS, > > local options, etc.). > > Would "shares the connection" be a better description? > No, it does not "share the connection". After conversion the RDMA stack is in full control of the TCP layer. The host cannot modify anything about the TCP layer, or use its services. For example the host cannot enable Nagle on the TCP connection after it has been turned over to the RNIC. It definetly cannot send or receive streaming data. Because of the nature of the MPA protocol, a TCP connection that has been enabled for MPA can *never* revert to non-MPA mode. It can only be used in MPA mode or terminated. It's the L2 and L3 layer data that is shared (or synchronized) between the host and the RNIC. > I think I get the concepts at least now...this follows what I > was suggesting earlier about having an RNIC support both NAPI > (New API for NICs) (code under drivers/net) and openib.org > Gen2 stack (code in drivers/infiniband). > > thanks, > grant > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux
--- Caitlin Bestler <[EMAIL PROTECTED]> wrote: > > > > > > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] > On Behalf Of > Michael Krause > Sent: Friday, May 27, 2005 7:25 AM > To: Sukanta ganguly > Cc: openib-general@openib.org; > [EMAIL PROTECTED] > Subject: Re: [Rdma-developers] Re: [openib-general] > OpenIB and > OpenRDMA: Convergence on common RDMA APIs and ULPs > for Linux > > > At 06:40 AM 5/27/2005, Sukanta ganguly wrote: > > > Venkata, > How will that work? If the RNIC offloads RDMA > and > TCP completely from the Operating System and does > not > share any state information then the application > running on the host will never be in the position > to > utilize the socket interface to use the > communication > logic to send and receive data between the remote > node > and itself. Some information needs to be shared. > How > much of it and what exactly needs to be shared is > the > question. > > > Ok. It all depends upon what level of integration > / interaction > a TOE and thus a RNIC will have with the host > network stack. For > example, if a customer wants to have TCP and IP > stats kept for the > off-loaded stack even if it is just being using for > RDMA, then there > needs to be a method defined to consolidate these > stats back into the > host network stack tool chain. Similarly, if one > wants to maintain a > single routing table to manage, etc. on the host, > then the RNIC needs to > access / update that information accordingly. One > can progress through > other aspects of integration, e.g. connection > management, security > interactions (e.g. DOS protection), and so forth. > What is exposed again > depends upon the level of integration and how > customers want to manage > their services. This problem also exists for IB but > most people have > not thought about this from a customer perspective > and how to integrate > the IB semantics into the way customers manage their > infrastructures, do > billing, etc. For some environments, they simply do > not care but if IB > is to be used in the enterprise space, then some > thought will be > required here since most IT don't see anything as > being "free" or > self-managed. > > Again, Sockets is an application API and not how > one > communicates to a TOE or RDMA component. The RNIC > PI has been proposed > as an interface to the RDMA functionality. The PI > supports all of the > iWARP and IB v 1.2 verbs. > > Mike > > > > I'd like to add that RNIC-PI is planning on > explicitly defining some of > these "obvious" dependencies > between the RDMA stack and the primary IP stack. For > example, the RDMA > stack cannot maintain > any connection in a state that contradicts current > IP stack routing. It > has to adapt or break the connection. > We can't have an RNIC that has its own ARP table > that is not in sync > with the host's ARP table. > > An iWarp RDMA stack gains the benefit of many > pre-existing network > services (such as DNS, ARP > and routing). But that also carries with it the need > to not contradict > those exisiting services. So it is > both a benefit and a restriction -- and a major > divergence from an IB > RDMA stack. > > > This is exactly what I was getting at and seems like the RNIC-PI is already on its way to do that. Thanks SG __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux
> -Original Message- > From: Grant Grundler [mailto:[EMAIL PROTECTED] > Sent: Friday, May 27, 2005 10:06 AM > To: Caitlin Bestler > Cc: Michael Krause; Sukanta ganguly; > [EMAIL PROTECTED]; openib-general@openib.org > Subject: Re: [Rdma-developers] Re: [openib-general] OpenIB > and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux > > On Fri, May 27, 2005 at 09:45:06AM -0700, Caitlin Bestler wrote: > > I'd like to add that RNIC-PI is planning on explicitly > defining some > > of these "obvious" dependencies between the RDMA stack and > the primary > > IP stack. For example, the RDMA stack cannot maintain any > connection > > in a state that contradicts current IP stack routing. It > has to adapt > > or break the connection. > > That's what I was just thinking. Could RNIC driver support > both existing linux NIC interfaces (e.g. use > ifconfig/ethtool) *and* openib RDMA interface? > > Essentially that's what openib.org does today but uses > ib_ipoib driver to support TCP/IP communication. > > Ie use an AF_INET socket to setup an RDMA connection like the > rdma_lat.c does: > https://openib.org/svn/gen2/trunk/src/userspace/perftest/ > > Of course, I'm "blissfully ignorant" of how ugly that might > be in real world implementation of RNIC driver. Seems simple > in concept at least. > For time-to-market reasons, and to avoid having to go hacking deeply into the kernel from a driver, many or most existing implementations require or at least prefer using TOE sockets. It is much easier to extract the TCP state from a socket that you already control than from the host stack. But that is exactly the sort of limitation that integrating with the kernel should eliminate. Given a stable interface to "get TCP state" from a host controlled connection I can't imagine why an RNIC vendor would not eagerly embrace it. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Michael KrauseSent: Friday, May 27, 2005 9:51 AMTo: Bernard Metzler; Sukanta gangulyCc: [EMAIL PROTECTED]; openib-general@openib.org; [EMAIL PROTECTED]Subject: Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux At 08:05 AM 5/27/2005, Bernard Metzler wrote: Sukanta, without touching any TOE issues (this question is about RDMA, right?), after transforming a TCP connection into RDMA mode and using an RDMA API, the socket file descriptor is not longer to be used for communication. Whether one uses a Socket FD or not is an application issue. A QP has a handle identifier which may be mapped to a FD or may be used directly by an application. There is no requirement that anything flow through the Sockets API - that is a choice for the application to make. True. The current RNIC-PI spec is deliberately vague about what an "LLP Handle" is, other than that it must include a Socket FD. It may be possible for it to include some sort of "connection request handle" for a pending connection on the offload device, but you would have to ensure that this "connection requestion handle" was reviewed by the Host IP stack so that it could forbid connections that contradict the host's IP firewall policies. In fact, on some implementations the stream socket resources will even get released. That is, if the socket was the direct consumer of the TCP stream, then now it is RDMAP/DDP/MPA. RDMA APIs such as IT-API defining a specific call to convert a socket based connection into RDMA mode (e.g., it_socket_convert()). Other consumers may directly start via an RDMA API, never opening a consumer controlled socket. Correct. There may never be a *consumer* controlled socket, but there will be a *host* controlled socket used by the Connection Manager and subject to normal host firewall controls. It is in fact highly desirable from both a security and performance basis for the streaming mode TCP connection to never be exposed to the application. Only specialized applications, such as iSER, require this capability. So, in RDMA mode, communication will happen via the RDMA API. At this stage, the kernel still have to keep completely in its hands the synchronisation of state information related to that offloaded connection(s) with the host stack (it would have to protect the local port used by the offloaded connection for example, others are routing, ARP, SNMP...), but it is not involved at the data path. This is true only when the host and the RNIC share a common IP address. If they are separate subnets, then there is no need to coordinate sans some of the reasons I noted in an earlier e-mail. I think there is great value and customer need to have an infrastructure that can coordinate information and enable customers to use the existing tool chains to understand what is going on in a given device or endnode. Good point. The key is that the RNIC needs to do co-ordinate all of these with the IP stack that owns the IP address. If it is that owner, it must co-ordinate with itself. (Actually that's a clause that needs to be made more explicit in the RNIC-PI spec, which was focused on the shared IP scenario). But when the RNIC is "sub-leasing" the IP it needs to co-ordinate its action with the owner of the IP address. With respect to the kernel based TCP stack, what is not needed is a hack into the stack and scatter/gather state information of the live TCP connection between kernel and RNIC, but to find one clean interface to transfer state information out of that stack and to the RNIC. At a minimum, one needs:- A host network stack get state call to extract connection state and quiesce the port from the host's perspective.- A host network stack set state call to populate the host structures with the associated state and enable the port- A RNIC get state call to extract all transport and RMDA context- A RNIC set state call to populate all transport and RDMA context An RNIC "get state" is nice, but not essential. It enables transfer of an RDMA state to another RNIC for failover purposes. But in terms of minimal control, the ability to kill an RDMA controlled connection is sufficient. Additionally, any transfer of RDMA Context will be limited to transfer between like models for some time, and even then it is not likely to be a standard feature. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux
I agree. SG --- Bernard Metzler <[EMAIL PROTECTED]> wrote: > Sukanta, > > without touching any TOE issues (this question is > about RDMA, right?), > after > transforming a TCP connection into RDMA mode and > using an RDMA API, > the socket file descriptor is not longer to be used > for communication. > In fact, on some implementations the stream socket > resources will > even get released. That is, if the socket was the > direct consumer > of the TCP stream, then now it is RDMAP/DDP/MPA. > RDMA APIs such as IT-API defining a specific call to > convert a socket > based > connection into RDMA mode (e.g., > it_socket_convert()). > Other consumers may directly start via an RDMA API, > never > opening a consumer controlled socket. > > So, in RDMA mode, communication will happen via the > RDMA API. At this > stage, the kernel still have to keep completely in > its hands the > synchronisation of state information related to that > offloaded > connection(s) with the host stack (it would have to > protect the > local port used by the offloaded connection for > example, others > are routing, ARP, SNMP...), but it is not involved > at the data path. > > With respect to the kernel based TCP stack, what is > not needed is > a hack into the stack and scatter/gather state > information of the live > TCP connection between kernel and RNIC, but to find > one clean interface > to transfer state information out of that stack and > to the RNIC. > > > With limited benefit, one could of course also > implement native > sockets over RDMA, where an in-kernel midlayer on > top of kernel > RDMA Verbs is doing the translation between send(), > receive() to > post_send, post_receive. But usage of 'true RDMA' > operations > like RDMA READ or WRITE might be limited, and I > don't see much value > for the user here. One variety of this approach with > less limited > access to RMDA benefits might be sockets with > extended RDMA > semantics. > > Bernard. > > [EMAIL PROTECTED] wrote on > 27.05.2005 15:40:43: > > > Venkata, > >How will that work? If the RNIC offloads RDMA > and > > TCP completely from the Operating System and does > not > > share any state information then the application > > running on the host will never be in the position > to > > utilize the socket interface to use the > communication > > logic to send and receive data between the remote > node > > and itself. Some information needs to be shared. > How > > much of it and what exactly needs to be shared is > the > > question. > > > > Thanks > > SG > > > > --- Venkata Jagana <[EMAIL PROTECTED]> wrote: > > > > > > > > > > > > > > > > > > > > > [EMAIL PROTECTED] > wrote on > > > 05/25/2005 09:47:00 > > > PM: > > > > > > > Venkata, > > > > Interesting coincidence: I was talking with > > > someone (at HP) today > > > > who knows substantially more than I do about > > > RNICs. > > > > They indicated RNICs need to manage TCP state > on > > > the card from userspace. > > > > I suspect that's only possible through a > private > > > interface > > > > (e.g. ioctl() or /proc) or the non-existant > (in > > > kernel.org) > > > > TOE implementation. Is this correct? > > > > > > > > > > Not correct. > > > > > > Since RNICs are offloaded adapters with RDMA > > > protocols layered on > > > top of TCP stack, they do maintain the TCP state > > > internally but > > > it does not expose to the host. RNIC expose only > > > RNIC Verbs interface > > > to the host bot not TOE interface. > > > > > > Thanks > > > Venkat > > > > > > > > > > > hth, > > > > grant > > > > > > > > > > > > > > > > > > --- > > > > SF.Net email is sponsored by: GoToMeeting - > the > > > easiest way to > > > collaborate > > > > online with coworkers and clients while > avoiding > > > the high cost of travel > > > and > > > > communications. There is no equipment to buy > and > > > you can meet as often as > > > > you want. Try it > > > > > > free.http://ads.osdn.com/?ad_id=7402&alloc_id=16135&op=click > > > > > ___ > > > > Rdma-developers mailing list > > > > [EMAIL PROTECTED] > > > > > https://lists.sourceforge.net/lists/listinfo/rdma-developers > > > > __ > > Do You Yahoo!? > > Tired of spam? Yahoo! Mail has the best spam > protection around > > http://mail.yahoo.com > > > > > > > --- > > This SF.Net email is sponsored by Yahoo. > > Introducing Yahoo! Search Developer Network - > Create apps using Yahoo! > > Search APIs Find out how you can build Yahoo! > directly into your own > > Applications - visit > http://developer.yahoo.net/?fr=offad-ysdn-ostg-q22005 > > ___ > > Rdma-developers mailing list > > [EMAIL PROTECTED] > > > https://lists.sourceforge.net/lists/listinfo/rdma-developers > __ Do You
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux
Mike, I am not sure I do understand what your were trying to communicate. Let me try and decode this. My basic point was to respond to Venkata's response about complete offload without any interaction with the host system. I disagree with that in its totality as I think there are dependencies and that needs to be specified in a formal manner, i.e. in the specs, so that we do not have multiple proprietary interafaces which change and application users have to change this consumption based on individual implementations. And you just brought up the reasons why what I was saying seemed to be justfied. Did I read that correctly? Thanks SG --- Michael Krause <[EMAIL PROTECTED]> wrote: > At 06:40 AM 5/27/2005, Sukanta ganguly wrote: > >Venkata, > >How will that work? If the RNIC offloads RDMA > and > >TCP completely from the Operating System and does > not > >share any state information then the application > >running on the host will never be in the position > to > >utilize the socket interface to use the > communication > >logic to send and receive data between the remote > node > >and itself. Some information needs to be shared. > How > >much of it and what exactly needs to be shared is > the > >question. > > Ok. It all depends upon what level of integration / > interaction a TOE and > thus a RNIC will have with the host network stack. > For example, if a > customer wants to have TCP and IP stats kept for the > off-loaded stack even > if it is just being using for RDMA, then there needs > to be a method defined > to consolidate these stats back into the host > network stack tool > chain. Similarly, if one wants to maintain a single > routing table to > manage, etc. on the host, then the RNIC needs to > access / update that > information accordingly. One can progress through > other aspects of > integration, e.g. connection management, security > interactions (e.g. DOS > protection), and so forth. What is exposed again > depends upon the level of > integration and how customers want to manage their > services. This problem > also exists for IB but most people have not thought > about this from a > customer perspective and how to integrate the IB > semantics into the way > customers manage their infrastructures, do billing, > etc. For some > environments, they simply do not care but if IB is > to be used in the > enterprise space, then some thought will be required > here since most IT > don't see anything as being "free" or self-managed. > > Again, Sockets is an application API and not how one > communicates to a TOE > or RDMA component. The RNIC PI has been proposed as > an interface to the > RDMA functionality. The PI supports all of the > iWARP and IB v 1.2 verbs. > > Mike > > > >Thanks > >SG > > > >--- Venkata Jagana <[EMAIL PROTECTED]> wrote: > > > > > > > > > > > > > > > > > > > > > [EMAIL PROTECTED] > wrote on > > > 05/25/2005 09:47:00 > > > PM: > > > > > > > Venkata, > > > > Interesting coincidence: I was talking with > > > someone (at HP) today > > > > who knows substantially more than I do about > > > RNICs. > > > > They indicated RNICs need to manage TCP state > on > > > the card from userspace. > > > > I suspect that's only possible through a > private > > > interface > > > > (e.g. ioctl() or /proc) or the non-existant > (in > > > kernel.org) > > > > TOE implementation. Is this correct? > > > > > > > > > > Not correct. > > > > > > Since RNICs are offloaded adapters with RDMA > > > protocols layered on > > > top of TCP stack, they do maintain the TCP state > > > internally but > > > it does not expose to the host. RNIC expose only > > > RNIC Verbs interface > > > to the host bot not TOE interface. > > > > > > Thanks > > > Venkat > > > > > > > > > > > hth, > > > > grant > > > > > > > > > > > > > > > > >--- > > > > SF.Net email is sponsored by: GoToMeeting - > the > > > easiest way to > > > collaborate > > > > online with coworkers and clients while > avoiding > > > the high cost of travel > > > and > > > > communications. There is no equipment to buy > and > > > you can meet as often as > > > > you want. Try it > > > > >free.http://ads.osdn.com/?ad_id=7402&alloc_id=16135&op=click > > > > > ___ > > > > Rdma-developers mailing list > > > > [EMAIL PROTECTED] > > > > > https://lists.sourceforge.net/lists/listinfo/rdma-developers > > > >__ > >Do You Yahoo!? > >Tired of spam? Yahoo! Mail has the best spam > protection around > >http://mail.yahoo.com > > > > > >--- > >This SF.Net email is sponsored by Yahoo. > >Introducing Yahoo! Search Developer Network - > Create apps using Yahoo! > >Search APIs Find out how you can build Yahoo! > directly into your own > >Applications - visit > http://developer.yahoo.net/?fr=offad-ysdn-ostg-q22005 > >_
RE: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux
> -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On > Behalf Of Talpey, Thomas > Sent: Thursday, May 26, 2005 5:28 AM > To: Venkata Jagana > Cc: [EMAIL PROTECTED]; openib-general@openib.org > Subject: [Rdma-developers] Re: [openib-general] OpenIB and > OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux > > At 02:20 AM 5/26/2005, Venkata Jagana wrote: > >I would like to really understand the technical reasons why > you say RNIC-PI is irrelevant to Linux kernel. > >RNIC-PI is developed to support not only the RNICs but it is > also IB compatible. > > I'm not Roland, but my belief is that until RNIC-PI exists as > working code on a reference implementation, it is in fact not > yet relevant to Linux. Perhaps you can outline the schedule > to get there? > That is a slightly circular argument. The primary reason that work to implement DAPL and IT-API over RNIC-PI is not already in progress is the pressure to come up with a "one stack" solution. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux
On Fri, May 27, 2005 at 09:58:44AM -0700, Caitlin Bestler wrote: > > I understand most of what you wrote but am still missing one bit: > > How is the RNIC told what the peer IP is it should communicate with? > > The TCP Connection is already established before the RNIC takes > control of the connection. Therefore the entire TCP state is already > established (source/dest IP Address/Port, negotiated options, EMSS, > local options, etc.). Would "shares the connection" be a better description? I think I get the concepts at least now...this follows what I was suggesting earlier about having an RNIC support both NAPI (New API for NICs) (code under drivers/net) and openib.org Gen2 stack (code in drivers/infiniband). thanks, grant ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux
On Fri, May 27, 2005 at 09:45:06AM -0700, Caitlin Bestler wrote: > I'd like to add that RNIC-PI is planning on explicitly defining some of > these "obvious" dependencies > between the RDMA stack and the primary IP stack. For example, the RDMA > stack cannot maintain > any connection in a state that contradicts current IP stack routing. It > has to adapt or break the connection. That's what I was just thinking. Could RNIC driver support both existing linux NIC interfaces (e.g. use ifconfig/ethtool) *and* openib RDMA interface? Essentially that's what openib.org does today but uses ib_ipoib driver to support TCP/IP communication. Ie use an AF_INET socket to setup an RDMA connection like the rdma_lat.c does: https://openib.org/svn/gen2/trunk/src/userspace/perftest/ Of course, I'm "blissfully ignorant" of how ugly that might be in real world implementation of RNIC driver. Seems simple in concept at least. > We can't have an RNIC that has its own ARP table that is not in sync > with the host's ARP table. Yes, the RDMA and NIC parts of the driver would have to be aware of each other. grant > An iWarp RDMA stack gains the benefit of many pre-existing network > services (such as DNS, ARP > and routing). But that also carries with it the need to not contradict > those exisiting services. So it is > both a benefit and a restriction -- and a major divergence from an IB > RDMA stack. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux
> -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On > Behalf Of Grant Grundler > Sent: Friday, May 27, 2005 9:29 AM > To: Michael Krause > Cc: Sukanta ganguly; [EMAIL PROTECTED]; > openib-general@openib.org > Subject: Re: [Rdma-developers] Re: [openib-general] OpenIB > and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux > > On Fri, May 27, 2005 at 07:24:44AM -0700, Michael Krause wrote: > ... > > Again, Sockets is an application API and not how one > communicates to a > > TOE or RDMA component. > > Mike, > What address family is used to open a socket over iWARP? AF_INET? > Or something else? > > I understand most of what you wrote but am still missing one bit: > How is the RNIC told what the peer IP is it should communicate with? > The TCP Connection is already established before the RNIC takes control of the connection. Therefore the entire TCP state is already established (source/dest IP Address/Port, negotiated options, EMSS, local options, etc.). ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux
At 08:05 AM 5/27/2005, Bernard Metzler wrote: Sukanta, without touching any TOE issues (this question is about RDMA, right?), after transforming a TCP connection into RDMA mode and using an RDMA API, the socket file descriptor is not longer to be used for communication. Whether one uses a Socket FD or not is an application issue. A QP has a handle identifier which may be mapped to a FD or may be used directly by an application. There is no requirement that anything flow through the Sockets API - that is a choice for the application to make. In fact, on some implementations the stream socket resources will even get released. That is, if the socket was the direct consumer of the TCP stream, then now it is RDMAP/DDP/MPA. RDMA APIs such as IT-API defining a specific call to convert a socket based connection into RDMA mode (e.g., it_socket_convert()). Other consumers may directly start via an RDMA API, never opening a consumer controlled socket. Correct. So, in RDMA mode, communication will happen via the RDMA API. At this stage, the kernel still have to keep completely in its hands the synchronisation of state information related to that offloaded connection(s) with the host stack (it would have to protect the local port used by the offloaded connection for example, others are routing, ARP, SNMP...), but it is not involved at the data path. This is true only when the host and the RNIC share a common IP address. If they are separate subnets, then there is no need to coordinate sans some of the reasons I noted in an earlier e-mail. I think there is great value and customer need to have an infrastructure that can coordinate information and enable customers to use the existing tool chains to understand what is going on in a given device or endnode. With respect to the kernel based TCP stack, what is not needed is a hack into the stack and scatter/gather state information of the live TCP connection between kernel and RNIC, but to find one clean interface to transfer state information out of that stack and to the RNIC. At a minimum, one needs: - A host network stack get state call to extract connection state and quiesce the port from the host's perspective. - A host network stack set state call to populate the host structures with the associated state and enable the port - A RNIC get state call to extract all transport and RMDA context - A RNIC set state call to populate all transport and RDMA context These calls should be "standardized" so that both sides of the infrastructure will be able to utilize without having to hack anything. With limited benefit, one could of course also implement native sockets over RDMA, where an in-kernel midlayer on top of kernel RDMA Verbs is doing the translation between send(), receive() to post_send, post_receive. But usage of 'true RDMA' operations like RDMA READ or WRITE might be limited, and I don't see much value for the user here. One variety of this approach with less limited access to RMDA benefits might be sockets with extended RDMA semantics. Sockets with RDMA extensions was proposed a couple of years back by some of us. There would be quite a bit of benefit as most developers know how to code to Sockets and the RDMA extensions are relatively simple when you think about it. It would also eliminate the need to use SDP by placing the RDMA knowledge requirement on the application. SDP has benefits for enabling existing Sockets applications to operate transparently over RDMA but the explicit use of RDMA extensions I think would provide greater benefit in the end. While it is important to provide legacy investment protection, real innovation will not occur until people start developing technology that enables a smarter customer. Simplification and performance also come with enabling a smarter customer. Mike Bernard. [EMAIL PROTECTED] wrote on 27.05.2005 15:40:43: > Venkata, > How will that work? If the RNIC offloads RDMA and > TCP completely from the Operating System and does not > share any state information then the application > running on the host will never be in the position to > utilize the socket interface to use the communication > logic to send and receive data between the remote node > and itself. Some information needs to be shared. How > much of it and what exactly needs to be shared is the > question. > > Thanks > SG > > --- Venkata Jagana <[EMAIL PROTECTED]> wrote: > > > > > > > > > > > > > > [EMAIL PROTECTED] wrote on > > 05/25/2005 09:47:00 > > PM: > > > > > Venkata, > > > Interesting coincidence: I was talking with > > someone (at HP) today > > > who knows substantially more than I do about > > RNICs. > > > They indicated RNICs need to manage TCP state on > > the card from userspace. > > > I suspect that's only possible through a private > > interface > > > (e.g. ioctl() or /proc) or the non-existant (in > > kernel.org) > > > TOE implementation. Is this correct? > > > > > > > Not correct. > > > > Since RNICs are offl
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux
At 09:29 AM 5/27/2005, Grant Grundler wrote: On Fri, May 27, 2005 at 07:24:44AM -0700, Michael Krause wrote: ... > Again, Sockets is an application API and not how one communicates to a TOE > or RDMA component. Mike, What address family is used to open a socket over iWARP? AF_INET? Or something else? TCP = AF_INET. Address family != Sockets. Sockets is an API that can operate over multiple address families. An application can be coded to Sockets, IT API, DAPL, or a verbs interface like the RNIC PI. It is a matter of choice as well as what is trying to be accomplished. The RNIC PI is an acceptable interface for any RDMA-focused ULP. There are pros / cons to using such a verbs interface directly but I do not believe any one can deny that a general-purpose verbs API is a good thing at the end of the day as it works for the volume verbs definition. Whether one applies further hardware semantics abstraction such as IT API / DAPL should be a choice for the individual subsystem as there is no single right answer across all subsystems. Attempting to force fit isn't practical. I understand most of what you wrote but am still missing one bit: How is the RNIC told what the peer IP is it should communicate with? The destination address (IB GID or IP) is derived from the CM services. This is where the two interconnects differ in what is required to physical inject a packet on the wire. This is why I call it out as separate from the verbs interface and something that could be abstracted to some extent but at the end of the day, really requires the subsystem to understand the underlying fabric type to make some intelligent choices. Given this effort is still nascent, most of the issues beyond basic bootstrap have not really been discussed as yet. > The RNIC PI has been proposed as an interface to the > RDMA functionality. The PI supports all of the iWARP and IB v 1.2 verbs. That's good. Folks from RDMA consortium will have to look at openib implementations and see whats missing/wrong. Then submit proposals to fill in the gaps. I'm obviously not the first one to say this. There are two open source efforts. The question is whether to move to a single effort (I tried to get this to occur before OpenIB was formally launched but it seem to fall on deaf ears for TTM marketing purposes) or whether to just coordinate on some of the basics. My preference remains that the efforts remained strictly focused on the RDMA infrastructure and interconnect-specific components and leave the ULP / services as separate efforts who will make their own decisions on how best to interface with the RDMA infrastructure. I expect most of the principals involved with openib.org do NOT have time to browse through RNIC PI at this point. They are struggling to get openib.org filled in sufficiently so it can go into a commercial distro (RH/SuSE primarily). Hence, why OpenRDMA needs to get source being developed to enable the RNIC community. If people find value in the work, then people can look at finding the right solution for both IB and iWARP when it makes sense. Revenue for them comes from selling IB equipment. Having openib.org code in kernel.org is a key enabler for getting into commercial distros. I expect the same is true for RNIC vendors as well. RNIC Vendors (and related switch Vendors) will have to decide which path is the right one for them to get the support into kernel.org. Several openib.org people have suggested one (like I have). RNIC folks need to listen and decide if the advice is good or not. If RNIC folks think they know better, then please take another look at where openib.org is today and where rdmaconsortium is. I'm certain openib.org would be dead now if policies and direction changes had not made last year as demanded by several key linux developers and "users" (Gov Labs). Understood. Mike ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux
With respect to the kernel based TCP stack, what is not needed is a hack into the stack and scatter/gather state information of the live TCP connection between kernel and RNIC, but to find one clean interface to transfer state information out of that stack and to the RNIC. Agreed that is an OS-specific service that should not be implemented in a device dependent way by each hardware vendor. But the information extracted from the kernel IP stack cannot be encoded as though it came from an InfiniBand CM. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Michael KrauseSent: Friday, May 27, 2005 7:25 AMTo: Sukanta gangulyCc: openib-general@openib.org; [EMAIL PROTECTED]Subject: Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux At 06:40 AM 5/27/2005, Sukanta ganguly wrote: Venkata, How will that work? If the RNIC offloads RDMA andTCP completely from the Operating System and does notshare any state information then the applicationrunning on the host will never be in the position toutilize the socket interface to use the communicationlogic to send and receive data between the remote nodeand itself. Some information needs to be shared. Howmuch of it and what exactly needs to be shared is thequestion. Ok. It all depends upon what level of integration / interaction a TOE and thus a RNIC will have with the host network stack. For example, if a customer wants to have TCP and IP stats kept for the off-loaded stack even if it is just being using for RDMA, then there needs to be a method defined to consolidate these stats back into the host network stack tool chain. Similarly, if one wants to maintain a single routing table to manage, etc. on the host, then the RNIC needs to access / update that information accordingly. One can progress through other aspects of integration, e.g. connection management, security interactions (e.g. DOS protection), and so forth. What is exposed again depends upon the level of integration and how customers want to manage their services. This problem also exists for IB but most people have not thought about this from a customer perspective and how to integrate the IB semantics into the way customers manage their infrastructures, do billing, etc. For some environments, they simply do not care but if IB is to be used in the enterprise space, then some thought will be required here since most IT don't see anything as being "free" or self-managed.Again, Sockets is an application API and not how one communicates to a TOE or RDMA component. The RNIC PI has been proposed as an interface to the RDMA functionality. The PI supports all of the iWARP and IB v 1.2 verbs. Mike I'd like to add that RNIC-PI is planning on explicitly defining some of these "obvious" dependencies between the RDMA stack and the primary IP stack. For example, the RDMA stack cannot maintain any connection in a state that contradicts current IP stack routing. It has to adapt or break the connection. We can't have an RNIC that has its own ARP table that is not in sync with the host's ARP table. An iWarp RDMA stack gains the benefit of many pre-existing network services (such as DNS, ARP and routing). But that also carries with it the need to not contradict those exisiting services. So it is both a benefit and a restriction -- and a major divergence from an IB RDMA stack. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux
On Fri, May 27, 2005 at 07:24:44AM -0700, Michael Krause wrote: ... > Again, Sockets is an application API and not how one communicates to a TOE > or RDMA component. Mike, What address family is used to open a socket over iWARP? AF_INET? Or something else? I understand most of what you wrote but am still missing one bit: How is the RNIC told what the peer IP is it should communicate with? > The RNIC PI has been proposed as an interface to the > RDMA functionality. The PI supports all of the iWARP and IB v 1.2 verbs. That's good. Folks from RDMA consortium will have to look at openib implementations and see whats missing/wrong. Then submit proposals to fill in the gaps. I'm obviously not the first one to say this. I expect most of the principals involved with openib.org do NOT have time to browse through RNIC PI at this point. They are struggling to get openib.org filled in sufficiently so it can go into a commercial distro (RH/SuSE primarily). Revenue for them comes from selling IB equipment. Having openib.org code in kernel.org is a key enabler for getting into commercial distros. I expect the same is true for RNIC vendors as well. RNIC Vendors (and related switch Vendors) will have to decide which path is the right one for them to get the support into kernel.org. Several openib.org people have suggested one (like I have). RNIC folks need to listen and decide if the advice is good or not. If RNIC folks think they know better, then please take another look at where openib.org is today and where rdmaconsortium is. I'm certain openib.org would be dead now if policies and direction changes had not made last year as demanded by several key linux developers and "users" (Gov Labs). grant ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux
Sukanta, without touching any TOE issues (this question is about RDMA, right?), after transforming a TCP connection into RDMA mode and using an RDMA API, the socket file descriptor is not longer to be used for communication. In fact, on some implementations the stream socket resources will even get released. That is, if the socket was the direct consumer of the TCP stream, then now it is RDMAP/DDP/MPA. RDMA APIs such as IT-API defining a specific call to convert a socket based connection into RDMA mode (e.g., it_socket_convert()). Other consumers may directly start via an RDMA API, never opening a consumer controlled socket. So, in RDMA mode, communication will happen via the RDMA API. At this stage, the kernel still have to keep completely in its hands the synchronisation of state information related to that offloaded connection(s) with the host stack (it would have to protect the local port used by the offloaded connection for example, others are routing, ARP, SNMP...), but it is not involved at the data path. With respect to the kernel based TCP stack, what is not needed is a hack into the stack and scatter/gather state information of the live TCP connection between kernel and RNIC, but to find one clean interface to transfer state information out of that stack and to the RNIC. With limited benefit, one could of course also implement native sockets over RDMA, where an in-kernel midlayer on top of kernel RDMA Verbs is doing the translation between send(), receive() to post_send, post_receive. But usage of 'true RDMA' operations like RDMA READ or WRITE might be limited, and I don't see much value for the user here. One variety of this approach with less limited access to RMDA benefits might be sockets with extended RDMA semantics. Bernard. [EMAIL PROTECTED] wrote on 27.05.2005 15:40:43: > Venkata, > How will that work? If the RNIC offloads RDMA and > TCP completely from the Operating System and does not > share any state information then the application > running on the host will never be in the position to > utilize the socket interface to use the communication > logic to send and receive data between the remote node > and itself. Some information needs to be shared. How > much of it and what exactly needs to be shared is the > question. > > Thanks > SG > > --- Venkata Jagana <[EMAIL PROTECTED]> wrote: > > > > > > > > > > > > > > [EMAIL PROTECTED] wrote on > > 05/25/2005 09:47:00 > > PM: > > > > > Venkata, > > > Interesting coincidence: I was talking with > > someone (at HP) today > > > who knows substantially more than I do about > > RNICs. > > > They indicated RNICs need to manage TCP state on > > the card from userspace. > > > I suspect that's only possible through a private > > interface > > > (e.g. ioctl() or /proc) or the non-existant (in > > kernel.org) > > > TOE implementation. Is this correct? > > > > > > > Not correct. > > > > Since RNICs are offloaded adapters with RDMA > > protocols layered on > > top of TCP stack, they do maintain the TCP state > > internally but > > it does not expose to the host. RNIC expose only > > RNIC Verbs interface > > to the host bot not TOE interface. > > > > Thanks > > Venkat > > > > > > > > hth, > > > grant > > > > > > > > > > > > --- > > > SF.Net email is sponsored by: GoToMeeting - the > > easiest way to > > collaborate > > > online with coworkers and clients while avoiding > > the high cost of travel > > and > > > communications. There is no equipment to buy and > > you can meet as often as > > > you want. Try it > > > free.http://ads.osdn.com/?ad_id=7402&alloc_id=16135&op=click > > > ___ > > > Rdma-developers mailing list > > > [EMAIL PROTECTED] > > > https://lists.sourceforge.net/lists/listinfo/rdma-developers > > __ > Do You Yahoo!? > Tired of spam? Yahoo! Mail has the best spam protection around > http://mail.yahoo.com > > > --- > This SF.Net email is sponsored by Yahoo. > Introducing Yahoo! Search Developer Network - Create apps using Yahoo! > Search APIs Find out how you can build Yahoo! directly into your own > Applications - visit http://developer.yahoo.net/?fr=offad-ysdn-ostg-q22005 > ___ > Rdma-developers mailing list > [EMAIL PROTECTED] > https://lists.sourceforge.net/lists/listinfo/rdma-developers ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux
At 06:40 AM 5/27/2005, Sukanta ganguly wrote: Venkata, How will that work? If the RNIC offloads RDMA and TCP completely from the Operating System and does not share any state information then the application running on the host will never be in the position to utilize the socket interface to use the communication logic to send and receive data between the remote node and itself. Some information needs to be shared. How much of it and what exactly needs to be shared is the question. Ok. It all depends upon what level of integration / interaction a TOE and thus a RNIC will have with the host network stack. For example, if a customer wants to have TCP and IP stats kept for the off-loaded stack even if it is just being using for RDMA, then there needs to be a method defined to consolidate these stats back into the host network stack tool chain. Similarly, if one wants to maintain a single routing table to manage, etc. on the host, then the RNIC needs to access / update that information accordingly. One can progress through other aspects of integration, e.g. connection management, security interactions (e.g. DOS protection), and so forth. What is exposed again depends upon the level of integration and how customers want to manage their services. This problem also exists for IB but most people have not thought about this from a customer perspective and how to integrate the IB semantics into the way customers manage their infrastructures, do billing, etc. For some environments, they simply do not care but if IB is to be used in the enterprise space, then some thought will be required here since most IT don't see anything as being "free" or self-managed. Again, Sockets is an application API and not how one communicates to a TOE or RDMA component. The RNIC PI has been proposed as an interface to the RDMA functionality. The PI supports all of the iWARP and IB v 1.2 verbs. Mike Thanks SG --- Venkata Jagana <[EMAIL PROTECTED]> wrote: > > > > > > > [EMAIL PROTECTED] wrote on > 05/25/2005 09:47:00 > PM: > > > Venkata, > > Interesting coincidence: I was talking with > someone (at HP) today > > who knows substantially more than I do about > RNICs. > > They indicated RNICs need to manage TCP state on > the card from userspace. > > I suspect that's only possible through a private > interface > > (e.g. ioctl() or /proc) or the non-existant (in > kernel.org) > > TOE implementation. Is this correct? > > > > Not correct. > > Since RNICs are offloaded adapters with RDMA > protocols layered on > top of TCP stack, they do maintain the TCP state > internally but > it does not expose to the host. RNIC expose only > RNIC Verbs interface > to the host bot not TOE interface. > > Thanks > Venkat > > > > > hth, > > grant > > > > > > > --- > > SF.Net email is sponsored by: GoToMeeting - the > easiest way to > collaborate > > online with coworkers and clients while avoiding > the high cost of travel > and > > communications. There is no equipment to buy and > you can meet as often as > > you want. Try it > free. http://ads.osdn.com/?ad_id=7402&alloc_id=16135&op=click > > ___ > > Rdma-developers mailing list > > [EMAIL PROTECTED] > > https://lists.sourceforge.net/lists/listinfo/rdma-developers __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com --- This SF.Net email is sponsored by Yahoo. Introducing Yahoo! Search Developer Network - Create apps using Yahoo! Search APIs Find out how you can build Yahoo! directly into your own Applications - visit http://developer.yahoo.net/?fr=offad-ysdn-ostg-q22005 ___ Rdma-developers mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/rdma-developers ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux
I completely agree with Michael's viewpoint here. Doing a clean RDMA focussed implementation would bring out the exact requirements that need to be changed. Retrofitting IB/Infiniband based API into the RDMA based subsystem just to reuse an existing base is not a good start. Thanks SG --- Michael Krause <[EMAIL PROTECTED]> wrote: > At 09:49 AM 5/26/2005, Sean Hefty wrote: > >Roland Dreier wrote: > >>I believe the way forward is to evolve the > existing drivers/infiniband > >>code already in Linux into a drivers/rdma that > supports both IB and > >>RNICs. To be extremely blunt, I believe the > RNIC-PI is irrelevant to > >>the Linux kernel -- no IB vendors will support > ripping out a working > >>midlayer and starting from scratch, and it doesn't > make sense to have > >>two essentially equivalent midlayers in the same > kernel. > > > >IMO, any APIs need to evolve out of the > implementation. Trying to fit an > >implementation under an existing API tends to lead > to either a poor > >implementation, or requires changes to the API > anyway. > > > >I think a useful path is for someone to implement > an RNIC driver and > >provide feedback on what changes would be required > of the Infiniband/core > >layer to support it. > > There needs to be a balance between establishing a > good, flexible > architecture and how applications or subsystems > interact with it. All of > this needs to be grounded in implementation > experience so there is a > healthy does of reality. The problem with the > iterative-TTM focused API > definitions is they rarely produce something that is > focused on the "end > game" for a solid interface. They create instant > legacy baggage that > people are unwilling to shed as time goes forward. > This legacy inertia is > what has stifled quite a bit of innovation when it > comes to software (some > hazard that 90% of the software created today is > focused primarily on > legacy investment protection than really new > innovation and when you think > about it for a bit of time, you can see that much of > this is re-inventing > the wheel or to band-aid over a problem and > packaging it as something > innovative). > > So, the problem becomes one of finding balance. The > people proposing the > various API are people who have implemented various > types of solutions so > they are coming with both real experience as well as > engineering judgement > of what is required to get the right infrastructure > in place for the "end > game" needs. From a practical perspective, one > needs to implement these > API and see what is really of value and what should > be changed. The key is > to avoid creating the legacy inertia that ends up > reducing fragmenting the > interfaces and causes lost productivity, quality > problems, etc. So before > people write off the various API as unnecessary or > as poor quality, there > should be some implementations developed and some > constructive debate of > what features are really of value and what might be > deprecated or > eliminated. There is no requirement to ever > implement all of an API as > this is where an iterative approach has value. > Implement what is needed to > demonstrate the desired value and decide then if it > is worthwhile. Just > please don't discount it or toss it all out because > it isn't all > implemented today or you don't like some aspect. > > > >>To put a really concrete proposal on the table, I > would suggest to > >>start by extending the current ib_client > registration structure > > > >Roland's proposal sounds like a reasonable and fair > way to > >begin. Building an abstraction on top of the > existing layers seems > >secondary to adding support for other RDMA devices. > > Getting the right verbs interface is paramount. > Whether one does IT API or > DAPL on top of that for a given subsystem is > secondary. > > Ideally, OpenIB and OpenRDMA should be focused on > developing the RDMA > infrastructure - RDMA verbs, connection management, > IB-specific ULP, > etc. They should not be focused on the upper > subsystems or ULP. Those > should be done as separate open source projects and > they can decide where > best to intersect with the Open* infrastructure. > This one-size-fits-all > approach for all subsystems to flow through a given > interface is simply > impractical to execute. It might happen over time > but it cannot be > force-fit. Let's focus the two efforts on getting > as common of an > infrastructure as possible. Some propose RNIC PI; > other can propose > something else if there is desire. RNIC PI has > value in that it has > focused on the common components between IB and > iWARP and left the > IB-specific and CM outside of its definition to > allow an OS to decide how > best to support or to allow IB to do what is needed > for its particular usages. > > > Mike __ Do you Yahoo!? Yahoo! Small Business -
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux
Venkata, How will that work? If the RNIC offloads RDMA and TCP completely from the Operating System and does not share any state information then the application running on the host will never be in the position to utilize the socket interface to use the communication logic to send and receive data between the remote node and itself. Some information needs to be shared. How much of it and what exactly needs to be shared is the question. Thanks SG --- Venkata Jagana <[EMAIL PROTECTED]> wrote: > > > > > > > [EMAIL PROTECTED] wrote on > 05/25/2005 09:47:00 > PM: > > > Venkata, > > Interesting coincidence: I was talking with > someone (at HP) today > > who knows substantially more than I do about > RNICs. > > They indicated RNICs need to manage TCP state on > the card from userspace. > > I suspect that's only possible through a private > interface > > (e.g. ioctl() or /proc) or the non-existant (in > kernel.org) > > TOE implementation. Is this correct? > > > > Not correct. > > Since RNICs are offloaded adapters with RDMA > protocols layered on > top of TCP stack, they do maintain the TCP state > internally but > it does not expose to the host. RNIC expose only > RNIC Verbs interface > to the host bot not TOE interface. > > Thanks > Venkat > > > > > hth, > > grant > > > > > > > --- > > SF.Net email is sponsored by: GoToMeeting - the > easiest way to > collaborate > > online with coworkers and clients while avoiding > the high cost of travel > and > > communications. There is no equipment to buy and > you can meet as often as > > you want. Try it > free.http://ads.osdn.com/?ad_id=7402&alloc_id=16135&op=click > > ___ > > Rdma-developers mailing list > > [EMAIL PROTECTED] > > https://lists.sourceforge.net/lists/listinfo/rdma-developers __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux
[EMAIL PROTECTED] wrote on 05/25/2005 09:47:00 PM: > Venkata, > Interesting coincidence: I was talking with someone (at HP) today > who knows substantially more than I do about RNICs. > They indicated RNICs need to manage TCP state on the card from userspace. > I suspect that's only possible through a private interface > (e.g. ioctl() or /proc) or the non-existant (in kernel.org) > TOE implementation. Is this correct? > Not correct. Since RNICs are offloaded adapters with RDMA protocols layered on top of TCP stack, they do maintain the TCP state internally but it does not expose to the host. RNIC expose only RNIC Verbs interface to the host bot not TOE interface. Thanks Venkat > > hth, > grant > > > --- > SF.Net email is sponsored by: GoToMeeting - the easiest way to collaborate > online with coworkers and clients while avoiding the high cost of travel and > communications. There is no equipment to buy and you can meet as often as > you want. Try it free.http://ads.osdn.com/?ad_id=7402&alloc_id=16135&op=click > ___ > Rdma-developers mailing list > [EMAIL PROTECTED] > https://lists.sourceforge.net/lists/listinfo/rdma-developers ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux
[EMAIL PROTECTED] wrote on 05/26/2005 04:33:46 AM: > On Wed, May 25, 2005 at 11:20:14PM -0700, Venkata Jagana wrote: > > I would like to really understand the technical reasons why you say > > RNIC-PI is irrelevant to Linux kernel. > > RNIC-PI is developed to support not only the RNICs but it is also IB > > compatible. This interface is > > something that is developed just like the open source process but with > > much broader community effort > > No, it's not. It's developed by an industry consortium that lacks any > taste. OpenRDMA community is actively involved in the development of this specification. Of course, your critical review input is taken and responded to. As always from Linux implementation standpoint, the implementation of this PI will evolve through open source community effort just like any other PI. Thanks Venkat ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
