Re: [openib-general] Re: [Rdma-developers] Meeting (07/22) summary:OpenRDMA community development discussion
[EMAIL PROTECTED] wrote on 07/29/2005 03:52:56 PM: On Fri, 29 Jul 2005 15:59:03 -0400 (EDT) James Lentini [EMAIL PROTECTED] wrote: On Fri, 29 Jul 2005, Woodruff, Robert J wrote: Venkat wrote, If anyone attended any one of the summits (netconf or kernel) and would be great if they can shed some light on this discussion. Roland attended the kernel summit and he was also at the InfiniBand BOF where we discussed the possibility of modifying the IB verbs to also support iWarp as probably the right way to go. Not sure if this was discussed at the kernel summit or not, but perhaps Roland can provide some insight on that question. The subject came up at the kernel summit when Jamal Hadi Salim reported on the presentations at Netconf, in particular Stephen Hemminger's, see http://vger.kernel.org/netconf2005.html). It was noted that iWARP vendors are working with the OpenIB community on a common interface. The consensus was that this is the right direction. The consensus was more of wait and see what comes up. My discussion at netconf was more of an informative session to get the participants to know that activity is going on and some code may be coming. It was all of 5 minutes at the end of a busy day, so it didn't count for much. Thanks Stephen for clarifying this. Yes, the basic code base in OpenRDMA is currently available for people to start hacking. My hope is that within next few months we'll also have some rnic driver available so that the basic set of enablement layers within both user and kernel space can be experimented with. Thanks Venkat___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Re: [Rdma-developers] Meeting (07/22) summary: OpenRDMA community development discussion
Christoph Hellwig [EMAIL PROTECTED] wrote on 07/29/2005 05:04:16 AM: your card drivers or rather start writing sane ones. Roland writing the new mthca driver of mellanox IB cards is what got the OpenIB ball rolling. Couldn't agree more on this. At a minimum, at least one RNIC vendor must try to opensource their basic card driver based on the kAL layer that UNM submitted already into OpenRDMA. Thanks Venkat___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: [Rdma-developers] Meeting (07/22) summary: OpenRDMA community development discussion
At OLS (and in previous forums), the kernel maintainers have made it *very* clear that there should only be one API. It is good to know but can you be specific about who those maintainers are and under what context they mentioned this at OLS and what ever the previous forums that you are referring to (wish I had gone to OLS but couldn't). Per networking summit report (http://lwn.net/Articles/144272/), my understanding is that the maintainers are looking for RDMA code but there is no mention about the API aspects. If anyone attended any one of the summits (netconf or kernel) and would be great if they can shed some light on this discussion. Thanks Venkat___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux
Exactly, the code matters from Linux community standpoint and the discussion around the convergence of common PI is mute until we have that header file definition but which will come out soon. However, I am quite glad to see the OpenIB and OpenRDMA communities in agreement on common ULP's and DAPL/IT-API (even though, there are some disagreements on these APIs). Also, as you pointed out, I absolutely agree the differences between Gen1 and Gen2 but which is exactly what I wanted to avoid with OpenRDMA and rather start from a clean slate right from the beginning through opensource fashion - basically, don't want the code to be dumped by some corporate developers. Thanks Venkat [EMAIL PROTECTED] wrote on 05/31/2005 09:38:51 AM: On Sat, May 28, 2005 at 04:26:43PM -0700, Caitlin Bestler wrote: ... if so what the best strategy for achieving it is (try to plan an IB/iWARP merge immediately or wait until there is an iWARP code base). If there is no iWARP code base, I fail to see how one can merge. Having a specification is one basis for communication. Linux developers normally use existing code as the basis. Committees submit CRs (Change Requests) to update specs. The CRs get voted on by the committee. Linux developers submit patches. The Linux subsystems maintainer(s) decide if patches are ok or not. Claiming that an InfiniBand-specific interface is somehow thinking long term is just plain ludicrous. It Works is worth 10x more to *any* customer than a transport neutral API that only exists as a spec. The specs are guides to how something *should* work and linux tries to comply with them (e.g. 802.3 or T10) where HW implementations actually follow the spec. That doesn't mean linux has to implement every brain damaged spec that some committee comes up withOTOH, rdmaconsortium.org does have a fair shot given I2O made it into the kernel. :^/ (I'm willing to have a conversation about why I think I2O is brain damaged if someone else is buying drinks. It's not total crap, but it certainly has it's downside.) Now it may be that the short term interest of the InfiniBand vendors is such that they cannot commit resources to helping build a transport neutral API. That is always a legitimate tradeoff, but it is short term corporate thinking. Please, that horse is already dead. They have offered to review patches to make the API transport neutral. Test that offer. Submit patches and move the conversation on to something that is more constructive. Last time I looked most of the commits being made to OpenIB (or sourceforge DAPL) were from being drawing paychecks from those evil corporations. Yes, so? The issue isn't the funding - it's the goals. Compare the gen1 stack (I'm being careful to not pick on any IB vendors) to the gen2 stack. The difference is between corporate code and linux code - mostly funded by the same corporation with several of the same programmers. gen1 stack came from somehing that attempted to build/run a shared user/kernel space on every distro. The Makefiles are just a mess - nevermind the code. grant --- This SF.Net email is sponsored by Yahoo. Introducing Yahoo! Search Developer Network - Create apps using Yahoo! Search APIs Find out how you can build Yahoo! directly into your own Applications - visit http://developer.yahoo.net/?fr=offad-ysdn-ostg-q22005 ___ Rdma-developers mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/rdma-developers ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux
I've been advocating rdmaconsortium folks submit patches against openib.org for several reasons: Probably, you meant openrdma.org opensource project but not a standards setting body (i.e. RDMA consortium - http://www.rdmaconsortium.org/home) :) 1) start with a code base that works 2) start with a code base that is already upstream 3) get advice/guidance from people who know how to collaborate in an open source environment. I thought (2) was the most important...but now I have to wonder if it's really (3). You are mistaken. I know people in the OpenRDMA community have worked with the opensource projects before and they know how to play and collaborate in an open source environment. The early part of the work in openrdma is in fact, a true example of that effort (which you may disagree with but having worked with several other opensource projects and with OpenIB, we have solved the issues which other projects including OpenIB have faced) and the next phase of work which is of course the code development, a key aspect of broader community effort. I think we are diverging from the real issue - the fundamental differences in the views of each community in how we can solve this common problem of supporting multiple RDMA fabrics, which is what we need to focus on. Just having OpenIB subsume control of anything iWARP or impose only DAPL for all RDMA infrastructure because it just happens to be there today seems rather stifling. Just stating that some OpenIB steering group is somehow empowered to decide this for Linux is also rather strange. AFAICT the openib.org steering group doesn't control the content of the svn.openib.org source tree. It manages things like web content, overall charter, etc Don't agree. If you have read the email thread on this discussion, you would find that steering committee need to decide whether openIB should work on including the support for iWARP. Not that I am supporting this idea -:) In the opensource world, developers should/will have the freedom to add what they want to do but of course, the acceptance of their contributions into mainline is completely a different matter. Thanks Venkat___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux
[EMAIL PROTECTED] wrote on 05/25/2005 09:47:00 PM: Venkata, Interesting coincidence: I was talking with someone (at HP) today who knows substantially more than I do about RNICs. They indicated RNICs need to manage TCP state on the card from userspace. I suspect that's only possible through a private interface (e.g. ioctl() or /proc) or the non-existant (in kernel.org) TOE implementation. Is this correct? Not correct. Since RNICs are offloaded adapters with RDMA protocols layered on top of TCP stack, they do maintain the TCP state internally but it does not expose to the host. RNIC expose only RNIC Verbs interface to the host bot not TOE interface. Thanks Venkat hth, grant --- SF.Net email is sponsored by: GoToMeeting - the easiest way to collaborate online with coworkers and clients while avoiding the high cost of travel and communications. There is no equipment to buy and you can meet as often as you want. Try it free.http://ads.osdn.com/?ad_id=7402alloc_id=16135op=click ___ Rdma-developers mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/rdma-developers ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux
Woodruff, Robert J [EMAIL PROTECTED] wrote on 05/26/2005 05:12:27 PM: RNIC-PI is at least an attempt at providing full control over both iWARP and IB while making as much common as possible. Where were you last year when the IB verbs header files/API were being discussed. Seems like that was the time to discuss new APIs. Now it is a bit late to propose something totally new like RNIC-PI. The best thing to do now is to try to influence the existing code base to meet your needs rather than recommend a totally new API. And the best way to do that is to make changes to the code that exists or write new code and send in patches to the list so that it can be discussed. woody It is just unfortunate to make such an IB-centric statement. I will ask the same question back to you - why didn't you consider RNICs when the IB APIs were designed a year ago? If that was considered a year ago, We wouldn't have this problem today, right? Nothing can be designed if we don't know what exactly need to be done for RNICs a year ago. The real answer is because it was too early to adopt RNIC-verbs at that time when the specs were still evolving in standard bodies (RDMAC/IETF). Again because we have developed something a year ago doesn't mean that we will have to live with that interface forever. Of course, the question is when and why should this change? As it was discussed before, we would like to see the common PI developed without impacting the IB development. Thanks Venkat___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux
[EMAIL PROTECTED] wrote on 05/27/2005 10:25:36 AM: Mike, I am not sure I do understand what your were trying to communicate. Let me try and decode this. My basic point was to respond to Venkata's response about complete offload without any interaction with the host system. I disagree with that in its totality as I think there are dependencies and that needs to be specified in a formal manner, i.e. in the specs, so that we do not have multiple proprietary interafaces which change and application users have to change this consumption based on individual implementations. And you just brought up the reasons why what I was saying seemed to be justfied. Did I read that correctly? Thanks SG Hello Sukanta, Yes, there will be interactions with the host system for an RNIC but through an interface something similar to RNIC-PI. The clarification that I was trying to make is that from an RNIC perspective, it doesn't have to expose any ToE interface other than the fact that adapters might be providing multiple services including ToE, RDMA, iSCSI but that's outside the scope of an RNIC interface. Thanks Venkat___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux
I am sure the developers in both of these communities have strong opinions in one way or
the other about the use of common interface and whatever it is but ultimately we need to
find a best possible way to move forward in order to support IB, RNICs and other future
RDMA fabrics. :)
Absolutely, we would like to see a common mid layer with RDMA-specific components
(for example, IB-CM and RNIC-CM would mostly be different but with very minimal commonality
between them) developed but also should work to provide a transport-neutral interface.
Today, we do not have a RDMA transport-neutral support with IB verbs PI in the kernel but
we would like to see such an interface so that we can extend it to support variety of RDMA fabrics.
Just because we have some interface supported in the kernel today doesn't mean that
we need to live with that interface forever. Within Linux kernel, we absolutely want to develop
an interface that is extendable to support the future fabrics and we can evolve this through
an open source process that is acceptable to linux kernel community.
And BTW, I am not suggesting to rip of this existing code right away but rather continue to use and
evolve it.
I would like to really understand the technical reasons why you say RNIC-PI is irrelevant to Linux kernel.
RNIC-PI is developed to support not only the RNICs but it is also IB compatible. This interface is
something that is developed just like the open source process but with much broader community effort
and also with review of linux OpenRDMA community and kernel community members.
Thanks
Venkat
Roland Dreier [EMAIL PROTECTED] wrote on 05/25/2005 07:35:52 PM:
I believe the way forward is to evolve the existing drivers/infiniband
code already in Linux into a drivers/rdma that supports both IB and
RNICs. To be extremely blunt, I believe the RNIC-PI is irrelevant to
the Linux kernel -- no IB vendors will support ripping out a working
midlayer and starting from scratch, and it doesn't make sense to have
two essentially equivalent midlayers in the same kernel.
To put a really concrete proposal on the table, I would suggest to
start by extending the current ib_client registration structure, which
looks like
struct ib_client {
char *name;
void (*add) (struct ib_device *);
void (*remove)(struct ib_device *);
struct list_head list;
};
by extending the current enum ib_node_type to something like
enum rdma_device_type {
RDMA_DEVICE_IB_CA,
RDMA_DEVICE_IB_SWITCH,
RDMA_DEVICE_IB_ROUTER,
RDMA_DEVICE_RNIC
};
Then the various pieces of code layered on top of the RDMA midlayer
can decide whether they want to deal with a particular device or not
by looking at the node_type member. For example, the IB CM, IPoIB,
etc. could ignore devices of type RDMA_DEVICE_RNIC, while SDP or iSER
would use all devices and the RNIC CM would take only devices of type
RDMA_DEVICE_RNIC.
Then someone would have to start implementing a low-level driver for a
specific RNIC, and find which modifications to the existing verbs are
required. For example, I believe the QP attribute structure passed
into the QP modify verb probably has to become a union containing the
IB attributes and the RNIC attributes. However, most verbs should
work fine with at most trivial modifications.
The existing OpenIB SDP code will be a good example to study as we
determine what abstractions need to be added to make it simple for
consumers to deal with the differences between IB and RNIC.
- R.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux
[EMAIL PROTECTED] wrote on 05/26/2005 04:33:46 AM: On Wed, May 25, 2005 at 11:20:14PM -0700, Venkata Jagana wrote: I would like to really understand the technical reasons why you say RNIC-PI is irrelevant to Linux kernel. RNIC-PI is developed to support not only the RNICs but it is also IB compatible. This interface is something that is developed just like the open source process but with much broader community effort No, it's not. It's developed by an industry consortium that lacks any taste. OpenRDMA community is actively involved in the development of this specification. Of course, your critical review input is taken and responded to. As always from Linux implementation standpoint, the implementation of this PI will evolve through open source community effort just like any other PI. Thanks Venkat ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux
Bob Woodruff [EMAIL PROTECTED] wrote on 05/26/2005 08:49:59 AM: Venkat I would like to start a discussion around the convergence of RDMA APIs and ULPs Venkat between OpenIB and OpenRDMA projects. Once the RNIC people have a mid-layer that interfaces with the RDMA API (kDAPL derivative), we can look at where there might be common code that could be shared between the InfiniBand mid-layer and the iWarp mid-layer. This would allow the RNIC vendors to develop code that could use the same ULPs without slowing down the InfiniBand development. Absolutely and this is exactly what I would like to see without impacting the IB development but supporting common ULPs is critical. Thanks Venkat ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux
I would like to start a discussion around the convergence of RDMA APIs and ULPs between OpenIB and OpenRDMA projects. As you all know, Infiniband and iWARP based RNICs support RDMA capabilities being exploited by both kernel and user based applications and which can take advantage of these RDMA capabilities through standards based RDMA APIs such as DAPL, IT-API (v1/v2). There exists a set of upper layer protocols, such as NFS, SRP/iSER, SDP, which are mostly kernel based and also exists user based middleware/applications such as DB2, Oracle, scientific applications which would like to use a common set of APIs supported by the underlying operating systems in order to work over different RDMA fabrics like IB and RNICs. >From Linux kernel perspective, it is undesirable to have a different set of APIs and ULPs supported for variety of reasons including but not limited to the duplication, testing effort etc. OpenIB and OpenRDMA projects are separate efforts and are actively working in its own paths to develop the corresponding RDMA support in Linux but we want to make sure we work together to avoid the duplication in providing the support. The proposal for both communities is to start thinking and discussing on how best we could accomplish this commonality between these two projects. BTW, To make this objective further clear - this proposal is not about merging these two projects since each project has its own objective of supporting its RDMA function and rather intended to steer both projects toward the goal of standardizing RDMA APIs and providing common ULPs as applicable. However, we also have a challenge to address in implementing these common ULPs and APIs since OpenIB is currently using verbs PI for Linux defined through an open source process and OpenRDMA is currently defining RNIC-PI (supporting RNIC and IB compatible verbs) for Linux based on the industry standard evolving through Opengroup/ICSC and open source community reviews. The ultimate challenge for us is to come up with a common PI acceptable in Linux while taking into account the standards, hardware vendors portability for device drivers, ULPs etc. Thanks, Venkat ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
