Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux
I did some research on the first topic. The unified event model was designed based on user feedback. User's of previous RDMA APIs found multiple event streams difficult to manage. They wanted the ability to have a single event stream that maintained ordering guarantees. DAPL allows users to have separate event dispatchers (EVDs) for different event types. So if you wanted to have connection events seperate from data transfer event, you can do that. Here's a little background on your second issue for those who aren't familiar with DAPL: To translate IP addresses to GIDs, the original reference implementation of DAPL looked up these mappings in the SA database. This implementation was dubbed ATS (Address Translation Service). ATS is only one possible way to implement this. Other methods could be used. On Tue, 31 May 2005, Libor Michalek wrote: On Tue, May 31, 2005 at 02:03:06PM -0700, Tom Duffy wrote: On Sat, 2005-05-28 at 09:13 +0200, Christoph Hellwig wrote: On Fri, May 27, 2005 at 03:56:58PM -0700, Bob Woodruff wrote: kDAPL is intended as a kernel-level API for RDMA enabled fabrics. As it was initially written, it does not meet the Linux coding style and that is why it is being totally reworked as we speak to meet that goal. The codingstyle alone isn't the problem. The whole design philosophy is rather odd. As one of the people trying to clean up kDAPL, I would like to know what you think, from a design philosophy, is wrong with it. We *can* correct any daim bramaged parts. Well, from a kernel API design philosophy the evd is somewhat odd. The whole idea behind the event model seems a bit convoluted. First multiplex a wide variety of events from the provider into a single event queue, and then have an API so the consumer can tell what type of event they actually have and can still receive the event notification in the provider's context. This seems to be a lot of work to first hide useful information, but also not loose the information in case the consumer really does want it. It appears to be a case of a decent userspace idea that doesn't make much sense in the kernel. Why is it there? I imagine it's to abstract a variety of OS kernels, which was one of the goals of the design. Also, I realize it's just an implementation detail, but I've got a number of issues with ATS. -Libor ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux
I hadn't heard that insmod was being removed from Linux. In fact the DAPL Plugfest successfully used kernel daemons and kdapltest to demonstrate DAT interoperability across multiple vendors: kernel to kernel, kernel to user and user to user. These are existing applications already deployed. I fail to see how you believe breaking them improves Linux. Software like the NFS over RDMA or SDP implementations are developed as kernel daemons outside of the official kernel first. Others develop special interest drivers and daemons that will never be of sufficient general interest to be incorporated in the kernel. That's why insmod exists. the ability to add prototype or specialty daemon in the kernel is an ability that virtually every operating system provides. And there are KDAPL applications runnign that way *today*. Whatever runs inside the kernel source tree can be totally unrelated to that, but it shouldn't export existing DAT symbols if it changes their meaning. And existing device drivers and DAT Provider libraries should not be broken either (assuming the OS services that they *are* relying on are still there. If those changed, the developers of those modules are responsible for udpating them). On 6/2/05, Christoph Hellwig [EMAIL PROTECTED] wrote: On Wed, Jun 01, 2005 at 06:33:14PM -0700, Caitlin Bestler wrote: As I stated earlier, kDAPL is designed for *applications* and was not really intended for use by the OS itself. As such it abstracts more than just the transport. I'm not sure what you call kernel applications, but I'm pretty sure we're not interested to support anything that comes close to this term in linux kernelspace. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux
On Thu, Jun 02, 2005 at 05:22:26AM -0700, Caitlin Bestler wrote: I hadn't heard that insmod was being removed from Linux. No one claimed that. In fact the DAPL Plugfest successfully used kernel daemons and kdapltest to demonstrate DAT interoperability across multiple vendors: kernel to kernel, kernel to user and user to user. And that matters for this discussion exactly how? These are existing applications already deployed. I fail to see how you believe breaking them improves Linux. Again, what application. Also who says we want to improve Linux to make it easier to use for everyone? We're certainly not interested in supporting freeloaders. Software like the NFS over RDMA or SDP implementations are developed as kernel daemons outside of the official kernel first. Others develop special interest drivers and daemons that will never be of sufficient general interest to be incorporated in the kernel. Okay, after these rants we come closer to what you actually mean with kernel applications. We're certainly interested in supporting filesystems or networking protocol, but like all other in-kernel APIs we don't care at all about out of tree projects. Especially as filesystems and network protocols are inherently part of the bigger kernel picture and benefit from integration with other parts of the kernel. Not even mentioning license implications as I very strongly doubt you can claim either of it is not a derived work. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux
On Wed, 1 Jun 2005, Tom Duffy wrote:
On Wed, 2005-06-01 at 12:04 +0200, Christoph Hellwig wrote:
That beeing said, one of the first thing you should get rid of if you
want to be able to take code from kdapl to the generic rdma code is way
it deals with handles. The kdapl code gives up language type safety
just to add its own bandaided type system below. Please always pass the
real struct pointer around, or at least the generic struct type which
might be embedded into a bigger structure, and get rid of all the magic
number checking and magic macros that expect special members at the
beginning of all objects behind these handles.
I absolutely agree with this and was planning on implementing said
changes in the near future.
James, can I get a hell yeah or a no freaking way from you before I
go off and start writing patches?
-tduffy
This could be an improvement. We just need to be careful that we don't
expose anything transport specific. Off the top of my head, I can
think of one way to do this:
- in dat.h, create a public structure for each object type:
struct dat_ep {
struct dat_provider *provider;
};
- in the transport provider (dapl.h) have a private structure that
contains the public one:
struct dapl_ep {
struct dat_ep dat_ep;
...
};
Is that the sort of thing you were thinking of?
I'll do some research here and find out why the original mechanism was
chosen.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux
On Thu, 2005-06-02 at 11:16 -0400, James Lentini wrote:
- in dat.h, create a public structure for each object type:
struct dat_ep {
struct dat_provider *provider;
};
- in the transport provider (dapl.h) have a private structure that
contains the public one:
struct dapl_ep {
struct dat_ep dat_ep;
...
};
Is that the sort of thing you were thinking of?
Yes, exactly. Then you can create small inline functions that would
cast back and forth.
The other issue right now is that dapl has a header struct that needs
to come first in all the structs. So, that would need to be changed.
I'll do some research here and find out why the original mechanism was
chosen.
Cool.
-tduffy
signature.asc
Description: This is a digitally signed message part
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux
On Thu, 2 Jun 2005, Tom Duffy wrote:
On Thu, 2005-06-02 at 11:16 -0400, James Lentini wrote:
- in dat.h, create a public structure for each object type:
struct dat_ep {
struct dat_provider *provider;
};
- in the transport provider (dapl.h) have a private structure that
contains the public one:
struct dapl_ep {
struct dat_ep dat_ep;
...
};
Is that the sort of thing you were thinking of?
Yes, exactly. Then you can create small inline functions that would
cast back and forth.
The other issue right now is that dapl has a header struct that needs
to come first in all the structs. So, that would need to be changed.
I think that issue will go away if we implement the above proposal.
The dapl objects all place the dapl_header structure first so that the
struct dat_provider pointer will be the first item in memory. This
allows the dat_* functions to cast the user's handle to a struct
dat_provider pointer and then call the appropriate provider function.
I'll do some research here and find out why the original mechanism was
chosen.
Cool.
-tduffy
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux
On 6/2/05, Tom Duffy [EMAIL PROTECTED] wrote:
On Thu, 2005-06-02 at 11:16 -0400, James Lentini wrote:
- in dat.h, create a public structure for each object type:
struct dat_ep {
struct dat_provider *provider;
};
- in the transport provider (dapl.h) have a private structure that
contains the public one:
struct dapl_ep {
struct dat_ep dat_ep;
...
};
Is that the sort of thing you were thinking of?
Yes, exactly. Then you can create small inline functions that would
cast back and forth.
They were actually #defines in the DAT 1.0 headers, but I agree
that inline functions are better. At the time the #defines were
adopted there was still a concern about compilers that did
not implement inlining efficiently. I believe that concern is
no longer valid in general, and it is especially not relevant
when gcc is known to be available.
The other issue right now is that dapl has a header struct that needs
to come first in all the structs. So, that would need to be changed.
That is what enables the use of the method table. It allows the
in-line function to know the actual method to call for 'dat_ep_post_send'
on this specific EP without knowing at compile time which Provider
code is being invoked.
The DAT 1.0 headers included some #define options to let that
be hard-coded in environments where it was known that there
was only a single provider.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux
On Thu, Jun 02, 2005 at 11:16:53AM -0400, James Lentini wrote:
This could be an improvement. We just need to be careful that we don't
expose anything transport specific. Off the top of my head, I can
think of one way to do this:
- in dat.h, create a public structure for each object type:
struct dat_ep {
struct dat_provider *provider;
};
- in the transport provider (dapl.h) have a private structure that
contains the public one:
struct dapl_ep {
struct dat_ep dat_ep;
...
};
Is that the sort of thing you were thinking of?
That would follow the way the VFS does for inodes. I just think dapl_*
is a really bad name for an IB-specific implementation. In the end
when we converge thing more into a common RDMA stack that should
probably become rdma_* for the transport-agnostic and ib_* for
IB-specific structure.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux
On Tue, May 31, 2005 at 02:03:06PM -0700, Tom Duffy wrote: On Sat, 2005-05-28 at 09:13 +0200, Christoph Hellwig wrote: On Fri, May 27, 2005 at 03:56:58PM -0700, Bob Woodruff wrote: kDAPL is intended as a kernel-level API for RDMA enabled fabrics. As it was initially written, it does not meet the Linux coding style and that is why it is being totally reworked as we speak to meet that goal. The codingstyle alone isn't the problem. The whole design philosophy is rather odd. As one of the people trying to clean up kDAPL, I would like to know what you think, from a design philosophy, is wrong with it. We *can* correct any daim bramaged parts. kDAPL is supposed to serve two needs: (1) provide an unified API for different RDMA transports (2) provide various higher level helpers as such it's largely duplicating what a proper RDMA stack should be. For (1) doing a proper RDMA stack should solve thing, and the discussion how to do it is already ongoing on this list. Once we have proper RMDA stack that part of KDAPL isn't needed at all anymore. The second is more interesting and there's indeed the need for some higher level helpers than the API at the level of the current OpenIB code offers. But having a separate layer, with different data structures, provider registration and a totally different API for that is utter nonsense. Instead the higher level helpers should operate on the same datastructures as the RDMA stack, or build new ones ontop of that. In addition some of the abstractions don't make much sense, the event handling has already been mentioned. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux
On Wed, Jun 01, 2005 at 09:36:09AM +0200, Christoph Hellwig wrote: For (1) doing a proper RDMA stack should solve thing, and the discussion how to do it is already ongoing on this list. Once we have proper RMDA stack that part of KDAPL isn't needed at all anymore. The second is more interesting and there's indeed the need for some higher level helpers than the API at the level of the current OpenIB code offers. But having a separate layer, with different data structures, provider registration and a totally different API for that is utter nonsense. Instead the higher level helpers should operate on the same datastructures as the RDMA stack, or build new ones ontop of that. In addition some of the abstractions don't make much sense, the event handling has already been mentioned. That beeing said, one of the first thing you should get rid of if you want to be able to take code from kdapl to the generic rdma code is way it deals with handles. The kdapl code gives up language type safety just to add its own bandaided type system below. Please always pass the real struct pointer around, or at least the generic struct type which might be embedded into a bigger structure, and get rid of all the magic number checking and magic macros that expect special members at the beginning of all objects behind these handles. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux
At 03:36 AM 6/1/2005, Christoph Hellwig wrote: kDAPL is supposed to serve two needs: (1) provide an unified API for different RDMA transports (2) provide various higher level helpers ... For (1) doing a proper RDMA stack should solve thing, and the discussion how to do it is already ongoing on this list. Once we have proper RMDA stack that part of KDAPL isn't needed at all anymore. I certainly agree, and have said so before, that if there is a portable layer (verbs or otherwise) which allows me to have a single NFS module working over both IB and iWARP, then kDAPL is not architecturally necessary. The verbs API however, isn't it. The current NFS/RDMA client, over kDAPL, is 3.6KLOC, on the order of the same amount of code as the sockets NFS: about 1800 lines for connection management and transport handling, about 800 for marshalling to and from RPC, and the rest managing the RPC's themselves. This is a HUGE advantage, because it means none of the RDMA complexity is in the RPC or NFS modules. In fact, the NFS module is completely unchanged (same binary). Which is exactly how want it! So, again, I'm happy to use another RDMA API. But the top-level requirements are: - must enable single upper layer implementation (one RPC/RDMA module) - must perform equally well for all transports - must not expose needless complexity - must exist :-) I think kDAPL already is such a thing. When the OpenIB kDAPL starts to work, we'll run the NFS/RDMA client on it the next day. BTW the Linux NFS/RDMA server (also on kDAPL) is accepting connections at CITI, now moving on to actually executing RPCs. Tom. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux
On Wed, 2005-06-01 at 12:04 +0200, Christoph Hellwig wrote: That beeing said, one of the first thing you should get rid of if you want to be able to take code from kdapl to the generic rdma code is way it deals with handles. The kdapl code gives up language type safety just to add its own bandaided type system below. Please always pass the real struct pointer around, or at least the generic struct type which might be embedded into a bigger structure, and get rid of all the magic number checking and magic macros that expect special members at the beginning of all objects behind these handles. I absolutely agree with this and was planning on implementing said changes in the near future. James, can I get a hell yeah or a no freaking way from you before I go off and start writing patches? -tduffy signature.asc Description: This is a digitally signed message part ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux
The need for a single unifying event disptaching method was *the* primary feedback received from VIA user when DAPL started. That included both kernel and user-mode applications, in fact one of the primary reasons why the DAT Collaborative was started was that there was no unrestricted API for VIA that could be used in the kernel. As I stated earlier, kDAPL is designed for *applications* and was not really intended for use by the OS itself. As such it abstracts more than just the transport. If you wrip out the EVDs it isn't really kDAPL anymore. Having a stripped subset of kDAPL with a clearly different name that is *only* what is needed to abstract transport differences might be a really neat idea, and having it be otherwise compatible with kDAPL would also be good. But I'd suggest understanding why something is present before ripping it out. EVDs are there precisely because dealing with multiple callbacks and completion queues was universally found to be unwieldly and unfriendly to application developers. On 5/31/05, Libor Michalek [EMAIL PROTECTED] wrote: Well, from a kernel API design philosophy the evd is somewhat odd. The whole idea behind the event model seems a bit convoluted. First multiplex a wide variety of events from the provider into a single event queue, and then have an API so the consumer can tell what type of event they actually have and can still receive the event notification in the provider's context. This seems to be a lot of work to first hide useful information, but also not loose the information in case the consumer really does want it. It appears to be a case of a decent userspace idea that doesn't make much sense in the kernel. Why is it there? I imagine it's to abstract a variety of OS kernels, which was one of the goals of the design. Also, I realize it's just an implementation detail, but I've got a number of issues with ATS. -Libor ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux
On 6/1/05, Christoph Hellwig [EMAIL PROTECTED] wrote: On Wed, Jun 01, 2005 at 09:36:09AM +0200, Christoph Hellwig wrote: For (1) doing a proper RDMA stack should solve thing, and the discussion how to do it is already ongoing on this list. Once we have proper RMDA stack that part of KDAPL isn't needed at all anymore. The second is more interesting and there's indeed the need for some higher level helpers than the API at the level of the current OpenIB code offers. But having a separate layer, with different data structures, provider registration and a totally different API for that is utter nonsense. Instead the higher level helpers should operate on the same datastructures as the RDMA stack, or build new ones ontop of that. In addition some of the abstractions don't make much sense, the event handling has already been mentioned. That beeing said, one of the first thing you should get rid of if you want to be able to take code from kdapl to the generic rdma code is way it deals with handles. The kdapl code gives up language type safety just to add its own bandaided type system below. Please always pass the real struct pointer around, or at least the generic struct type which might be embedded into a bigger structure, and get rid of all the magic number checking and magic macros that expect special members at the beginning of all objects behind these handles. That is a good suggestion, and is fully compatible with DAT defnitions. In fact the first version of the reference implementation did use struct pointers as the DAT handles. The indirection method was added after the fact to reduce problems with bad pointers in sloppy user-mode applications, and is specifically something that any DAT implementation is free to revert back to the struct pointer. I fully agree that having the indirected handled is a layer of safety that is not called for in a kernel (or any embedded application for that matter). DAT Consumers are not supposed to rely upon such safety mechanisms, and the DAT handle is opaque to them. So changing the representation of the DAT Handle is *totally* legitimate. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux
On 5/31/05, Libor Michalek [EMAIL PROTECTED] wrote: Also, I realize it's just an implementation detail, but I've got a number of issues with ATS. ATS was a workaround the fact that OS naming systems could not be relied upon to deal with IPv6 format addresses (i.e. GIDs). That is an artifact of DAT being distributed outside of the OS. Working in full integration with the kernel I would see no reason why ATS could not be replaced by any other solution that provided the same answers. I suspect being integrated with subnet management and the kernel that there are in fact a lot of new options that were not available in the context where ATS was designed. ATS is not visible to the application. There's a reason for that. Any solution that can translate dat_ep_connect to the correct remote endpoint works and will not break existing DAT applications as long as the new solution can accept at least the same data as ATS does now (or equivalent). ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux
On Sat, May 28, 2005 at 04:26:43PM -0700, Caitlin Bestler wrote: ... if so what the best strategy for achieving it is (try to plan an IB/iWARP merge immediately or wait until there is an iWARP code base). If there is no iWARP code base, I fail to see how one can merge. Having a specification is one basis for communication. Linux developers normally use existing code as the basis. Committees submit CRs (Change Requests) to update specs. The CRs get voted on by the committee. Linux developers submit patches. The Linux subsystems maintainer(s) decide if patches are ok or not. Claiming that an InfiniBand-specific interface is somehow thinking long term is just plain ludicrous. It Works is worth 10x more to *any* customer than a transport neutral API that only exists as a spec. The specs are guides to how something *should* work and linux tries to comply with them (e.g. 802.3 or T10) where HW implementations actually follow the spec. That doesn't mean linux has to implement every brain damaged spec that some committee comes up withOTOH, rdmaconsortium.org does have a fair shot given I2O made it into the kernel. :^/ (I'm willing to have a conversation about why I think I2O is brain damaged if someone else is buying drinks. It's not total crap, but it certainly has it's downside.) Now it may be that the short term interest of the InfiniBand vendors is such that they cannot commit resources to helping build a transport neutral API. That is always a legitimate tradeoff, but it is short term corporate thinking. Please, that horse is already dead. They have offered to review patches to make the API transport neutral. Test that offer. Submit patches and move the conversation on to something that is more constructive. Last time I looked most of the commits being made to OpenIB (or sourceforge DAPL) were from being drawing paychecks from those evil corporations. Yes, so? The issue isn't the funding - it's the goals. Compare the gen1 stack (I'm being careful to not pick on any IB vendors) to the gen2 stack. The difference is between corporate code and linux code - mostly funded by the same corporation with several of the same programmers. gen1 stack came from somehing that attempted to build/run a shared user/kernel space on every distro. The Makefiles are just a mess - nevermind the code. grant ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux
On Sat, May 28, 2005 at 05:18:39PM -0700, Caitlin Bestler wrote:
Verus..
struct rdma_xyz {
/* common fields */
};
struct rdma_xyz_ib {
struct rdma_xyz common;
/* ib fields */
};
struct rdma_xyz_iwarp {
struct rdma_xyz common;
/* iwarp fields */
};
The latter style is extensible, but makes it difficult to properly
allocate a buffer that works for all variants.
The latter assumes the transport specific code is owns responsibilty
for allocating/deallocating those buffers.
It also forces the generic code to be completely ignorant of
the transport specific stuff. It doesn't allow the programmer
to hacking around in the public unions.
The union style is also already in use in both IT-API and RNIC-PI.
I personally prefer sub-classing to unions, but I have found myself in the
minority on *most* projects where the issue has been discussed.
One reason is that sub-classing provides very little type-safety.
struct sockaddr is an example of this. It takes manual inspection
to ensure that the variants are properly differentiated and it is
still common for developers to pass in a plain struct sockaddr
without realizing that it is not large enough for a struct sockaddr_in6.
IMHO, unions are a sort of casts on whole structures.
Neither method really offers an advantage in type checking.
Both require one knows which type is the right one.
grant
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux
Exactly, the code matters from Linux community standpoint and the discussion around the convergence of common PI is mute until we have that header file definition but which will come out soon. However, I am quite glad to see the OpenIB and OpenRDMA communities in agreement on common ULP's and DAPL/IT-API (even though, there are some disagreements on these APIs). Also, as you pointed out, I absolutely agree the differences between Gen1 and Gen2 but which is exactly what I wanted to avoid with OpenRDMA and rather start from a clean slate right from the beginning through opensource fashion - basically, don't want the code to be dumped by some corporate developers. Thanks Venkat [EMAIL PROTECTED] wrote on 05/31/2005 09:38:51 AM: On Sat, May 28, 2005 at 04:26:43PM -0700, Caitlin Bestler wrote: ... if so what the best strategy for achieving it is (try to plan an IB/iWARP merge immediately or wait until there is an iWARP code base). If there is no iWARP code base, I fail to see how one can merge. Having a specification is one basis for communication. Linux developers normally use existing code as the basis. Committees submit CRs (Change Requests) to update specs. The CRs get voted on by the committee. Linux developers submit patches. The Linux subsystems maintainer(s) decide if patches are ok or not. Claiming that an InfiniBand-specific interface is somehow thinking long term is just plain ludicrous. It Works is worth 10x more to *any* customer than a transport neutral API that only exists as a spec. The specs are guides to how something *should* work and linux tries to comply with them (e.g. 802.3 or T10) where HW implementations actually follow the spec. That doesn't mean linux has to implement every brain damaged spec that some committee comes up withOTOH, rdmaconsortium.org does have a fair shot given I2O made it into the kernel. :^/ (I'm willing to have a conversation about why I think I2O is brain damaged if someone else is buying drinks. It's not total crap, but it certainly has it's downside.) Now it may be that the short term interest of the InfiniBand vendors is such that they cannot commit resources to helping build a transport neutral API. That is always a legitimate tradeoff, but it is short term corporate thinking. Please, that horse is already dead. They have offered to review patches to make the API transport neutral. Test that offer. Submit patches and move the conversation on to something that is more constructive. Last time I looked most of the commits being made to OpenIB (or sourceforge DAPL) were from being drawing paychecks from those evil corporations. Yes, so? The issue isn't the funding - it's the goals. Compare the gen1 stack (I'm being careful to not pick on any IB vendors) to the gen2 stack. The difference is between corporate code and linux code - mostly funded by the same corporation with several of the same programmers. gen1 stack came from somehing that attempted to build/run a shared user/kernel space on every distro. The Makefiles are just a mess - nevermind the code. grant --- This SF.Net email is sponsored by Yahoo. Introducing Yahoo! Search Developer Network - Create apps using Yahoo! Search APIs Find out how you can build Yahoo! directly into your own Applications - visit http://developer.yahoo.net/?fr=offad-ysdn-ostg-q22005 ___ Rdma-developers mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/rdma-developers ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux
On Fri, 27 May 2005, Bob Woodruff wrote: Caitlin wrote, Both uDAPL and kDAPL were designed for *application* use. Even kDAPL is more intended for use by a kernel daemon that is loaded separately from the kernel than for use within the kernel itself. kDAPL is intended as a kernel-level API for RDMA enabled fabrics. As it was initially written, it does not meet the Linux coding style and that is why it is being totally reworked as we speak to meet that goal. An ideal API for use within the kernel would abstract as much as possible (without requiring emulation), and then have transport specific unions or enum values. It would hide no control options, merely provide common controls for common capabilities. So for every new RDMA device type that comes along, you need to add a new enum, and unions for device class specific stuff, etc. Seems rather static and not easily extended. Not to mention that testing nightmare when the thing has to support 20 different types of RDMA enabled devices. I think code like that could get pretty ugly pretty fast. I'd rather see a registration mechanism like what we already have with DAPL that does not require any code changes to add a new RDMA device/provider. We have already proven that this works in DAPL as I know if at least 3 providers, IB, Myrinet, and RNIC (Ammasso) that were developed separately and were able to co-exist without any changes (enums and device class unions) in the DAT mid-layer. I assume that this can also be done with kDAPL in the kernel, but I defer to the DAPL experts to answer that one. Correct. The DAT API (kernel and user) is designed to support heterogeneous providers. The modifications we are making in https://openib.org/svn/gen2/users/jlentini/ will not change that. james ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux
At 06:47 AM 5/28/2005, Christoph Hellwig wrote: On Sat, May 28, 2005 at 05:17:54AM -0700, Sukanta ganguly wrote: That's a pretty bold statement. Linux grew up to be popular via mass acceptance. Seems like that charter has changed and a few have control over Linux and its future. The My way or the highway philosophy has gotten embedded in the Linux way of life. Life is getting tough. You're totally missing the point. Linux is successfull exactly because it's lookinf for the right solution, not something the business people need short-term. Hence why some of us contend that the end-game, i.e. the right solution, is not necessarily the short-term implementation that is present today that just evolves creating that legacy inertia that I wrote about earlier. I think there is validity to having an implementation to critique - accept, reject, modify. I think there is validity to examining industry standards as the basis for new work / implementation. If people are unwilling to discuss these standards and only stay focused on their business people's short-term needs, then some might contend as above that Linux is evolving to be much like the dreaded Pacific NW company in the end. Not intending to offend anyone but if there can be no debate without implementation on what is the right solution, then people might as well just go off and implement and propose their solution for incorporation into the Linux kernel. It may be that OpenIB wins in the end or it may be that it does not. Just having OpenIB subsume control of anything iWARP or impose only DAPL for all RDMA infrastructure because it just happens to be there today seems rather stifling. Just stating that some OpenIB steering group is somehow empowered to decide this for Linux is also rather strange. Open source is about being open and not under the control of any one entity in the end. Perhaps that is no longer the case. Mike ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux
On Tue, May 31, 2005 at 02:31:19PM -0700, Michael Krause wrote: ... Not intending to offend anyone but if there can be no debate without implementation on what is the right solution, then people might as well just go off and implement and propose their solution for incorporation into the Linux kernel. That is certainly one option. I didn't see anyone in openib.org trying to take that choice away. Is it easier to submit a new subsystem than fixup an existing one? I honestly don't know the answer since both options could fail depending on how people approach them. But my gut feeling is if rdmaconsortium can't play nicely with openib.org, they won't be able to play nicely with kernel.org either. I've been advocating rdmaconsortium folks submit patches against openib.org for several reasons: 1) start with a code base that works 2) start with a code base that is already upstream 3) get advice/guidance from people who know how to collaborate in an open source environment. I thought (2) was the most important...but now I have to wonder if it's really (3). Several very good people are driving openib.org developement. Just having OpenIB subsume control of anything iWARP or impose only DAPL for all RDMA infrastructure because it just happens to be there today seems rather stifling. Just stating that some OpenIB steering group is somehow empowered to decide this for Linux is also rather strange. steering group is Committee talk. AFAICT the openib.org steering group doesn't control the content of the svn.openib.org source tree. It manages things like web content, overall charter, etc. People do NOT have to be members of the steering committee or openib.org to become either maintainers or to submit code. Open source is about being open and not under the control of any one entity in the end. Perhaps that is no longer the case. No. SOME entity always controls what goes in (or not) any given source tree. That has nothing to do with open source. Open source is about collaboration and being able to fork if that collaboration ceases to be useful. One can substitute trust for the word collaboration and it would be accurate too. Figure out how to build trust (without contracts!) and then how to get things done in open source becomes clear. hth, grant ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux
I completely agree. I think this thread was not started to get one of the projects out of the way of the other. I would think it was started to coordinate the development cycles of two related projects, where one project is admittedly much more advanced, just due to the since years now availability of IB technology. I also think that it is not yet proven that good software must always get created by writing something down and then reshaping it to meet upcoming requirements. OpenRDMA has choosen to start by trying to identify main requirements and then agreeing upon an appropriate architecture. We did not start with discussing the style of commentary lines, because it was assumed that the style of commentary lines is less important and even easier to fix. Enabling iWARP under Linux is not an easy task and we are dependent on the open source communities help and support to make this happen. We are not in the position nor willing to bypass this procedure - and its always good to have fruitful discussion. Bernard. [EMAIL PROTECTED] wrote on 31.05.2005 23:31:19: At 06:47 AM 5/28/2005, Christoph Hellwig wrote: On Sat, May 28, 2005 at 05:17:54AM -0700, Sukanta ganguly wrote: That's a pretty bold statement. Linux grew up to be popular via mass acceptance. Seems like that charter has changed and a few have control over Linux and its future. The My way or the highway philosophy has gotten embedded in the Linux way of life. Life is getting tough. You're totally missing the point. Linux is successfull exactly because it's lookinf for the right solution, not something the business people need short-term. Hence why some of us contend that the end-game, i.e. the right solution, is not necessarily the short-term implementation that is present today that just evolves creating that legacy inertia that I wrote about earlier. I think there is validity to having an implementation to critique - accept, reject, modify. I think there is validity to examining industry standards as the basis for new work / implementation. If people are unwilling to discuss these standards and only stay focused on their business people's short- term needs, then some might contend as above that Linux is evolving to be much like the dreaded Pacific NW company in the end. Not intending to offend anyone but if there can be no debate without implementation on what is the right solution, then people might as well just go off and implement and propose their solution for incorporation into the Linux kernel. It may be that OpenIB wins in the end or it may be that it does not. Just having OpenIB subsume control of anything iWARP or impose only DAPL for all RDMA infrastructure because it just happens to be there today seems rather stifling. Just stating that some OpenIB steering group is somehow empowered to decide this for Linux is also rather strange. Open source is about being open and not under the control of any one entity in the end. Perhaps that is no longer the case. Mike___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux
On Tue, May 31, 2005 at 02:03:06PM -0700, Tom Duffy wrote: On Sat, 2005-05-28 at 09:13 +0200, Christoph Hellwig wrote: On Fri, May 27, 2005 at 03:56:58PM -0700, Bob Woodruff wrote: kDAPL is intended as a kernel-level API for RDMA enabled fabrics. As it was initially written, it does not meet the Linux coding style and that is why it is being totally reworked as we speak to meet that goal. The codingstyle alone isn't the problem. The whole design philosophy is rather odd. As one of the people trying to clean up kDAPL, I would like to know what you think, from a design philosophy, is wrong with it. We *can* correct any daim bramaged parts. Well, from a kernel API design philosophy the evd is somewhat odd. The whole idea behind the event model seems a bit convoluted. First multiplex a wide variety of events from the provider into a single event queue, and then have an API so the consumer can tell what type of event they actually have and can still receive the event notification in the provider's context. This seems to be a lot of work to first hide useful information, but also not loose the information in case the consumer really does want it. It appears to be a case of a decent userspace idea that doesn't make much sense in the kernel. Why is it there? I imagine it's to abstract a variety of OS kernels, which was one of the goals of the design. Also, I realize it's just an implementation detail, but I've got a number of issues with ATS. -Libor ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux
I've been advocating rdmaconsortium folks submit patches against openib.org for several reasons: Probably, you meant openrdma.org opensource project but not a standards setting body (i.e. RDMA consortium - http://www.rdmaconsortium.org/home) :) 1) start with a code base that works 2) start with a code base that is already upstream 3) get advice/guidance from people who know how to collaborate in an open source environment. I thought (2) was the most important...but now I have to wonder if it's really (3). You are mistaken. I know people in the OpenRDMA community have worked with the opensource projects before and they know how to play and collaborate in an open source environment. The early part of the work in openrdma is in fact, a true example of that effort (which you may disagree with but having worked with several other opensource projects and with OpenIB, we have solved the issues which other projects including OpenIB have faced) and the next phase of work which is of course the code development, a key aspect of broader community effort. I think we are diverging from the real issue - the fundamental differences in the views of each community in how we can solve this common problem of supporting multiple RDMA fabrics, which is what we need to focus on. Just having OpenIB subsume control of anything iWARP or impose only DAPL for all RDMA infrastructure because it just happens to be there today seems rather stifling. Just stating that some OpenIB steering group is somehow empowered to decide this for Linux is also rather strange. AFAICT the openib.org steering group doesn't control the content of the svn.openib.org source tree. It manages things like web content, overall charter, etc Don't agree. If you have read the email thread on this discussion, you would find that steering committee need to decide whether openIB should work on including the support for iWARP. Not that I am supporting this idea -:) In the opensource world, developers should/will have the freedom to add what they want to do but of course, the acceptance of their contributions into mainline is completely a different matter. Thanks Venkat___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux
On Tue, 2005-05-31 at 16:43 -0700, Venkata Jagana wrote: AFAICT the openib.org steering group doesn't control the content of the svn.openib.org source tree. It manages things like web content, overall charter, etc Don't agree. If you have read the email thread on this discussion, you would find that steering committee need to decide whether openIB should work on including the support for iWARP. Not that I am supporting this idea -:) Please don't confuse the development effort going on this list (openib-general) with the corporation that sponsors the development. OpenIB as a (non-profit) corporation is setup with a charter and has bylaws, etc. Its goal may be IB specific at the moment. But, the developers and the development on this list and in the subversion repository don't answer to the OpenIB board of directors. Developers are free to write whatever code they chose to. There is nothing stopping the maintainers from taking iWARP patches *today*. OpenIB, as a corporation, may not put its name behind the work, but that is another matter. If the board of directors of OpenIB decide to cease sponsorship because the developers don't jive with its corporate goals, then development can continue elsewhere. -tduffy signature.asc Description: This is a digitally signed message part ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux
On Tue, May 31, 2005 at 04:43:58PM -0700, Venkata Jagana wrote: I've been advocating rdmaconsortium folks submit patches against openib.org for several reasons: Probably, you meant openrdma.org opensource project but not a standards setting body (i.e. RDMA consortium - http://www.rdmaconsortium.org/home) :) Yes - sorry. My bad. You are mistaken. I know people in the OpenRDMA community have worked with the opensource projects before and they know how to play and collaborate in an open source environment. I likely am. But comments about requiring commitment and business planning resources suggest otherwise. The early part of the work in openrdma is in fact, a true example of that effort (which you may disagree with but having worked with several other opensource projects and with OpenIB, we have solved the issues which other projects including OpenIB have faced) and the next phase of work which is of course the code development, a key aspect of broader community effort. Well, a true example of that effort would have included code. I think we are diverging from the real issue - the fundamental differences in the views of each community in how we can solve this common problem of supporting multiple RDMA fabrics, which is what we need to focus on. If there is a fundemental difference, it's something along the lines of: openrmda: Hey! We have this transport neutral RNIC PI spec that needs IB support! openib: Nice. Where is the code for iWarp? openrdma: Uhm, well, we've only written the spec so far. openib: Ok. What do you want from us? openrdma: Well, we want you to review this RNIC-PI spec and then write the code to support IB. openib: Are you crazy? We have a working implementation. And it's in kernel.org. openrmda: We know. That's why we should collaborate. RNIC PI spec is transport neutral. Could you review it and then implement it in openib.org? openib: No. You can submit patches and we'll review those. openrmda: Ok. But I'm not gonna write any code unless someone commits to accept it. We can't plan our business unless someone commits resources to work on accepting our patches. openib: No. You can submit patches and we'll review those. ... I'm trying to NOT be sarcastic - just summarize what I've understood so far. Please correct or post your own version (sans rude talk by certain people). ... Don't agree. If you have read the email thread on this discussion, you would find that steering committee need to decide whether openIB should work on including the support for iWARP. Not that I am supporting this idea -:) Tom answered this nicely already. In the opensource world, developers should/will have the freedom to add what they want to do Open source developers have *some* allegiance to their funders. HP pays me to look out for their interests - but I don't do that unconditionally. If an HP person is pushing for the wrong things that I know won't fly, I have an obligation to push back. but of course, the acceptance of their contributions into mainline is completely a different matter. We agree acceptance of contributions is conditional. But earlier emails stated someone needed a firm commitment that openrdma RNIC PI would get accepted into openib.org. There's a disconnect there. hth, grant ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux
On Fri, May 27, 2005 at 03:56:58PM -0700, Bob Woodruff wrote: Caitlin wrote, Both uDAPL and kDAPL were designed for *application* use. Even kDAPL is more intended for use by a kernel daemon that is loaded separately from the kernel than for use within the kernel itself. kDAPL is intended as a kernel-level API for RDMA enabled fabrics. As it was initially written, it does not meet the Linux coding style and that is why it is being totally reworked as we speak to meet that goal. The codingstyle alone isn't the problem. The whole design philosophy is rather odd. An ideal API for use within the kernel would abstract as much as possible (without requiring emulation), and then have transport specific unions or enum values. It would hide no control options, merely provide common controls for common capabilities. So for every new RDMA device type that comes along, you need to add a new enum, and unions for device class specific stuff, etc. Seems rather static and not easily extended. Not to mention that testing nightmare when the thing has to support 20 different types of RDMA enabled devices. I think code like that could get pretty ugly pretty fast. read the _as much as possible_ above. There's a reason you'll need new ABIs for the socket interface aswell when adding new address families. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux
On Fri, May 27, 2005 at 08:04:26PM -0700, Caitlin Bestler wrote: A commitment that after merger that the API will be transport neutral is essential. That is, IB HCAs do not have to emulate iWARP. iWARP HCAs do not have to emulate IB. You won't get any commitments from us at all. Is there anything in the general structure of RNIC-PI that you the API shows the usual signs of design by committee is is totally focused on tacking up separate propritary modules instead of focussing integrated stacks. It makes the usual mistake of mixing up user and kernel APIs, and the APIs are much too complicated. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux
--- Christoph Hellwig [EMAIL PROTECTED] wrote: On Fri, May 27, 2005 at 06:24:46PM -0700, Caitlin Bestler wrote: Oh that's a great development plan for me to propose to my business people. Let's go write lots of code, and *after* we write it the IB people (our competitors) have agreed to *look* at it (but they're all busy, don't forget) and *then* we just might get merged in. fortunately we don't care about your business people. either you do something sane, or your hardware won't we supported by linux at all. tough life. That's a pretty bold statement. Linux grew up to be popular via mass acceptance. Seems like that charter has changed and a few have control over Linux and its future. The My way or the highway philosophy has gotten embedded in the Linux way of life. Life is getting tough. --- This SF.Net email is sponsored by Yahoo. Introducing Yahoo! Search Developer Network - Create apps using Yahoo! Search APIs Find out how you can build Yahoo! directly into your own Applications - visit http://developer.yahoo.net/?fr=offad-ysdn-ostg-q22005 ___ Rdma-developers mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/rdma-developers __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux
On Sat, May 28, 2005 at 05:17:54AM -0700, Sukanta ganguly wrote: That's a pretty bold statement. Linux grew up to be popular via mass acceptance. Seems like that charter has changed and a few have control over Linux and its future. The My way or the highway philosophy has gotten embedded in the Linux way of life. Life is getting tough. You're totally missing the point. Linux is successfull exactly because it's lookinf for the right solution, not something the business people need short-term. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux
On 5/28/05, Christoph Hellwig [EMAIL PROTECTED] wrote: On Sat, May 28, 2005 at 05:17:54AM -0700, Sukanta ganguly wrote: That's a pretty bold statement. Linux grew up to be popular via mass acceptance. Seems like that charter has changed and a few have control over Linux and its future. The My way or the highway philosophy has gotten embedded in the Linux way of life. Life is getting tough. You're totally missing the point. Linux is successfull exactly because it's lookinf for the right solution, not something the business people need short-term. Such myopic cheerleading gets annoying and accomplishes nothing. The topic under discussion is whether a low level API for RDMA is necessary (as opposed to a higher level API such as kDAPL) and if so what the best strategy for achieving it is (try to plan an IB/iWARP merge immediately or wait until there is an iWARP code base). Claiming that an InfiniBand-specific interface is somehow thinking long term is just plain ludicrous. Now it may be that the short term interest of the InfiniBand vendors is such that they cannot commit resources to helping build a transport neutral API. That is always a legitimate tradeoff, but it is short term corporate thinking. Last time I looked most of the commits being made to OpenIB (or sourceforge DAPL) were from being drawing paychecks from those evil corporations. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux
On 5/27/05, Bob Woodruff [EMAIL PROTECTED] wrote:
So for every new RDMA device type that comes along, you need to add a new
enum, and unions for device class specific stuff, etc.
Seems rather static and not easily extended. Not
to mention that testing nightmare when the thing has to support
20 different types of RDMA enabled devices.
I think code like that could get pretty ugly pretty fast.
It would not be every new device, it would be every new RDMA
verbs layer API. Gen2 already covers the needs of InfiniBand.
RNIC-PI covers the needs of InfiniBand, iWARP/TCP and iWARP/SCTP.
I would not anticipate a high rate of additional RDMA verbs
appearing, particularly if the two existing verb APIs are unified.
Creating new verb APIs is something vendors do to enable new
markets, not because they have excess underutilized staff that
needs to be kept busy.
Well, who knows what vendors with excess underutilized staff do,
there haven't been sightings of such creatures in memory.
But in any event, that can be dealt with in one of two ways:
struct rdma_xyz_ib {
};
struct rdma_xyz_iwarp {
};
union rdma_xyz_xport {
struct rdma_xyz_ib ib;
struct rdma_xyz_iwarp iwarp;
};
struct rdma_xyz {
/* common fields */
union rdma_xyz_xport xport;
};
Verus..
struct rdma_xyz {
/* common fields */
};
struct rdma_xyz_ib {
struct rdma_xyz common;
/* ib fields */
};
struct rdma_xyz_iwarp {
struct rdma_xyz common;
/* iwarp fields */
};
The latter style is extensible, but makes it difficult to properly
allocate a buffer
that works for all variants.
The union style is also already in use in both IT-API and RNIC-PI.
I personally prefer sub-classing to unions, but I have found myself in the
minority on *most* projects where the issue has been discussed.
One reason is that sub-classing provides very little type-safety.
struct sockaddr is an example of this. It takes manual inspection
to ensure that the variants are properly differentiated and it is
still common for developers to pass in a plain struct sockaddr
without realizing that it is not large enough for a struct sockaddr_in6.
I'd rather see a registration mechanism like what we already have
with DAPL that does not require any code changes to add a new RDMA
device/provider. We have already proven that this works in DAPL
as I know if at least 3 providers, IB, Myrinet, and RNIC (Ammasso)
that were developed separately and were able to co-exist without
any changes (enums and device class unions) in the DAT mid-layer.
I assume that this can also be done with kDAPL in the kernel, but
I defer to the DAPL experts to answer that one.
There would be a registration method. And each device would load
a dynamic method table. What is in question is how many signatures
are supported.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [Rdma-developers] RE: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux
-Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Woodruff, Robert J Sent: Thursday, May 26, 2005 5:12 PM To: Caitlin Bestler Cc: Venkata Jagana; [EMAIL PROTECTED]; openib-general@openib.org Subject: [Rdma-developers] RE: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux RNIC-PI is at least an attempt at providing full control over both iWARP and IB while making as much common as possible. Where were you last year when the IB verbs header files/API were being discussed. Seems like that was the time to discuss new APIs. Now it is a bit late to propose something totally new like RNIC-PI. The best thing to do now is to try to influence the existing code base to meet your needs rather than recommend a totally new API. And the best way to do that is to make changes to the code that exists or write new code and send in patches to the list so that it can be discussed. woody OpenIB's announced charter was IB specific. There was no announcement that the intent was to define *all* RDMA services. Where were you when RNIC-PI was discussing these issues? It's announced charter *was* to be transport neutral and OS neutral. But that's water under the bridge. The fact is that both IB and iWARP vendors have existing development efforts and any realistic plan will recognize that. The question is to what degree code can be made common without requiring anyone to start over from scratch. That may involve merely defining critical interfaces with the rest of the OS, and then splitting into two transport dependent sub-systems. Or it may be possible to come up with a transport-neutral low level API. The latter goal is indeed desirable, but I am skeptical that it is compatible with the needs of either IB or iWARP vendors to be making forward progress on actual code today. It may make more sense to agree on the fundamental interfaces to the rest of the kernel, and on approximately what transport neutral verbs might look like someday, and then to develop in parallel until there is two branches of code that both work before attempting to merge them. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [Rdma-developers] RE: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux
On Fri, 2005-05-27 at 10:07 -0700, Caitlin Bestler wrote: OpenIB's announced charter was IB specific. There was no announcement that the intent was to define *all* RDMA services. The OpenIB board will be voting, maybe even at the next meeting, whether to expand OpenIB's charter to include *all* RDMA services. -tduffy signature.asc Description: This is a digitally signed message part ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux
Caitlin wrote, Both uDAPL and kDAPL were designed for *application* use. Even kDAPL is more intended for use by a kernel daemon that is loaded separately from the kernel than for use within the kernel itself. kDAPL is intended as a kernel-level API for RDMA enabled fabrics. As it was initially written, it does not meet the Linux coding style and that is why it is being totally reworked as we speak to meet that goal. An ideal API for use within the kernel would abstract as much as possible (without requiring emulation), and then have transport specific unions or enum values. It would hide no control options, merely provide common controls for common capabilities. So for every new RDMA device type that comes along, you need to add a new enum, and unions for device class specific stuff, etc. Seems rather static and not easily extended. Not to mention that testing nightmare when the thing has to support 20 different types of RDMA enabled devices. I think code like that could get pretty ugly pretty fast. I'd rather see a registration mechanism like what we already have with DAPL that does not require any code changes to add a new RDMA device/provider. We have already proven that this works in DAPL as I know if at least 3 providers, IB, Myrinet, and RNIC (Ammasso) that were developed separately and were able to co-exist without any changes (enums and device class unions) in the DAT mid-layer. I assume that this can also be done with kDAPL in the kernel, but I defer to the DAPL experts to answer that one. James/Arkady Comments ? woody ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux
-Original Message- From: Bob Woodruff [mailto:[EMAIL PROTECTED] Sent: Friday, May 27, 2005 3:57 PM To: Caitlin Bestler; Roland Dreier; 'James Lentini'; [EMAIL PROTECTED] Cc: Venkata Jagana; [EMAIL PROTECTED]; openib-general@openib.org Subject: RE: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux Caitlin wrote, Both uDAPL and kDAPL were designed for *application* use. Even kDAPL is more intended for use by a kernel daemon that is loaded separately from the kernel than for use within the kernel itself. kDAPL is intended as a kernel-level API for RDMA enabled fabrics. As it was initially written, it does not meet the Linux coding style and that is why it is being totally reworked as we speak to meet that goal. One of the primary features of both uDAPL and kDAPL is a unified event reporting mechanism. This feature was adopted by IT-API as well. The intent of this feature, which I am mildly familiar with, was to simplify the writing of applications -- whether those applications were written to run in user or kernel space. This unification is nice, and in my opinion vital, for applications. Roland had properly pointed out that some kDAPL features go beyond what is stricly necessary to achieve transport neutrality. I agree. Event unification is extremely valuable *for applications*, but it is not *necessary* to provide a transport neutral definition of RDMA services *within* the kernel. In an ideal world the in-kernel API would provide transport and vendor neutral definitions of reliable RDMA services, and as little else as possible. That said, if I were developing a kernel daemon today I would definitely use kDAPL. It is defined today, and its extras do not cost all that much, and largely duplicate things that my application would have to do anyway. An ideal API for use within the kernel would abstract as much as possible (without requiring emulation), and then have transport specific unions or enum values. It would hide no control options, merely provide common controls for common capabilities. So for every new RDMA device type that comes along, you need to add a new enum, and unions for device class specific stuff, etc. Seems rather static and not easily extended. Not to mention that testing nightmare when the thing has to support 20 different types of RDMA enabled devices. I think code like that could get pretty ugly pretty fast. I'd rather see a registration mechanism like what we already have with DAPL that does not require any code changes to add a new RDMA device/provider. We have already proven that this works in DAPL as I know if at least 3 providers, IB, Myrinet, and RNIC (Ammasso) that were developed separately and were able to co-exist without any changes (enums and device class unions) in the DAT mid-layer. I assume that this can also be done with kDAPL in the kernel, but I defer to the DAPL experts to answer that one. We are discussing a low level interface to be used by the most privileged code within a system. Controls should not be hidden from it. Each transport should be expected to standardize their own transport specific controls, and should greatly minimize them and use controls defined in RDMA terms as much as possible. But there are InfiniBand-specific control options required to fully control an InfiniBand HCA, and there are iWARP-specific controls required to fully control an iWARP RNIC. If you don't like it, go get the protocols respecified. The DAPL code actually required *extensive* changes to support iWARP (I did the first round of them myself). And the glue layer required for an iWARP vendor is extensive precisely because it must pretend to be an InfiniBand device to Sourceforge common code. Implementing iWARP under an API designed for InfiniBand is a mistake I do not wish to repeat. The sourceforge DAPL requires extensive parallel data between the DAPL layer and the verbs layer, that has a measurable impact on system performance. RNIC-PI not only avoids requiring either IB HCAs to pretend to be iWARP RNICs, or iWARP RNICs to pretend to be IB HCAs, it also provides features such as kernel mode completions and 'os_data' markers that eliminate the need for parallel DAPl/verbs data structures. Once this lowest-possible-RDMA-API is defined it will make it possible for *most* applications to work with only transport neutral fields and enums, and virtually all applications to do so for their non-error paths. But such an API is not a few minor tweaks away from the Gen2 verbs. Trying to sweep the differences under the rug in a low level API is what produces truly ugly code. But once there is an error, one of the requirements of a as low as possible API is that it not conceal data from the kernel. That means there will be transport dependencies hidden in unions. It's inescapable. ___
RE: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux
Catlin wrote, The sourceforge DAPL requires extensive parallel data between the DAPL layer and the verbs layer, that has a measurable impact on system performance. RNIC-PI not only avoids requiring either IB HCAs to pretend to be iWARP RNICs, or iWARP RNICs to pretend to be IB HCAs, it also provides features such as kernel mode completions and 'os_data' markers that eliminate the need for parallel DAPl/verbs data structures. Once this lowest-possible-RDMA-API is defined it will make it possible for *most* applications to work with only transport neutral fields and enums, and virtually all applications to do so for their non-error paths. But such an API is not a few minor tweaks away from the Gen2 verbs. Trying to sweep the differences under the rug in a low level API is what produces truly ugly code. The IB companies are not going to throw away a working implementation and move to RNIC-PI, but people have already said that they are willing to work with the iWarp community to add support and do what is right for Linux. However, as Roland said earlier, Without seeing some real patches from the iWARP side, it's hard for me to see any value in continuing to participate in this debate. - R. woody ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux
-Original Message- From: Bob Woodruff [mailto:[EMAIL PROTECTED] Sent: Friday, May 27, 2005 5:36 PM To: Caitlin Bestler; Roland Dreier; James Lentini; [EMAIL PROTECTED] Cc: Venkata Jagana; [EMAIL PROTECTED]; openib-general@openib.org Subject: RE: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux However, as Roland said earlier, Without seeing some real patches from the iWARP side, it's hard for me to see any value in continuing to participate in this debate. - R. Oh that's a great development plan for me to propose to my business people. Let's go write lots of code, and *after* we write it the IB people (our competitors) have agreed to *look* at it (but they're all busy, don't forget) and *then* we just might get merged in. If nobody is willing to discuss what the characteristics of this merged API will look like *before* we have to code it then there is no motivation to work inside OpenIB at all. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux
On Fri, May 27, 2005 at 06:24:46PM -0700, Caitlin Bestler wrote: Oh that's a great development plan for me to propose to my business people. Let's go write lots of code, and *after* we write it the IB people (our competitors) have agreed to *look* at it (but they're all busy, don't forget) and *then* we just might get merged in. You will never get a guarantee your code will go into kernel.org regardless of which path you take. I've seen three of the lead developers (a) promise to review any proposed code changes and (b) indicate they are willing to merge in support for RNICs. That's more than most people get. If nobody is willing to discuss what the characteristics of this merged API will look like *before* we have to code it then there is no motivation to work inside OpenIB at all. The openib.org APIs today only exists as code. Bob Woodruff is writing a paper documenting some of the interfaces but it's not done yet. Code is the primary currency to exchange ideas in open source forums. It doesn't have to be working code - psuedo code is probably sufficient to start with in this case. Look at the relevant openib.org header files (e.g. ib_verbs.h, ib_user_verbs.h) and make a list of changes needed for it to be useful to an RNIC. Just pointing at RNIC-PI isn't interesting. hth, grant ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux
-Original Message-
From: Grant Grundler [mailto:[EMAIL PROTECTED]
Sent: Friday, May 27, 2005 7:15 PM
To: Caitlin Bestler
Cc: Bob Woodruff; Roland Dreier; James Lentini;
[EMAIL PROTECTED]; Venkata Jagana;
[EMAIL PROTECTED]; openib-general@openib.org
Subject: Re: [Rdma-developers] Re: [openib-general] OpenIB
and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux
On Fri, May 27, 2005 at 06:24:46PM -0700, Caitlin Bestler wrote:
Oh that's a great development plan for me to propose to my business
people. Let's go write lots of code, and *after* we write it the IB
people (our competitors) have agreed to *look* at it (but
they're all
busy, don't
forget) and *then* we just might get merged in.
You will never get a guarantee your code will go into
kernel.org regardless of which path you take.
I've seen three of the lead developers (a) promise to review
any proposed code changes and (b) indicate they are willing
to merge in support for RNICs.
A commitment that after merger that the API will be transport
neutral is essential. That is, IB HCAs do not have to emulate
iWARP. iWARP HCAs do not have to emulate IB.
I fail to see why an advance agreement about the goal cannot
be reached. The iWARP vendors are willing to concede that
current Gen2 verb modules still have to be supported in a
merged solution.
What is your definiton of a merged solution? Is there a commitment
that RDMA Services does not mean InfiniBand, and the iWARP
vendors have to pretend to be InfiniBand but we'll add one or
two fields for problems they *can't* solve.
Look at the relevant openib.org header files (e.g. ib_verbs.h,
ib_user_verbs.h) and make a list of changes needed for it to
be useful to an RNIC. Just pointing at RNIC-PI isn't interesting.
Well since we are talking about what a final transport neutral
API would look like, as a guide when writing new code that will
be compatible with the final merger, then looking at an example
of an API that attempts to be transport neutral strikes me as
highly relevant.
Is there anything in the general structure of RNIC-PI that you
do not like? The IB-specific portions are different than Gen2,
and that certainly would not be true of any final merged verbs.
So far the only comments received are that RNIC-PI was not written
to be inside the Linux Kernel. That's correct, it was designed to
be an OS neutral API that could be supported on a wide variety
of Operating systems. Changes *have* been made, such as always
having struct/enum/union tags, precisely to guarantee that a
Linux version of rnicpi.h can be written that will be interoperable
with code written to the man pages.
As for gen2, here's one minor example:
struct ib_qp_attr {
enum ib_qp_stateqp_state; // States
defined are IB specific
enum ib_qp_statecur_qp_state;
enum ib_mtu path_mtu; // The path MTU
is not a QP attribute in iWARP,
// The LLP
maintains that. The maximum DDP Segment
// size is
exported to the DDP layer, but not to
// the consumer.
enum ib_mig_state path_mig_state; // Path migration is
handled at L2 or L3, it is
// not visisble
to an iWARP QP.
u32 qkey; // There is no
qkey
u32 rq_psn; // There is no psn.
There is an MSN for send/recvs.
// It is not
normally exported to the consumer.
u32 sq_psn; // There is no psn.
u32 dest_qp_num;// The destination QP is
not known.
int qp_access_flags; // presumably mostly
compatible, but its not documented
// in the .h
file
struct ib_qp_capcap;// different
capacities are controllable between
// iWARP and IB.
RNIC-PI defined the union of the
// two, and left
it to the providers to simply replicate
// counts that
are redundant under their transport.
// Example:
RDMAC allows different sge capacities
// depending on
the type of work request (send vs. write).
struct ib_ah_attr ah_attr;// entirely IB
specific information. TCP equivalents are
// not part of
QP state.
struct ib_ah_attr alt_ah_attr;// see ah_attr
Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux
On Fri, May 27, 2005 at 08:04:26PM -0700, Caitlin Bestler wrote:
You will never get a guarantee your code will go into
kernel.org regardless of which path you take.
I've seen three of the lead developers (a) promise to review
any proposed code changes and (b) indicate they are willing
to merge in support for RNICs.
A commitment that after merger that the API will be transport
neutral is essential. That is, IB HCAs do not have to emulate
iWARP. iWARP HCAs do not have to emulate IB.
commitment is a heavy word. One that should only be
used to describe individual actions in an open source forum.
I think you've already got agreement (not commitment)
that this is a reasonable goal - it totally depends on
what the proposed patches look like and if the person doing
the work is trustworthy and competent. The folks running the
openib source tree are a very accomodating and helpful crowd.
Don't get discourage by initial negative feedback. Once you
get it (process wise) things should move pretty quickly.
Sounds like you've never worked on an opensource project
before. Have you read Documentation/SubmittingPatches
in your favorite source tree?
Do you want other pointers on how to collaborate on
open source projects?
I fail to see why an advance agreement about the goal cannot
be reached. The iWARP vendors are willing to concede that
current Gen2 verb modules still have to be supported in a
merged solution.
That's a good first step. I think you've already gotten
as much advance agreement as anyone is willing to give.
Trust me, it doesn't get much better than what you and
Venkata (IBM) have been promised already.
What is your definiton of a merged solution?
One that starts with a code change proposal. Start with
any existing file in openib.org tree and suggest changes
that would make it fit your needs better.
The more specific, the better.
One piece at time.
Is there a commitment
that RDMA Services does not mean InfiniBand, and the iWARP
vendors have to pretend to be InfiniBand but we'll add one or
two fields for problems they *can't* solve.
No. This is open source. We depend on maintainers to use good
judgement and listen to other people. Projects without such
leaders don't survive. Linus has a cute, short essay on
that in Documentation/ManagementStyle. Please read it.
It's quite amusing and insightful at the same time.
Look at the relevant openib.org header files (e.g. ib_verbs.h,
ib_user_verbs.h) and make a list of changes needed for it to
be useful to an RNIC. Just pointing at RNIC-PI isn't interesting.
Well since we are talking about what a final transport neutral
API would look like, as a guide when writing new code that will
be compatible with the final merger, then looking at an example
of an API that attempts to be transport neutral strikes me as
highly relevant.
Uhm, sort of. Yes, in that the design probably has relevance.
The implementation less so. This discussion is about starting
with openib gen2, right?
If it's not, I'm wasting my time.
Is there anything in the general structure of RNIC-PI that you
do not like? The IB-specific portions are different than Gen2,
and that certainly would not be true of any final merged verbs.
So far the only comments received are that RNIC-PI was not written
to be inside the Linux Kernel. That's correct, it was designed to
be an OS neutral API that could be supported on a wide variety
of Operating systems. Changes *have* been made, such as always
having struct/enum/union tags, precisely to guarantee that a
Linux version of rnicpi.h can be written that will be interoperable
with code written to the man pages.
I barely have time to keep up with code changes going on
in openib.org tree and don't understand details of 1/2 of those.
I'm the wrong person to ask even about specifics in openib.org.
My expertise is not in IB transport (it's PCI chipsets and
platform IRQ/DMA support.)
I've worked full time on open source projects for the past 5 years and
have a clue how they do (and don't) work. Some definitely work better
than others. It always revolves around the people leading the project.
openib.org is doing fairly well right now IMHO.
As for gen2, here's one minor example:
Good - that's a start.
ISTR Someone already pointed out some attr fields need to change.
struct ib_qp_attr {
enum ib_qp_stateqp_state; // States
defined are IB specific
Please don't use C++ style comments.
I talk about how this field (and others) *might* be organized below.
enum ib_qp_statecur_qp_state;
enum ib_mtu path_mtu; // The path MTU
is not a QP attribute in iWARP,
It's ok if *some* fields are unused I think.
(not that this needs to be here)
// The LLP
maintains that. The maximum DDP Segment
// size is
exported to the DDP layer, but
Re: [Rdma-developers] RE: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux
At 09:29 AM 5/26/2005, Talpey, Thomas wrote: At 11:49 AM 5/26/2005, Bob Woodruff wrote: Finally, until there is some consensus about allowing TCP offload in Linux, I see no need to start to hack up the InfiniBand stack to support iWarp. It is not a requirement that TCP offload be supported in order to support iWARP. Upper layers which don't perform an rdma-upgrade (e.g. NFSv3/RDMA) don't even care about speaking sockets. TOE != Sockets. TOE is simply the TCP/IP protocol stack that is off-loaded for any TCP-based application / sub-system to use. And I believe a Connection Manager can and should address this for the ones that do. Tying iWARP to the TOE albatross furthers nothing. iWARP != TOE though one uses an integrated TOE in a RNIC if you want performance. Connection management is unique to IB as it is to TCP. It must exist as a separate interface for subsystems to use. Whether one can abstract a common API or just accept that IB and TCP are different and call accordingly once understanding the underlying hardware is debatable. I completely agree with your other point that a lower-layer cut-in is preferable to another top-layer API. We already have APIs which provide transport-independent RDMA services to existing upper layers (such as NFS/RDMA and iSER, coded in fact to kDAPL). kDAPL is being used by some sub-systems. One can argue that a verbs interface is a cleaner way to get directly to the hardware without requiring an intermediary API like DAPL (user or kernel). Caitlin will provide some good counters why this isn't necessarily perfect. Both sides have valid points in the debate. In the end, it is what people implement that matter and whether that gets accepted by customers in the end as good enough technology. Just because there are some point-in-time implementations for NFS/RDMA and iSER does not mean that kDAPL is the right or only answer. They are only what has been implemented to date. If someone builds a better implementation using another interface, then that could easily replace what is there today. The RNIC PI or IB verbs interfaces articulate the intent of those of us who created the verbs in the first place. We intentionally left it open as to whether further abstraction was required such as IT API or DAPL. We did intend for subsystems like MPI, SDP, iSER, etc. to be able to use the verbs directly if desired. I think many of us viewed IT API and DAPL as a potential point of abstraction to insert other value-add functionality that should be transparent to the applications but should not pollute the verbs, e.g. virtualization or specific types of load balancing or fail-over. Do we have solid justification to add more? NFS/RDMA says no. I don't believe any one person speaks for a given technology especially in an open source world. In fact, I suspect you'll find a large number of people who will disagree with this assertion as well. :) * Feel free to ignore the following * I also find it unprofessional some of the diatribes that occur at times on these reflectors. It would be more constructive if people focused on technical issues instead just painting broad negative statements. We all have opinions on how things should be developed. Keeping things objective is much more constructive. I always try to keep my responses focused on whether I would appreciate such a response or whether the response tone / language would be acceptable at my place of work. Colorful language, unnecessary inflammatory statements, etc. result in people loosing credibility and thus effectiveness in any environment. The worse thing for any engineer / professional is to be simply written off as a hot-head who is no longer being heard by any. Regards, Mike ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
