Re: [PATCH 1/2] libibverbs: Allow 3rd party extensions to verb routines
On 1/2/2012 7:34 PM, Hefty, Sean wrote: this field is a -- union -- how would that work if more than one extension is to be applied for a structure? The fields at the end of the structure should only be accessed if the structure is of the correct type. In this case, ext.xrc_recv is only available if the qp type is xrc recv. Maybe I wasn't clear enough, I was thinking on a case where one extends a structure and later on a 2nd extension is applied to the same structure, but the functionality / list of use cased related to the 1st extension isn't disjoint with that of the 2nd, e.g the second extends the first... maybe generally we can have a union named ext and on rare cases added ext2, etc? 2. indeed, reality wise, new features, much of the time will also interact with existing data structures... so what happens if we extend a structure but the the extended strucutre is actually a field within another existing structure e.g suppose we want to extend ibv_ah_attr which is within ibv_qp_attr e.g to be used for the RAW Ethernet QPs. I don't see how we can be backward compatible with apps linked against libibverbs with the internal structure size being smaller, correct? so extended fields must be on the END always - in the actual structure they are added and if this structure is a field of another structure then we can't just extend it and need to come up with new structure which is in turn used as the field? New features want to interact with existing structures and functions, which is what makes providing a clean separation difficult. We can extend the structures using the above method as long as we have some sort type field available. Where one is not available, we need to add one. See the proposed struct ibv_srq for an example. The extended SRQ type is only available by calling ibv_create_xsrq(), since ibv_create_srq() cannot know whether the user supports the extended ibv_srq_init_attr or not. Yes, I understand that, in 2nd thought, for the case of extending a structure which sits within another structure, e.g ibv_ah_attr within ibv_qp_attr, maybe we don't have much choice and rather add ib_ah_ext_attr and place it in the end of ibv_qp_attr --- and here's a case for double extensions... as we need the ext union of ibv_qp_attr to contain struct ibv_ah_ext_attr prim , struct ibv_ah_ext_attr alt and possibly also struct qp_attr_ext qp_ext... ibv_wc and ibv_send_wr are allocated by the caller, so those are more difficult to deal with. I agree that the size of those structures cannot change. It may be possible that some of the features you mentioned could be set as part of the qp attributes (ibv_modify_qp), and for the others, I'm not sure. Run time checks shouldn't be a big deal, since we already have to check things like ibv_wr_opcode and ibv_send_flags anyway. But it could be that we require a new function, similar to ibv_create_xsrq. I'd like to better understand the allocated by the caller ... are more difficult to deal with part of your response - for ibv_send_wr - if the caller have set a new IBV_WR_NEW_FEATURE value for the wr type, they surely aware to the new fields and actually the size of the structure can change as of structs allocated by the library. As for ibv_wc, yep, looks like we can't change the size unless we want to write a copatility layer that also comes into play in fast path calls, specifically ibv_poll_cq and translates from the new ibv_wc to the old ibv_wc structure. Or. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH 1/2] libibverbs: Allow 3rd party extensions to verb routines
I'd like to better understand the allocated by the caller ... are more difficult to deal with part of your response - for ibv_send_wr - if the caller have set a new IBV_WR_NEW_FEATURE value for the wr type, they surely aware to the new fields and actually the size of the structure can change as of structs allocated by the library. As for ibv_wc, yep, looks like we can't change the size unless we want to write a copatility layer that also comes into play in fast path calls, specifically ibv_poll_cq and translates from the new ibv_wc to the old ibv_wc structure. You're right. I was thinking more of ibv_wc, which has issues, since it may be used as an array. ibv_send_wr is probably okay, since we walk a list using pointers. - Sean -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH 1/2] libibverbs: Allow 3rd party extensions to verb routines
1. for libibverbs some structures extended field is added at their end
saying Following fields only available if device supports extensionse.g
@@ -590,6 +634,13 @@ struct ibv_qp {
pthread_mutex_t mutex;
pthread_cond_t cond;
uint32_tevents_completed;
+
+ /* Following fields only available if device supports
extensions */
+ union {
+ struct {
+ struct ibv_xrcd *xrcd;
+ } xrc_recv;
+ } ext;
};
but this field is a -- union -- how would that work if more than one
extension is to be applied for a structure?
The fields at the end of the structure should only be accessed if the structure
is of the correct type. In this case, ext.xrc_recv is only available if the qp
type is xrc recv.
2. indeed, reality wise, new features, much of the time will also
interact with existing data structures... so what happens if we extend a
structure but the the extended strucutre is actually a field within
another existing structure e.g suppose we want to extend ibv_ah_attr
which is within ibv_qp_attr e.g to be used for the RAW Ethernet QPs. I
don't see how we can be backward compatible with apps linked against
libibverbs with the internal structure size being smaller, correct? so
extended fields must be on the END always - in the actual structure they
are added and if this structure is a field of another structure then we
can't just extend it and need to come up with new structure which is in
turn used as the field?
New features want to interact with existing structures and functions, which is
what makes providing a clean separation difficult. We can extend the
structures using the above method as long as we have some sort type field
available. Where one is not available, we need to add one. See the proposed
struct ibv_srq for an example. The extended SRQ type is only available by
calling ibv_create_xsrq(), since ibv_create_srq() cannot know whether the user
supports the extended ibv_srq_init_attr or not.
3. The usage of #ifdef IBV_NEW_FEATURE_OPS (IBV_XRC_OPS) in libmlx4
will become tireding to follow maybe and could be replaced by placing
dependency on a version of libibverbs that supports IBV_NEW_FEATURE_OPS?
Yes, you could make that replacement, which will work in this case.
4. can we somehow come up with a method to avoid IBV_NEW_FEATURE_OPS for
--every-- new feature?
or call an existing function with some new enum value
This probably depends on the feature.
5. what happens if we just want to enhance an -- existing -- function -
suppose we want to enhance ibv_post_send / ibv_poll_cq to support
features like LSO, checksum offload, masked atomic operations, fast
memory remote invalidate, etc so we add IBV_WR_NEW_FEATURE /
IB_WC_NEW_FEATURE enum values, this step is simple, again if we go the
way of the applicayion ensuring through dependencies that they are
loaded against libibverbs that supports IB_{WR,WC}_NEW_FEATURE.
Now we come up with a need to extend struct ibv_send_wr and struct
ibv_wc - where we should be very careful - walking on glasses for the
backward compatibility thing - so only adding at the end on as union
field which doesn't change the size of the union, correct? for
ibv_post_send we can rely on the provider library to return error if
they don't support NEW_FEATURE, but this would happen in run time... is
there a way to avoid that, should we better go and add
ib_post_send_new_feature? this would be very tedious doing for each
new_feature and not applicable to ibv_poll_cq - any idea what should be
done here?
ibv_wc and ibv_send_wr are allocated by the caller, so those are more difficult
to deal with. I agree that the size of those structures cannot change. It may
be possible that some of the features you mentioned could be set as part of the
qp attributes (ibv_modify_qp), and for the others, I'm not sure. Run time
checks shouldn't be a big deal, since we already have to check things like
ibv_wr_opcode and ibv_send_flags anyway. But it could be that we require a new
function, similar to ibv_create_xsrq.
- Sean
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 1/2] libibverbs: Allow 3rd party extensions to verb routines
On Thu, Dec 29, 2011 at 3:43 PM, Or Gerlitz ogerl...@mellanox.com wrote:
5. what happens if we just want to enhance an -- existing -- function -
suppose we want to enhance ibv_post_send / ibv_poll_cq to support features
like LSO, checksum offload, masked atomic operations, fast memory remote
invalidate, etc so we add IBV_WR_NEW_FEATURE / IB_WC_NEW_FEATURE enum
values, this step is simple, again if we go the way of the applicayion
ensuring through dependencies that they are loaded against libibverbs that
supports IB_{WR,WC}_NEW_FEATURE.
Not only for the ABI between the Linux kernel and Linux user space but
also for Linux shared libraries it is required that the ABI of a new
version is backwards compatible with previous versions. So any
functionality that can't be added in a backwards compatible way to an
existing function should be added as a new system call, sysfs file or
shared library function - whatever is appropriate.
Bart.
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 1/2] libibverbs: Allow 3rd party extensions to verb routines
or call an existing function with some new enum value
5. what happens if we just want to enhance an -- existing -- function
- suppose we want to enhance ibv_post_send / ibv_poll_cq to support
features like LSO, checksum offload, masked atomic operations, fast
memory remote invalidate, etc so we add IBV_WR_NEW_FEATURE /
IB_WC_NEW_FEATURE enum values, this step is simple, again if we go the
way of the applicayion ensuring through dependencies that they are
loaded against libibverbs that supports IB_{WR,WC}_NEW_FEATURE.
Another element to take into account / use here, is the device
capabilities, in the kernel RDMA stack this is the way to go for
supporting extended features - those who are not common to all devices,
e.g the way ipoib queries the device and concludes if checksum/lso
offloads can be exposed to the network stack, or the rpc/rdma layer
decides what memory management model to use for rdma conditioned if
memory extensions etc are supported. But the kernel is one piece, so if
a cap is defined and there's a driver down there advertizing it, all is
in place software wise, where in user space, there are some more layers
to address, as this thread attempts to do.
Or.
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 1/2] libibverbs: Allow 3rd party extensions to verb routines
On Tuesday 02 August 2011 08:38, Jason Gunthorpe wrote: The hope is once the infrastructure is in libibverbs there will not be as much need to change libibverbs, just the apps and the drivers. If I understand correctly, the various additions which would normally be made to libibverbs would then be made by third-party libraries which extend libibverbs to support their additions. These additions would include the new ibv_cmd_xxx functions (the core functions reside in src/cmd.c), and new, additional, enum values of the form IBV_USER_VERBS_CMD_, of which the core enum is in file include/infiniband/kern_abi.h. The modified apps would then include the header files of the 3rd party additions after the libibverbs headers when compiling. Each new third-party package would need such a library. While this will lead to a multiplicity of new libraries (one per addition), the core libibverbs package would remain as is. Am I correct? If so, shouldn't the current XRC userspace implementation do the same (and take the XRC-specific additions out of libibverbs and put them into a separate library)? Note that coordination between third parties would still be required to insure that there is no collision of enum values between the various packages. -Jack -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH 1/2] libibverbs: Allow 3rd party extensions to verb routines
If I understand correctly, the various additions which would normally be made
to libibverbs
would then be made by third-party libraries which extend libibverbs to support
their additions.
It may help to read about extensions in opengl:
http://www.opengl.org/registry/doc/rules.html
Additions can be made to both. Obviously Roland has the final say on any
changes to ibverbs, but what I envision is:
If a feature is based on an industry standard and all necessary kernel changes
are upstream, then the feature should be integrated into ibverbs. An
application would simply call ibv_new_feature() to make use of the feature (or
call an existing function with some new enum value). Internally, ibverbs may
need to obtain a new interface from the provider library.
If there is no published specification for a feature (maybe it's still under
development), kernel patches are needed, and there are customers who want to
use the feature immediately, then a vendor can define an extension. In this
case, the application may call vendor_ops = ibv_get_ext_ops(), followed by
vendor_ops-new_feature(). Or the app may call ibv_some_existing_function()
using a vendor specific enum value.
These additions would include the new ibv_cmd_xxx functions
(the core functions reside in src/cmd.c), and new, additional, enum values of
the form IBV_USER_VERBS_CMD_, of which the core enum is in file
include/infiniband/kern_abi.h.
If an app uses an extension, then there are no changes to ibverbs. The
provider library either needs to use an existing ibv_cmd_* call or call into
the kernel itself. If the provider needs a new command, it could declare it as:
enum {
MLX4_USER_VERBS_CMD_BASE = IBV_EXTENSION_VENDOR
IBV_EXTENSION_BASE_SHIFT,
MLX4_USER_VERBS_CMD_NEW_FEATURE ...
};
This requires a kernel patch to uverbs maintained by the vendor. Note that
this means that the vendor can continue to support their version of a feature
(with continued kernel patches) even once the feature is merged into ibverbs.
Each new third-party package would need such a library.
While this will lead to a multiplicity of new libraries (one per addition),
the core libibverbs package would remain as is.
I didn't follow this. The new feature could be integrated directly into the
provider library (e.g. mlx4).
Am I correct? If so, shouldn't the current XRC userspace implementation do
the same (and
take the XRC-specific additions out of libibverbs and put them into a separate
library)?
XRC could be added as an extension, but since it's based on a published
specification, IMO it makes more sense being integrated directly into ibverbs
with the necessary kernel changes pushed upstream. Extensions are more
difficult for apps to use than an integrated feature.
Note that coordination between third parties would still be required to insure
that there is
no collision of enum values between the various packages.
The use of enum ibv_extension_type should prevent collisions. It may be that
different vendors use the same value for different objects, but that doesn't
result in a collision. The scope of a vendor specific value is per device.
- Sean
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 1/2] libibverbs: Allow 3rd party extensions to verb routines
On Tue, Aug 02, 2011 at 10:53:05AM +0300, Jack Morgenstein wrote: These additions would include the new ibv_cmd_xxx functions (the core functions reside in src/cmd.c), and new, additional, enum values of the form IBV_USER_VERBS_CMD_, of which the core enum is in file include/infiniband/kern_abi.h. I believe the only case where using an extension would make sense if it if can be implemented entirely within the low level driver. So you can't add new ibv_cmd calls to ibverbs, must duplicate them in your driver, etc. I'm a little unclear on how the application is going to get the access enums and other structure definitions, though.. I suppose the low level driver can install a .h file as well. Each new third-party package would need such a library. While this will lead to a multiplicity of new libraries (one per addition), the core libibverbs package would remain as is. I definitely don't want to see this.. Jason -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 1/2] libibverbs: Allow 3rd party extensions to verb routines
On Thursday 16 June 2011 22:27, Hefty, Sean wrote: Basically, I wasn't referring to such new verbs as vendor extensions, but rather as new verbs we want to add at this and/or future points of time which didn't exist at the time the IB stack and specifically its kernel/user ABIs/APIs were written (couple of years ago...). To be clear, there are 2 sides to ibverbs - the app side, and the provider library. On the app side, new functionality would be added directly to libibverbs. I would reuse what's there if possible, and provide direct API calls where needed. For example, the xrc patch adds: ibv_create_xsrq() ibv_open_xrcd() ibv_close_xrcd() as new APIs. On the provider side, the necessary calls are obtained by ibverbs calling get_ext_ops(). I haven't come up with another way of extended verbs that would be as easy for an application to use, given that most of the calls and data structures are reusable. - Sean Hi Sean, I'm not sure about what this mechanism saves us over bumping the ABI numbers. Actually, I think I do see a problem here, under the following situation: - All libraries (app, libibverbs, and libmlx4) support extensions. - A new verb (say extension ib_new_verb was added as an extension to the app, to libibverbs, and to libmlx4 and everything was compiled. - The APP is built on the full configuration, but is run on a configuration which has the verb extension added to libmlx4, but NOT to libibverbs (new libibverbs was installed originally, so that libmlx4 would succeed in the install, then somehow libibverbs was rolled back to before ib_new_verb was added). When the APP tries to run, it calls: ibv_get_device_ext_ops(struct ibv_device *device, ib_new_verb); This call will succeed (in the current implementation). However, the verb helper function (ibv_cmd_new_verb) is not present in libibverbs, so things will crash (if, indeed, libmlx4 can be loaded at all - In fact, I'm not sure if it will fail loading because of unresolved references). Indeed, I am not sure that the app can run at all due to unresolved references. The problem here is that the new additions need support in libibverbs in order to work (this is not simply a pass-through by libibverbs to the lower layer). I may be wrong about all this -- userspace is not really my expertise. If I am not wrong, what, then, is the advantage of this methodology over simply bumping the ABI numbers? -Jack -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [PATCH 1/2] libibverbs: Allow 3rd party extensions to verb routines
In fact, I'm not sure if it will fail loading because of unresolved references). Indeed, I am not sure that the app can run at all due to unresolved references. I'm not sure what exactly we can do in the situation you described. I would expect the app to see an unresolved reference. I may be wrong about all this -- userspace is not really my expertise. If I am not wrong, what, then, is the advantage of this methodology over simply bumping the ABI numbers? I'm not sure I fully understand your question. In an ideal case, any new feature would be added to the upstream ibverbs. The extensions would be used to support some new feature that ibverbs is not aware of. For example, a vendor could expose APIs that offload MPI collectives, which may still be under development. - Sean
Re: [PATCH 1/2] libibverbs: Allow 3rd party extensions to verb routines
In fact, I'm not sure if it will fail loading because of unresolved references). Indeed, I am not sure that the app can run at all due to unresolved references. The dlopen of mlx4 should fail due to unresolved references, so the net effect will be that no apps cannot open the RDMA device in this situation - so it is an invalid system configuration that is properly detected by the runtime. The problem here is that the new additions need support in libibverbs in order to work (this is not simply a pass-through by libibverbs to the lower layer). The hope is once the infrastructure is in libibverbs there will not be as much need to change libibverbs, just the apps and the drivers. Jason -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] [PATCH 1/2] libibverbs: Allow 3rd party extensions to verb routines
On Sat, Jun 04, 2011 at 12:34:46AM +, Hefty, Sean wrote: In order to support OFED or vendor specific calls, define a generic extension mechanism. This allows OFED, an RDMA vendor, or another registered 3rd party (for example, the librdmacm) to define RDMA extensions. After looking at this a bit I'm a bit concerned that your first extension goes around and requires edits to the various structs anyhow, which sort of defeats the purpose of making extensions, IMHO.. Do you think such changes will be required so often that this is just extra complexity? For instance, could you implement the multicast self loop flag feature using an extension ? Jason -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [RFC] [PATCH 1/2] libibverbs: Allow 3rd party extensions to verb routines
After looking at this a bit I'm a bit concerned that your first extension goes around and requires edits to the various structs anyhow, which sort of defeats the purpose of making extensions, IMHO.. My thinking is that extensions mainly provide: - a way for a vendor or organization to provide add-on functionality to an application (so we don't break things like the xrc patches did) - a mechanism for libibverbs to obtain additional functionality from a provider library (where existing ibv_context_ops are insufficient) Once a feature has been added directly to libibverbs, I would treat it slightly differently. For example, I've since modified libibverbs to export direct APIs for opening/closing xrcd's and creating extended srq's. An app doesn't need to call ibv_get_ext_ops() directly for xrc qp's; however, it would make use of the define for IBV_XRC_OPS if it wanted to support older versions of libibverbs. Changing ibv_send_wr, ibv_qp_init_attr, ibv_srq, ibv_qp, etc. seemed like a much better alternative for providing XRC support than introducing an entire new set of structures and APIs. Do you think such changes will be required so often that this is just extra complexity? For instance, could you implement the multicast self loop flag feature using an extension ? Based on history, I'd anticipate a small number of extensions provided by OFA or a vendor: tag matching? off-loaded MPI collective operations? who knows... Certain ibverbs APIs more easily support extensions that others. Adding XRC QP support to the existing APIs without breaking existing apps is fairly easy, since the qp_type indicates if extended attributes are available. XRC SRQs required a new API. Supporting the multicast self loop flag is definitely doable; it's really just a matter of how easy it would be for an application to use it. :) - Sean -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] [PATCH 1/2] libibverbs: Allow 3rd party extensions to verb routines
On 06/03/2011 07:34 PM, Hefty, Sean wrote: In order to support OFED or vendor specific calls, define a generic extension mechanism. This allows OFED, an RDMA vendor, or another registered 3rd party (for example, the librdmacm) to define RDMA extensions. Will this mechanism allow an RDMA provider driver to export a new qp-related operation for use internally bit the supporting provider library? IE Not exposes to the RDMA application, but an internal interface between the library and driver. I have need for this with the T4 driver. Users which make use extensions are aware that they are not only using an extended call, but are given information regarding how widely the extension by be supported. Can you expand on the above sentence? I don't get the how widely supported angle? Steve. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] [PATCH 1/2] libibverbs: Allow 3rd party extensions to verb routines
On 06/06/2011 01:07 PM, Roland Dreier wrote: On Mon, Jun 6, 2011 at 9:30 AM, Steve Wisesw...@opengridcomputing.com wrote: Will this mechanism allow an RDMA provider driver to export a new qp-related operation for use internally bit the supporting provider library? IE Not exposes to the RDMA application, but an internal interface between the library and driver. I have need for this with the T4 driver. Not sure I follow this... how specifically would this work? Why does the userspace library need help to talk to the kernel driver? - R. I'm investigating ways to support a kernel mode ring the QP doorbell for user mode QPs. This can allow optimization and coalescing of db-credits to improve the ring rate for large amounts of qps. Currently for experimentation, I hacked this by posting a special send WR (with opcode 0xdeadbeef :) ) which ends up calling my db ring function, but I need to come up with an acceptable solution. I was thinking about maybe enhancing the modify_qp verb, but perhaps this new extension proposal is a better way? I was also pondering some sort of provider-specific ioctl. But this is really internal to the provider lib and driver. I just don't want to have to implement a full character device interface for this since the uverbs interface provides this already. Steve. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] [PATCH 1/2] libibverbs: Allow 3rd party extensions to verb routines
On 06/06/2011 01:28 PM, Hefty, Sean wrote: Will this mechanism allow an RDMA provider driver to export a new qp- related operation for use internally bit the supporting provider library? IE Not exposes to the RDMA application, but an internal interface between the library and driver. I have need for this with the T4 driver. Sorry about my bad English: internally bit the should be internally by the. And IE Not exposes should be IE Not exposed. I don't fully understand this request. The idea is that libibverbs does not change as new extensions are added by providers, and that there is only 1 version of libibverbs (from Roland's tree). This does not try to extend the kernel interfaces in any way. If kernel patches are required, my thinking is that the provider library should communicate directly with the patched kernel. That said, libibverbs _could_ obtain some sort of non-published interface to a provider and make use of it. Roland would need to accept any such patches. Btw, adding the ibv_extension_mask to kernel commands (ib_user_verbs_cmd_*), rather than simply taking the next value, should help avoid breaking the ABI when dealing with patched kernels. See my answer to Roland's question as to what I'm trying to do. I guess your proposal isn't what I'm needing... Users which make use extensions are aware that they are not only using an extended call, but are given information regarding how widely the extension by be supported. Can you expand on the above sentence? I don't get the how widely supported angle? The idea is that the extension name indicates if it's common to verbs, specific to a vendor, or supported by some external group, such as OFA. E.g. you can have vendor specific XRC ops, OFA XRC ops, or ibverbs XRC ops. - Sean I see. Thanks. Steve. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [RFC] [PATCH 1/2] libibverbs: Allow 3rd party extensions to verb routines
I'm not quite sure I understand the use for enum ibv_extension_type. Where/how is this used? The intent is that this allows users to extend existing enums without conflicts. For example, /* Extend IBV_QP_TYPE for XRC */ #define OFA_QPT_XRC ((enum ibv_qp_type) \ (IBV_EXTENSION_OFA IBV_EXTENSION_BASE_SHIFT) + 6) This defines a new QP type for XRC. This new QP type is only usable as part of an OFA specific extension. When XRC QP types are added directly to libibverbs, it receives a new number because there's no guarantee that the upstream version of XRC support will match what OFA published. (Obviously it's too late to handle XRC QPs in this way.) The trade-off is that the upper X number of bits (8 in the patch) of most enums end up being reserved for extension use. The benefit is that existing functions, like ibv_create_qp, can support extensions, versus duplicating functions and structures. - Sean -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC] [PATCH 1/2] libibverbs: Allow 3rd party extensions to verb routines
I'm not quite sure I understand the use for enum ibv_extension_type.
Where/how is this used?
Ira
On Jun 3, 2011, at 5:34 PM, Hefty, Sean wrote:
In order to support OFED or vendor specific calls, define a
generic extension mechanism. This allows OFED, an RDMA vendor,
or another registered 3rd party (for example, the librdmacm)
to define RDMA extensions.
Users which make use extensions are aware that they are not
only using an extended call, but are given information regarding
how widely the extension by be supported. Support for extended
functions, data structures, and enums are defined.
Extensions are referenced by name. There is an assumption that
extension names are prefixed relative to the supporting party.
Until an extension has been incorporated into libibverbs, it
should be defined in an appropriate external header file.
For example, OFA could provide a header file with their definition
for XRC extensions. A partial view of such a header file might
look something similar to:
#ifndef OFA_XRC_H
#define OFA_XRC_H
#include infiniband/verbs.h
#define OFA_XRC_OPS ofa-xrc
/* Extend IBV_QP_TYPE for XRC */
#define OFA_QPT_XRC ((enum ibv_qp_type) \
(IBV_EXTENSION_OFA IBV_EXTENSION_BASE_SHIFT) + 6)
struct ofa_xrcd {
struct ibv_context *context;
};
struct ofa_xrc_ops {
struct ofa_xrcd * (*open_xrcd)(struct ibv_context *context,
inf fd, int oflags);
int * (*close_xrcd)(struct ofa_xrcd *xrcd);
/* other functions left as exercise to the reader */
};
#endif /* OFA_XRC_H */
Driver libraries that support extensions are given a new
registration call, ibv_register_device_ext(). Use of this call
indicates to libibverbs that the library allocates extended
versions of struct ibv_device and struct ibv_context.
The following new APIs are added to libibverbs to applications
to use to determine if an extension is supported and to obtain the
extended function calls.
ibv_have_ext_ops - returns true if an extension is supported
ibv_get_device_ext_ops - return extended operations for a device
ibv_get_ext_ops - return extended operations for an open context
To maintain backwards compatibility with existing applications,
internally, the library uses the last byte of the device name
to record if the device was registered with extension support.
Signed-off-by: Sean Hefty sean.he...@intel.com
---
Compile tested only at this point. I'm still working on writing
an XRC sample program.
include/infiniband/driver.h |1 +
include/infiniband/verbs.h | 40 +++-
src/device.c| 18 ++
src/ibverbs.h | 18 ++
src/init.c | 17 -
src/libibverbs.map |5 +
src/verbs.c |9 +
7 files changed, 106 insertions(+), 2 deletions(-)
diff --git a/include/infiniband/driver.h b/include/infiniband/driver.h
index 9a81416..e48abfd 100644
--- a/include/infiniband/driver.h
+++ b/include/infiniband/driver.h
@@ -57,6 +57,7 @@ typedef struct ibv_device *(*ibv_driver_init_func)(const
char *uverbs_sys_path,
int abi_version);
void ibv_register_driver(const char *name, ibv_driver_init_func init_func);
+void ibv_register_driver_ext(const char *name, ibv_driver_init_func
init_func);
int ibv_cmd_get_context(struct ibv_context *context, struct ibv_get_context
*cmd,
size_t cmd_size, struct ibv_get_context_resp *resp,
size_t resp_size);
diff --git a/include/infiniband/verbs.h b/include/infiniband/verbs.h
index 0f1cb2e..b82cd3a 100644
--- a/include/infiniband/verbs.h
+++ b/include/infiniband/verbs.h
@@ -55,6 +55,15 @@
BEGIN_C_DECLS
+enum ibv_extension_type {
+ IBV_EXTENSION_COMMON,
+ IBV_EXTENSION_VENDOR,
+ IBV_EXTENSION_OFA,
+ IBV_EXTENSION_RDMA_CM
+};
+#define IBV_EXTENSION_BASE_SHIFT 24
+#define IBV_EXTENSION_MASK 0xFF00
+
union ibv_gid {
uint8_t raw[16];
struct {
@@ -92,7 +101,8 @@ enum ibv_device_cap_flags {
IBV_DEVICE_SYS_IMAGE_GUID = 1 11,
IBV_DEVICE_RC_RNR_NAK_GEN = 1 12,
IBV_DEVICE_SRQ_RESIZE = 1 13,
- IBV_DEVICE_N_NOTIFY_CQ = 1 14
+ IBV_DEVICE_N_NOTIFY_CQ = 1 14,
+ IBV_DEVICE_EXTENSIONS = 1 (IBV_EXTENSION_BASE_SHIFT - 1)
};
enum ibv_atomic_cap {
@@ -623,6 +633,13 @@ struct ibv_device {
chardev_path[IBV_SYSFS_PATH_MAX];
/* Path to infiniband class device in sysfs */
charibdev_path[IBV_SYSFS_PATH_MAX];
+
+ /* Following fields only available if device supports extensions */
+ void *private;
+ int (*have_ext_ops)(struct ibv_device
