Re: [openib-general] [RFC] userspace IB SA support
>>This was my original approach a couple of months back, but wasn't accepted as >>mer gable upstream because it increased the size of the user to kernel >>interface. > > > Can you point me at this ? I must have missed it. To clarify, I didn't request that the code be merged upstream. I only queried about the approach, and merging that into svn. My query was likely done off list. Part of the feedback was to ensure that the design was discussed on the list to get more input, which is what this is doing. See the following threads / related messages for more details: http://openib.org/pipermail/openib-general/2006-August/025271.html http://openib.org/pipermail/openib-general/2006-August/025434.html - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [RFC] userspace IB SA support
On Fri, 2007-01-12 at 12:29, Sean Hefty wrote: > >>Would we be okay with extending the IOCTL interface to allow multicast > >>joins, > >>notice registration, and event reporting? Or would it be acceptable to > >>change > >>the ib_umad read/write interface to add a command? > > > > > > What do you have in mind here ? > > I was thinking of one of two possibilities here. Currently there are IOCTL > calls to register/unregister with the MAD layer. Additional IOCTL calls > could > be added to join/leave multicast groups and register/unregister for SA > events. > Multicast and SA events would need to be reported through another IOCTL of > some > sort. > > The alternative basically rewrites the ib_umad interface to allow read and > write > to carry some sort of command, rather than mapping them directly to sending > and > receiving a MAD. This is how most of the RDMA kernel to user interfaces are > written. For example, let read return an event type (MAD received, multicast > event, etc.), along with the event data (the MAD, etc.). Do we really want to go down this approach ? > >>>As an alternative, a new kernel userspace SA module could be created to > >>>explicitly interface with the kernel ib_sa. > > > > IMO, this is the best way to go. > > This was my original approach a couple of months back, but wasn't accepted as > mer gable upstream because it increased the size of the user to kernel > interface. Can you point me at this ? I must have missed it. -- Hal > If we can agree that this approach is usable, we can discuss more > specific implementation details. > > - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [RFC] userspace IB SA support
>>Would we be okay with extending the IOCTL interface to allow multicast joins, >>notice registration, and event reporting? Or would it be acceptable to >>change >>the ib_umad read/write interface to add a command? > > > What do you have in mind here ? I was thinking of one of two possibilities here. Currently there are IOCTL calls to register/unregister with the MAD layer. Additional IOCTL calls could be added to join/leave multicast groups and register/unregister for SA events. Multicast and SA events would need to be reported through another IOCTL of some sort. The alternative basically rewrites the ib_umad interface to allow read and write to carry some sort of command, rather than mapping them directly to sending and receiving a MAD. This is how most of the RDMA kernel to user interfaces are written. For example, let read return an event type (MAD received, multicast event, etc.), along with the event data (the MAD, etc.). >>>As an alternative, a new kernel userspace SA module could be created to >>>explicitly interface with the kernel ib_sa. > > IMO, this is the best way to go. This was my original approach a couple of months back, but wasn't accepted as mer gable upstream because it increased the size of the user to kernel interface. If we can agree that this approach is usable, we can discuss more specific implementation details. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [RFC] userspace IB SA support
On Thu, 2007-01-11 at 13:38, Sean Hefty wrote: > Sean Hefty wrote: > > Adding this functionality to the existing ib_umad module would add an extra > > dependency of ib_umad on the ib_sa module. Multicast join / leave > > operations > > could be done by adding additional IOCTLs, by embedding the request as a > > send_mad call, or by modifying the ib_umad send interface. > > Given that the ibibumad interface is intended to send and receive MADs, I > would > rather not abuse the interface by changing the behavior of > umad_send/umad_recv. > These calls map directly to ib_umad write and read. I tend to agree with this. > Would we be okay with extending the IOCTL interface to allow multicast joins, > notice registration, and event reporting? Or would it be acceptable to > change > the ib_umad read/write interface to add a command? What do you have in mind here ? > > As an alternative, a new kernel userspace SA module could be created to > > explicitly interface with the kernel ib_sa. IMO, this is the best way to go. -- Hal > Or do people preferred this approach over changing the ib_umad interface? > > I'm looking for something that will be acceptable to merge upstream. > > - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [RFC] userspace IB SA support
On Tue, 2007-01-09 at 14:49, Sean Hefty wrote: > Tang, Changqing wrote: > > Where do you put these new user functions ? > > That is part of what I'd like input on. My thought was to add them to the > libibumad library, but how exactly is not clear yet. Wouldn't it be more straightforward to have it be a separate (libibsa) library ? libibumad is primarily send and receive user MADs (so it is much lower level). -- Hal > > Also when is it available ? > > I would like to have something by early February or sooner. > > - Sean > > ___ > openib-general mailing list > openib-general@openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [RFC] userspace IB SA support
Sean Hefty wrote: > Adding this functionality to the existing ib_umad module would add an extra > dependency of ib_umad on the ib_sa module. Multicast join / leave operations > could be done by adding additional IOCTLs, by embedding the request as a > send_mad call, or by modifying the ib_umad send interface. Given that the ibibumad interface is intended to send and receive MADs, I would rather not abuse the interface by changing the behavior of umad_send/umad_recv. These calls map directly to ib_umad write and read. Would we be okay with extending the IOCTL interface to allow multicast joins, notice registration, and event reporting? Or would it be acceptable to change the ib_umad read/write interface to add a command? > As an alternative, a new kernel userspace SA module could be created to > explicitly interface with the kernel ib_sa. Or do people preferred this approach over changing the ib_umad interface? I'm looking for something that will be acceptable to merge upstream. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [RFC] userspace IB SA support
>1. Join a multicast group - needs to use the ib_sa multicast capability. Adding this functionality to the existing ib_umad module would add an extra dependency of ib_umad on the ib_sa module. Multicast join / leave operations could be done by adding additional IOCTLs, by embedding the request as a send_mad call, or by modifying the ib_umad send interface. As an alternative, a new kernel userspace SA module could be created to explicitly interface with the kernel ib_sa. >2. Receive notification of multicast errors. Add to this, receive notification of the join as well. This could be accomplished by returning the relevant data (SA attribute only) through libibumad recv_mad. This would change the behavior of the user/kernel interface, but I'm not sure how many applications would be affected by the change. >3. Leave a multicast group. >4. Register to receive SA events - needs to use the ib_sa notice capability. >5. Receive notification of events. >6. Deregister from SA events. SA event notification could be handled similarly to multicast... - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [RFC] userspace IB SA support
Sean Hefty wrote: > Today, userspace support for SA related operations is limited to the libibmad > interface, which supports sending and receiving MADs only. I've been assigned > with the task of exposing multicast and informinfo support to userspace. > Specifically, the following functionality is needed: > > 1. Join a multicast group - needs to use the ib_sa multicast capability. > 2. Receive notification of multicast errors. > 3. Leave a multicast group. > 4. Register to receive SA events - needs to use the ib_sa notice capability. > 5. Receive notification of events. > 6. Deregister from SA events. > > Are there any preferences for how this is added? I'm a user; I can't comment on how this is implemented, but I'm very interested in what the API will look like. I think I discussed this on-list before, but a big feature for me (Open MPI) is to be able to request and join an unused multicast group. The exact address (or format, i.e. IP or MGID/MLID) of the group is not important as long as I can query it and pass it on out-of-band to peers. This avoids hard-coding of multicast addresses and/or outside (MPI user) input, which could result in multiple MPI jobs unknowingly using the same multicast group. The other big concern is control over which hca/port is joined to a multicast group -- IIRC this was a problem I had with the RDMA CM. Open MPI specifically tends to open every available network interface (multiple IB ports, as well as say GM/IB/TCP together) for bandwidth aggregation and failover purposes, so controlling which interfaces data goes out over is important. I hope this is useful information -- let me know if there is more input I can provide, or testing I can do. Andrew ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [RFC] userspace IB SA support
Tang, Changqing wrote: > Where do you put these new user functions ? That is part of what I'd like input on. My thought was to add them to the libibumad library, but how exactly is not clear yet. > Also when is it available ? I would like to have something by early February or sooner. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [RFC] userspace IB SA support
Where do you put these new user functions ? Do you create a new library ? I hope not to create a new library, we already have so many libraries now, it is hard to manage For users using dlopen(). Also when is it available ? Thanks. --CQ > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Sean Hefty > Sent: Tuesday, January 09, 2007 11:55 AM > To: Dotan Barak > Cc: Roland Dreier; openib > Subject: Re: [openib-general] [RFC] userspace IB SA support > > > What about path query or any SA query from the user level ? > > Yes, I'd like that too, but libibumad can provide this > functionality - even if the interface isn't ideal. > Duplicating the ib_sa kernel module functionality in > userspace could allow for a simpler interface however. > > Trying to use the existing interfaces for multicast or notice > registration, however, is more problematic, which is why I > was trying to focus on those areas specifically. > > - Sean > > ___ > openib-general mailing list > openib-general@openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [RFC] userspace IB SA support
> What about path query or any SA query from the user level ? Yes, I'd like that too, but libibumad can provide this functionality - even if the interface isn't ideal. Duplicating the ib_sa kernel module functionality in userspace could allow for a simpler interface however. Trying to use the existing interfaces for multicast or notice registration, however, is more problematic, which is why I was trying to focus on those areas specifically. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [RFC] userspace IB SA support
Sean Hefty wrote: > Today, userspace support for SA related operations is limited to the libibmad > interface, which supports sending and receiving MADs only. I've been assigned > with the task of exposing multicast and informinfo support to userspace. > Specifically, the following functionality is needed: > > 1. Join a multicast group - needs to use the ib_sa multicast capability. > 2. Receive notification of multicast errors. > 3. Leave a multicast group. > 4. Register to receive SA events - needs to use the ib_sa notice capability. > 5. Receive notification of events. > 6. Deregister from SA events. > > Are there any preferences for how this is added? > What about path query or any SA query from the user level ? Thanks Dotan ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] [RFC] userspace IB SA support
Today, userspace support for SA related operations is limited to the libibmad interface, which supports sending and receiving MADs only. I've been assigned with the task of exposing multicast and informinfo support to userspace. Specifically, the following functionality is needed: 1. Join a multicast group - needs to use the ib_sa multicast capability. 2. Receive notification of multicast errors. 3. Leave a multicast group. 4. Register to receive SA events - needs to use the ib_sa notice capability. 5. Receive notification of events. 6. Deregister from SA events. Are there any preferences for how this is added? - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
