RE: [openib-general] [RFC] OpenSM Interactive Console
Title: RE: [openib-general] [RFC] OpenSM Interactive Console
Yes this MIB needs some cleanup.
I would love to hear from the community some feedback regarding SM MIB usefulness.
In the past we did not get any push for interactive SM or online configurable SM so I did not see any reason to work on it.
I do not think it is a huge task to make SM MIB work with OpenSM. At least not the 90% of it that I glanced through.
Eitan Zahavi
Design Technology Director
Mellanox Technologies LTD
Tel:+972-4-9097208
Fax:+972-4-9593245
P.O. Box 586 Yokneam 20692 ISRAEL
> -Original Message-
> From: Hal Rosenstock [mailto:[EMAIL PROTECTED]]
> Sent: Wednesday, October 26, 2005 7:44 PM
> To: Eitan Zahavi
> Cc: Troy Benjegerdes; openib-general@openib.org
> Subject: RE: [openib-general] [RFC] OpenSM Interactive Console
>
> Hi Eitan,
>
> I sit corrected. There are R/W parameters in the SM MIB as you indicate. I was
> thinking of all the other IPoIB MIBs. It's been a while since I looked at the SM MIB.
>
> Also, the SM MIB (draft-ietf-ipoib-subnet-manager-mib-00) expired a while ago. At a
> minimum, it needs to be dusted off. That would include updating it for IBA 1.2.
>
> -- Hal
>
>
>
> From: Eitan Zahavi [mailto:[EMAIL PROTECTED]]
> Sent: Tue 10/25/2005 5:19 AM
> To: Hal Rosenstock
> Cc: Troy Benjegerdes; openib-general@openib.org
> Subject: Re: [openib-general] [RFC] OpenSM Interactive Console
>
>
>
> Hal Rosenstock wrote:
> > On Mon, 2005-10-24 at 14:38, Eitan Zahavi wrote:
> >
> >>Hal Rosenstock wrote:
> >>
> >>>On Mon, 2005-10-24 at 03:08, Eitan Zahavi wrote:
> >>>
> >>>
> I would suggest to use SNMP for the tasks below. IETF IPoIB group
> >
> > has
> >
> defined an SNMP MIB that can support the required functionality
> >
> > below.
> >
> >>>
> >>>The IETF SNMP MIBs are one way of presenting the information to the
> >>>outside world. There are other possible management interfaces. The
> >
> > SNMP
> >
> >>>MIB instrumentation would need to use lower layer APIs to get this
> >>>information out of the SM.
> >>
> >>Yes but the IETF SM MIB is the only one that is close to a standard
> >
> > way.
> >
> >>It does not require low level interface if it will integrate into the
> >
> > OpenSM code.
> >
> >>One way to do it is buy extending OpenSM with an AgentX interface.
> >>
> >>IMO one clear advantage of using SNMP for SM integration is that the
> >
> > code will work with any SM that is IETF compliant.
> >
> >>Also if you want to write a "client server" type of application on top
> >
> > of an SM you
> >
> >>can either stick to sending MADs which translate into SA client based
> >
> > application or
> >
> >>you better stay with some known protocol for management (like SNMP)
> >
> > and not develop yet another protocol for
> >
> >>doing exactly the same things as SNMP already supports.
> >
> >
> > There are limitations in the SNMP MIBs. One is that they are RO so they
> > are more for monitoring. Also, many environments do not use SNMP. It is
> > unclear how much of a requirement it is to manage any SM or how many
> > other SMs support the SM MIB. (There are other IB associated MIBs too).
>
> SNMP MIBs are certainly not just RO a simple example from the SM MIB:
> ibSmPortInfoLMC OBJECT-TYPE
> SYNTAX Unsigned32(0..7)
> MAX-ACCESS read-write
> STATUS current
> DESCRIPTION
> "LID mask for multipath support. User should take extra caution
> when setting this value, since any change will effect packet
> routing."
> ::= { ibSmPortInfoEntry 19 }
>
>
> I agree that it is possible that currently no SM is supporting the SM MIB.
> But it does make sense to have ALL of the them support it. Such that they can
> be activated/deactivated and configured in the manner.
>
> Most unix distributions and windows box have standard SNMP agent and client
> included in them
> So it does not take more then simple bash or C code to interact with the SM if it
> supports SNMP.
>
> >
> >
> Everything but the dynamic partitioning (OpenSM does not have
> partition manager to this moment)
> >>>
> >>>
> >>>What Troy meant by partitioning is not necessarily IB partitioning.
> >>
> >>How are you sure about that? Troy - please comment.
> >
> >
> > I think you missed an email on this.
> >
> >
> and forwarding of Performance
> Monitoring traps (which are generated by the PM) can be done through
> osmsh or through SA client today.
> >>>
> >>>
> >>>What PerfMgr are you referring to ?
> >>
> >>No specific one. But the specification does not require the SM too.
> >
> >
> > Huh ? What spec ? An SM is required in a subnet. There is no subnet
> > without this. There is a subnet without a PerfMgr.
> Yes its a typo I meant PM. SM is a requirement. You know I did not mean that.
> >
> >
> >>For various reasons (like load) it might make more sense to have the
> >
> > PM distri
[openib-general] RE: Osmtest removal from Gen2 main trunk
Title: RE: Osmtest removal from Gen2 main trunk Hi , Hal . No problem , it can wait till next week. -Original Message- From: Hal Rosenstock [mailto:[EMAIL PROTECTED]] Sent: Wednesday, October 26, 2005 6:36 PM To: Liran Sorani Cc: openib-general@openib.org Subject: RE: Osmtest removal from Gen2 main trunk Hi Liran, I'm out at SC05 staging. Can this wait until I get back (no later than early next week) ? I want to do a side by side comparison before osmtest is removed from the trunk. -- Hal From: Liran Sorani [mailto:[EMAIL PROTECTED]] Sent: Tue 10/25/2005 1:35 AM To: Hal Rosenstock Cc: openib-general@openib.org Subject: Osmtest removal from Gen2 main trunk Hi , Hal . Since now the Osmtest is updated (in all stack flavours) under ibtp repository (https://openib.org/svn/trunk/contrib/mellanox/ibtp/), I'd like to remove it from main trunk : https://openib.org/svn/gen2/trunk/src/userspace/management/osm/osmtest. New updates will be checked into ibtp repository only , thanks . -Original Message- From: Liran Sorani Sent: Sunday, October 23, 2005 9:01 AM To: 'Hal Rosenstock'; Liran Sorani Cc: openib-general@openib.org Subject: RE: [openib-general] InfiniBand Test Project (IBTP) - Update Currently only a minor bug fix in osmt_service flow , and cosmetics changes to fit WinIb stack . -Original Message- From: Hal Rosenstock [mailto:[EMAIL PROTECTED]] Sent: Thursday, October 20, 2005 1:01 PM To: Liran Sorani Cc: openib-general@openib.org Subject: RE: [openib-general] InfiniBand Test Project (IBTP) - Update On Thu, 2005-10-20 at 03:49, Liran Sorani wrote: > Hi , Hal . > The Linux & WinIB are the same , except for several cosmetic changes . I was referring to the (differences in the) Linux one in ibtp and the Linux one under gen2/trunk. > Regarding Makefile.in , it's an outcome of autogen , I'll remove it . Thanks. -- Hal > -Original Message- > From: Hal Rosenstock [mailto:[EMAIL PROTECTED]] > Sent: Wednesday, October 19, 2005 10:25 PM > To: Liran Sorani > Cc: openib-general@openib.org > Subject: Re: [openib-general] InfiniBand Test Project (IBTP) - Update > > > On Wed, 2005-10-19 at 15:33, Liran Sorani wrote: > > Hi , > > We've updated IBTP tree with Osmtest sources both on ibal (WinIB) > and > > Gen2 stacks : > > > https://openib.org/svn/trunk/contrib/mellanox/ibtp/ibal/ulp/opensm/user/osmtest > > > > > https://openib.org/svn/trunk/contrib/mellanox/ibtp/gen2/userspace/management/osm/osmtest > > > > Osmtest is the main verification tool for OpenSM , include various > SA > > (Good / Bad) flows. > > Attached to each directory a short README file for setup and usage > > information. > > How is the Linux one different from osmtest in the trunk ? > > Also, (nit): > I think > https://openib.org/svn/trunk/contrib/mellanox/ibtp/gen2/userspace/management/osm/osmtest/Makefile.in > is a generated file and should be removed. > > -- Hal > > > > Liran Sorani > > > Mellanox Technologies LTD. > > > mailto:[EMAIL PROTECTED] > > > Phone: +972(4)9097200 Ext: 214 > > > Israel, Yokneam P.O.B 586 ZIP 20692 > > > > > > > > > > > > > > > __ > > > > ___ > > openib-general mailing list > > openib-general@openib.org > > http://openib.org/mailman/listinfo/openib-general > > > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] last version for 2.6.9 backport
> BTW. I was not able to test the pathscale driver as I do not > have any of their H/W, so if someone that has H/W could > test it, that would be great. I think we might have some hardware lying around. :-) Will try it out. Johann ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] Automated userspace build error
Nish wrote, >On a related note, do you (or anyone else) have any suggestions for >build-testing all of the userspace components? There isn't a top-level >Makefile of any kind to make it easy :/ >Thanks, >Nish If you look at the openib download page, Makia posted a userspace source RPM, although it is a bit out of date. I also have a similar build proceedure that I use internally, basically building all of the usermode components and then building an RPM to allow easy installation on other nodes for testing There are also .spec files for most of the individual libraries, if you prefer to build RPMs for individual libraries. I find it easier just to lump it all into one big usermode component RPM and one kernel-mode component RPM. woody ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Automated userspace build error
On 25.10.2005 [15:22:56 -0700], Roland Dreier wrote: > Nishanth> Hrm, well, I'm testing the latest svn (3865), did the > Nishanth> patch just get checked in? > > Yeah, I only noticed it and fixed it after your original email. I > just meant that I had already checked it in before sending my reply. > Sorry for the confusion... No worries, I figured that's what happened. On a related note, do you (or anyone else) have any suggestions for build-testing all of the userspace components? There isn't a top-level Makefile of any kind to make it easy :/ Thanks, Nish ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] Re: RFC userspace CMA
> -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of > Michael S. Tsirkin > Sent: Wednesday, October 26, 2005 1:44 PM > To: Sean Hefty > Cc: openib > Subject: [openib-general] Re: RFC userspace CMA > > > But I mean, we can already send ARP packets from userspace, cant we? > No, non-privileged users are not allowed to modify the ARP table, open /dev/arp or to send raw Ethernet. You can use ARP to query from non-privileged userspace. But nothing beyond that. If you check the man page you'll also note that the ARP daemon specifically listens to ensure that nobody else is impersonating it. That's exactly the type of safety check that is blocked if IP addresses are passed via private data where the fact that the data is an IP address is not defined in the wire protocol. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: RFC userspace CMA
Michael S. Tsirkin wrote: But I mean, we can already send ARP packets from userspace, cant we? That I don't know. Are there APIs to send requests and view responses in userspace? - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: device add/remove in userspace
Quoting r. Sean Hefty <[EMAIL PROTECTED]>: > Subject: Re: device add/remove in userspace > > Roland Dreier wrote: > > Sean> Is there a way for a userspace application to know if a > > Sean> device has been added or removed? > > > > We don't really handle this right now. It could probably be made to > > work on top of hotplug/udev/hal/something but it seems tricky to me. > > Any idea on how we currently want to handle this? Can a device remove occur > if > a usermode app is using the device? Well, an easy way out would be to pass an event up to the application, and block device removal until the application closes all resources. This doesnt seem to be hard to do, given exiting mechanisms of passing asynchronous events up to userspace. Ideally, of course, we would free the device without waiting for userspace, but that may be trickier to accomplish: probably need to modify all QPs to error state, then close them and close all CQs. Further, need to unmap the device memory mapped into application's memory. > The kernel CMA will block until all connection identifiers associated with > the > removed device are freed. I export the device removal up to userspace > through > the uCMA, but that leaves the kernel blocked until the userspace application > cleans up. Doing something more intelligent requires a lot more effort. > > - Sean I think this is reasonable, short-term. -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: [PATCH #2] new uDAPL openIB provider using socket CM, corrected license headers
> James, > > This version includes updated license headers per your request. > > -arlin Committed in revision 3882. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: RFC userspace CMA
Quoting r. Sean Hefty <[EMAIL PROTECTED]>: > Subject: Re: RFC userspace CMA > > Michael S. Tsirkin wrote: > > Quoting Sean Hefty <[EMAIL PROTECTED]>: > > > >>I considered, and continue to consider implementing on top of ucm. The > >>drawbacks are: it requires more kernel modules: one for the CM, one for SA > >>query, and one for address translation. > > > > > > Cant address translation be done with exiting kernel/user interface? > > There's no kernel/user interface for ib_addr, which is what the kernel CMA > uses. > To use the ib_at kernel/user interface, ib_at would need to be fixed to > avoid > crashing the system. ib_addr is based off of the ib_at/sdp implementations, > but > limited to ARP translation only. > > It would also require userspace components for other RDMA CMs, such as iWarp. > > - Sean > But I mean, we can already send ARP packets from userspace, cant we? -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: device add/remove in userspace
Roland Dreier wrote: Sean> Is there a way for a userspace application to know if a Sean> device has been added or removed? We don't really handle this right now. It could probably be made to work on top of hotplug/udev/hal/something but it seems tricky to me. Any idea on how we currently want to handle this? Can a device remove occur if a usermode app is using the device? The kernel CMA will block until all connection identifiers associated with the removed device are freed. I export the device removal up to userspace through the uCMA, but that leaves the kernel blocked until the userspace application cleans up. Doing something more intelligent requires a lot more effort. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: [PATCH] new uDAPL openIB provider using socket CM
On Tue, 25 Oct 2005, Arlin Davis wrote: > James, > > Here is a patch to add an optional openIB uDAPL provider that uses the socket > CM for anyone having > problems scaling out with the uCM/uAT version. To build the new provider, > simply "make > VERBS=openib_scm". This version does not require IPoIB, uCM, or uAT. > > -arlin > > Signed-off by: Arlin Davis <[EMAIL PROTECTED]> Some of the new files in dapl/openib_scm use this license: dapl/openib_scm/dapl_ib_util.c > +/* > + * This Software is licensed under both of the following two licenses: > + * > + * 1) under the terms of the "Common Public License 1.0" a copy of which is > + *in the file LICENSE.txt in the root directory. The license is also > + *available from the Open Source Initiative, see > + *http://www.opensource.org/licenses/cpl.php. > + * OR > + * > + * 2) under the terms of the "The BSD License" a copy of which is in the file > + *LICENSE2.txt in the root directory. The license is also available from > + *the Open Source Initiative, see > + *http://www.opensource.org/licenses/bsd-license.php. > + * > + * Licensee has the right to choose either one of the above two licenses. > + * > + * Redistributions of source code must retain both the above copyright > + * notice and either one of the license notices. > + * > + * Redistributions in binary form must reproduce both the above copyright > + * notice, either one of the license notices in the documentation > + * and/or other materials provided with the distribution. > + */ and other files use this license dapl/openib_scm/dapl_ib_mem.c > +/* > + * This Software is licensed under one of the following licenses: > + * > + * 1) under the terms of the "Common Public License 1.0" a copy of which is > + *available from the Open Source Initiative, see > + *http://www.opensource.org/licenses/cpl.php. > + * > + * 2) under the terms of the "The BSD License" a copy of which is > + *available from the Open Source Initiative, see > + *http://www.opensource.org/licenses/bsd-license.php. > + * > + * 3) under the terms of the "GNU General Public License (GPL) Version 2" a > + *copy of which is available from the Open Source Initiative, see > + *http://www.opensource.org/licenses/gpl-license.php. > + * > + * Licensee has the right to choose one of the above licenses. > + * > + * Redistributions of source code must retain the above copyright > + * notice and one of the license notices. > + * > + * Redistributions in binary form must reproduce both the above copyright > + * notice, one of the license notices in the documentation > + * and/or other materials provided with the distribution. > + */ I'd like all of the files to use the later. Is that acceptable to you? If so, please send a new patch with this change. james ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] RFC userspace CMA
Ah, I think I understand the confusion. 0 in this case, doesn't mean
"wildcard", it means "assign a port for me". I think you already handle
this in IB... from ib_cm_listen ...
spin_lock_irqsave(&cm.lock, flags);
if (service_id == IB_CM_ASSIGN_SERVICE_ID) {
--->cm_id->service_id = cpu_to_be64(cm.listen_service_id++);
--->cm_id->service_mask = __constant_cpu_to_be64(~0ULL);
} else {
cm_id->service_id = service_id;
cm_id->service_mask = service_mask;
}
cur_cm_id_priv = cm_insert_listen(cm_id_priv);
On Wed, 2005-10-26 at 11:02 -0700, Sean Hefty wrote:
> Tom Tucker wrote:
> >>Something that didn't make sense for the kernel rdma_cm running over IB was
> >>adding a backlog parameter to the listen request. (The IB CM is callback
> >>driven, so there's not really a backlog.) I will probably add this to the
> >>userspace API. Does iWarp need a backlog parameter in the kernel?
> >
> > It is needed by some adapters. For the AMSO1100 it's passed down to the
> > adapter to reserve syn cache entries for incoming connections.
>
> There's no problem adding this. It just that for IB, the user can control
> the
> backlog themselves, which gives a little more flexibility. I suppose we can
> define it as the maximum number of unacknowledged connection requests that a
> user can have. Where acknowledging a connection request is done by accepting
> or
> rejecting the connection. I'll work on adding this after getting the
> userspace
> CMA up.
>
> > Yes it's funky. Basically, the listen in this context is the combination
> > of a 'bind' and a 'listen'. If you specify 0 for a port number on bind,
> > the stack will allocate one for you.
>
> I need to think about this more. I'm not sure how to handle a bind of 0 over
> IB. Can you handle this on the bind only, as opposed to listen?
>
> - Sean
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] RFC userspace CMA
Tom Tucker wrote: Something that didn't make sense for the kernel rdma_cm running over IB was adding a backlog parameter to the listen request. (The IB CM is callback driven, so there's not really a backlog.) I will probably add this to the userspace API. Does iWarp need a backlog parameter in the kernel? It is needed by some adapters. For the AMSO1100 it's passed down to the adapter to reserve syn cache entries for incoming connections. There's no problem adding this. It just that for IB, the user can control the backlog themselves, which gives a little more flexibility. I suppose we can define it as the maximum number of unacknowledged connection requests that a user can have. Where acknowledging a connection request is done by accepting or rejecting the connection. I'll work on adding this after getting the userspace CMA up. Yes it's funky. Basically, the listen in this context is the combination of a 'bind' and a 'listen'. If you specify 0 for a port number on bind, the stack will allocate one for you. I need to think about this more. I'm not sure how to handle a bind of 0 over IB. Can you handle this on the bind only, as opposed to listen? - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] RFC userspace CMA
Caitlin Bestler wrote:
So it sounds like the justification for the RDMA CM being
a distinct module is to centralize handling of device addition
and removal. Beyond that you are incorporating IB-specific
but device-independent logic. As a goal, the iWARP side
should be migrating there as well.
As a general rule, the code is organized into a group of general functions that
are transport independent, and functions that are specific to a given transport.
For example, rdma_accept() is written as:
if (!cma_comp(id_priv, CMA_CONNECT))
return -EINVAL;
switch (id->device->node_type) {
case IB_NODE_CA:
ret = cma_accept_ib(id_priv, conn_param);
break;
default:
ret = -ENOSYS;
break;
}
if (ret)
goto reject;
return 0;
The IB specific code is in separate functions from the transport specific code,
but shares the same file. I did not want to try to define common lower-level
interfaces, such as a cma_accept_iwarp(), at this point.
- Sean
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] RFC userspace CMA
On Wed, 2005-10-26 at 08:41 -0700, Sean Hefty wrote: > Tom Tucker wrote: > > FYI, I've started writing the iw_cm that sits below the rdma_cm. Here's > > the general picture I have in mind. > > > > +-+ > > | RDMA CM | > > +-+-+-+ > > | | > > ++ ++ > > | | > > +-+ +++ > > | IB CM | | IW CM | > > +++ +++ > > | | > > +_ +_ > > +-+|+-+| > > | IB devs ||| IW devs || > > +-+ +-+ > > This is what I was envisioning as well. > > > I am also migrating the current iw_cm.h file to match the interfaces in > > the rdma_cm more closely. > > Note that there are still some changes occurring to the rdma_cm to support > userspace. I'm concerned about how well these changes map to iWarp, since > the > changes expose the three-way CM handshake used by IB. > > > In general, the IW CM methods look very much like sockets connect, > > listen, and accept. There is an iw_cm_id like the ib_cm_id that > > encapsulates the 5-tuple, a callback for IW CM events and a "provider > > handle" that represents the adapter "connection cookie". The iw_cm_id is > > passed to connect, accept, etc... > > Something that didn't make sense for the kernel rdma_cm running over IB was > adding a backlog parameter to the listen request. (The IB CM is callback > driven, so there's not really a backlog.) I will probably add this to the > userspace API. Does iWarp need a backlog parameter in the kernel? It is needed by some adapters. For the AMSO1100 it's passed down to the adapter to reserve syn cache entries for incoming connections. > > > depending on the model. This means that calls like listen with a local > > port wildcard can't return until the "listen_reply" comes back from the > > adapter. > > I didn't quite follow this. Right now, the rdma_cm only tries to support > wildcard IP addresses. Are you wanting to support listening on any port as > well? What is a listen_reply? > Yes it's funky. Basically, the listen in this context is the combination of a 'bind' and a 'listen'. If you specify 0 for a port number on bind, the stack will allocate one for you. MPI uses this to allocate a port and then advertises this port to a central server (node of rank 0) who tells the other servers how to contact each other. This avoids having to allocate a well known port for each node in the MPI cluster and allows multiple apps to run concurrently without allocating additional well-known ports. The listen_reply from the adapter returns the port chosen and the status of the listen request. It is the somewhat analagous to the insert_listen_ in the IB CM framework. > - Sean > > ___ > openib-general mailing list > openib-general@openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] RFC userspace CMA
> -Original Message- > From: Sean Hefty [mailto:[EMAIL PROTECTED] > Sent: Wednesday, October 26, 2005 10:40 AM > To: Caitlin Bestler > Cc: Tom Tucker; openib > Subject: Re: [openib-general] RFC userspace CMA > > Caitlin Bestler wrote: > > How much logic is really in the RDMA CM? > > > > If it is sufficiently small, which is what my expectation > is, would it > > make sense to simply make the IB CM and IW CM conform to the same > > polymorphic interface? (Making the "RDMA CM" little more than a > > re-directing inline function). > > > > But if there is any substantial portion of common logic > then the above > > structure definitely makes sense. > > The kernel CMA is about 660 lines of code. It performs QP > transitions for the user, abstracts device remove/addition, > plus controls the mapping from IP to IB. > (The mapping function makes use of external modules, such > as ib_addr and > sa_query.) > > In some places, the implementation of the API of the RDMA CM > does little more than redirecting to the IW/IB CM, coupled > with synchronization for device removal. However, it also > handles the IB CM callbacks to provide simpler connection > notification. > So it sounds like the justification for the RDMA CM being a distinct module is to centralize handling of device addition and removal. Beyond that you are incorporating IB-specific but device-independent logic. As a goal, the iWARP side should be migrating there as well. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] [RFC] OpenSM Interactive Console
Hi Eitan,
I sit corrected. There are R/W parameters in the SM MIB as you indicate. I was
thinking of all the other IPoIB MIBs. It's been a while since I looked at the
SM MIB.
Also, the SM MIB (draft-ietf-ipoib-subnet-manager-mib-00) expired a while ago.
At a minimum, it needs to be dusted off. That would include updating it for IBA
1.2.
-- Hal
From: Eitan Zahavi [mailto:[EMAIL PROTECTED]
Sent: Tue 10/25/2005 5:19 AM
To: Hal Rosenstock
Cc: Troy Benjegerdes; openib-general@openib.org
Subject: Re: [openib-general] [RFC] OpenSM Interactive Console
Hal Rosenstock wrote:
> On Mon, 2005-10-24 at 14:38, Eitan Zahavi wrote:
>
>>Hal Rosenstock wrote:
>>
>>>On Mon, 2005-10-24 at 03:08, Eitan Zahavi wrote:
>>>
>>>
I would suggest to use SNMP for the tasks below. IETF IPoIB group
>
> has
>
defined an SNMP MIB that can support the required functionality
>
> below.
>
>>>
>>>The IETF SNMP MIBs are one way of presenting the information to the
>>>outside world. There are other possible management interfaces. The
>
> SNMP
>
>>>MIB instrumentation would need to use lower layer APIs to get this
>>>information out of the SM.
>>
>>Yes but the IETF SM MIB is the only one that is close to a standard
>
> way.
>
>>It does not require low level interface if it will integrate into the
>
> OpenSM code.
>
>>One way to do it is buy extending OpenSM with an AgentX interface.
>>
>>IMO one clear advantage of using SNMP for SM integration is that the
>
> code will work with any SM that is IETF compliant.
>
>>Also if you want to write a "client server" type of application on top
>
> of an SM you
>
>>can either stick to sending MADs which translate into SA client based
>
> application or
>
>>you better stay with some known protocol for management (like SNMP)
>
> and not develop yet another protocol for
>
>>doing exactly the same things as SNMP already supports.
>
>
> There are limitations in the SNMP MIBs. One is that they are RO so they
> are more for monitoring. Also, many environments do not use SNMP. It is
> unclear how much of a requirement it is to manage any SM or how many
> other SMs support the SM MIB. (There are other IB associated MIBs too).
SNMP MIBs are certainly not just RO a simple example from the SM MIB:
ibSmPortInfoLMC OBJECT-TYPE
SYNTAX Unsigned32(0..7)
MAX-ACCESS read-write
STATUS current
DESCRIPTION
"LID mask for multipath support. User should take extra caution
when setting this value, since any change will effect packet
routing."
::= { ibSmPortInfoEntry 19 }
I agree that it is possible that currently no SM is supporting the SM MIB.
But it does make sense to have ALL of the them support it. Such that they can
be activated/deactivated and configured in the manner.
Most unix distributions and windows box have standard SNMP agent and client
included in them
So it does not take more then simple bash or C code to interact with the SM if
it supports SNMP.
>
>
Everything but the dynamic partitioning (OpenSM does not have
partition manager to this moment)
>>>
>>>
>>>What Troy meant by partitioning is not necessarily IB partitioning.
>>
>>How are you sure about that? Troy - please comment.
>
>
> I think you missed an email on this.
>
>
and forwarding of Performance
Monitoring traps (which are generated by the PM) can be done through
osmsh or through SA client today.
>>>
>>>
>>>What PerfMgr are you referring to ?
>>
>>No specific one. But the specification does not require the SM too.
>
>
> Huh ? What spec ? An SM is required in a subnet. There is no subnet
> without this. There is a subnet without a PerfMgr.
Yes its a typo I meant PM. SM is a requirement. You know I did not mean that.
>
>
>>For various reasons (like load) it might make more sense to have the
>
> PM distributed.
>
> Sure. Also, the PerfMgr need not be colocated with the SM anyhow.
>
>
>>Anyway, my point is that the SM is not the owner of PM trap reporting.
>
> It is the PM that
>
>>should support Reporting (I.e InformInfo registration and Trap
>
> forwarding) for PM traps.
>
>>But the spec does not define such traps anyway.
>
>
> My point was that the PerfMgr is beyond the IBA spec. It is only the PMA
> that is defined and has no traps so these will all need synthesis by the
> PerfMgr.
Agree.
>
> -- Hal
>
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] RFC userspace CMA
Caitlin Bestler wrote: How much logic is really in the RDMA CM? If it is sufficiently small, which is what my expectation is, would it make sense to simply make the IB CM and IW CM conform to the same polymorphic interface? (Making the "RDMA CM" little more than a re-directing inline function). But if there is any substantial portion of common logic then the above structure definitely makes sense. The kernel CMA is about 660 lines of code. It performs QP transitions for the user, abstracts device remove/addition, plus controls the mapping from IP to IB. (The mapping function makes use of external modules, such as ib_addr and sa_query.) In some places, the implementation of the API of the RDMA CM does little more than redirecting to the IW/IB CM, coupled with synchronization for device removal. However, it also handles the IB CM callbacks to provide simpler connection notification. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] RFC userspace CMA
> -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Tom Tucker > Sent: Wednesday, October 26, 2005 5:01 AM > To: Sean Hefty > Cc: openib > Subject: Re: [openib-general] RFC userspace CMA > > Sean: > > FYI, I've started writing the iw_cm that sits below the > rdma_cm. Here's the general picture I have in mind. > > +-+ > | RDMA CM | > +-+-+-+ > | | > ++ ++ > | | > +-+ +++ > | IB CM | | IW CM | > +++ +++ > | | > +_ +_ > +-+|+-+| > | IB devs ||| IW devs || > +-+ +-+ > > The purpose of the IW CM is to abstract the two different > connection models used by the iWARP side: offloaded and host > integrated, and to act as a shim between device specific > connection data structures and the rdma_cm data structures. > How much logic is really in the RDMA CM? If it is sufficiently small, which is what my expectation is, would it make sense to simply make the IB CM and IW CM conform to the same polymorphic interface? (Making the "RDMA CM" little more than a re-directing inline function). But if there is any substantial portion of common logic then the above structure definitely makes sense. My main concern is that the Connection Request reported up from here has the same semantics over IB CM as it does over IW CM. The remote IP Address of an established connection is pretty fairly validated. It originated from privileged code on the remote side (presumably the kernel) and the routers actually got packets back to that machine using that address. So a daemon using that address for client validation is totally reasonable, over an IP network. An equivalent degree of reliability should be there for IB. Using the SM to validate a translation achieved that. There are certainly other ways to achieve it. But we need to be clear that it is part of what the application is expecting when it is told that remote IP Address X is requesting a connection. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] round 2 - proposal for socketbased connectionmodel
This is the whole purpose of the protocol. It is OS independent and ensures interoperability. Nobody will change their OS protocol implementation so it can communicate to Linux (or any other OS or vendor) that invented its own protocol... It is not OS (linux no exception) job to invent protocols. But I think this argument have been bitten enough already. Arkady Arkady Kanevsky email: [EMAIL PROTECTED] Network Appliance phone: 781-768-5395 375 Totten Pond Rd. Fax: 781-895-1195 Waltham, MA 02451-2010 central phone: 781-768-5300 > -Original Message- > From: Woodruff, Robert J [mailto:[EMAIL PROTECTED] > Sent: Wednesday, October 26, 2005 11:33 AM > To: Kanevsky, Arkady; Sean Hefty > Cc: [EMAIL PROTECTED]; openib-general@openib.org > Subject: RE: [openib-general] round 2 - proposal for > socketbased connectionmodel > > > Arkady wrote, > >This is what we are trying to avoid. > >ULP should not change regardless whether or not it is running on IB, > >iWARP, VIA or any other RDMA transport. > > The whole point of the CMA is that the ULP can code to an > API that is independent of RDMA interconnect. The > CMA wire protocol can be documented to allow > non-Linux hosts to connect to a Linux box using > the same protocol. There is no need to change the existing > IB CM protocol to accomplish this. All that is needed is > to document that CMA protocol (contained in the private data > field of the IB CM requests). > > woody > > > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [swg] RE: [openib-general] RE: [dat-discussions] round 2 - proposal forsocket based connection model
Of course, you can encode versions into service Id. But that will mix concepts. And I do not believe that is worse it to provide a couple more bytes of Consumer private data. This encoding will not be enough to give Consumer 64 bytes of private data. The port numbers are mapped differently for different protocol numbers (families). If we only concern with TCP port mapping this will not be needed. But ULP right now make its decision by standard socket 5-tuple which does include it. I prefer that we do not require any changes in ULP to run over IB. We can do that in the API if there is no need to support more than just TCP. IN this case API can always return the protocol number for TCP to a Consumer. One concern I have is that some existing ULPs (say SDP) rely on the existing format of the private data. Thus, it would not want to use this CM encoding. I do not want to force it to change. Thus, a bit in CM which indicate whether encoding is present looks like a right approach. Arkady Arkady Kanevsky email: [EMAIL PROTECTED] Network Appliance phone: 781-768-5395 375 Totten Pond Rd. Fax: 781-895-1195 Waltham, MA 02451-2010 central phone: 781-768-5300 > -Original Message- > From: Yaron Haviv [mailto:[EMAIL PROTECTED] > Sent: Wednesday, October 26, 2005 12:21 PM > To: Kanevsky, Arkady; Sean Hefty > Cc: [EMAIL PROTECTED]; openib-general@openib.org; > [EMAIL PROTECTED] > Subject: [swg] RE: [openib-general] RE: [dat-discussions] > round 2 - proposal forsocket based connection model > > > > -Original Message- > > From: [EMAIL PROTECTED] [mailto:openib-general- > > [EMAIL PROTECTED] On Behalf Of Kanevsky, Arkady > > Sent: Tuesday, October 25, 2005 1:26 PM > > To: Sean Hefty > > Cc: [EMAIL PROTECTED]; openib-general@openib.org; dat- > > [EMAIL PROTECTED] > > Subject: RE: [openib-general] RE: [dat-discussions] round 2 > - proposal > > forsocket based connection model > > > > Think of a single API that supports iWARP and IB (transport > independent > > API). > > To a connection listener it provides the IP 5-tuple + private data. > > For IB it means that CM parses REQ and extracts IP 5-tuple > as separate > > fields from private data. Listener does not parse the private data > > encoding of the proposal. > > > > So CM need to know if it need to encode IP 5-tuple on > requestor side > > and if need to parse on responder side. Arkady > > > > Arkady, I agree with Sean you can encode the Dest Port in the > ServiceID > And if you really want to verify its using that format you can look at > the upper 48 bits in the serviceID. > > We may need to distinguish between Explicit RDMA protocols (iSER, > NFS-RDMA, RDP, etc') and Implicit RDMA (SDP, where the Socket > application doesn't know it is using RDMA), this can be done > in 3 ways: > a. port mapper, b. different ServiceID prefix, or c. a bit in > the CM REQ > Header. > > Also I'm not sure why we need the Protocol (UDP, TCP, SCTP, > ..) since we > emulate RDMA we shouldn't care if its TCP or SCTP, and UDP is > unconnected and cant drive RDMA anyway > > Yaron > > > > > > Arkady Kanevsky email: [EMAIL PROTECTED] > > Network Appliance phone: 781-768-5395 > > 375 Totten Pond Rd. Fax: 781-895-1195 > > Waltham, MA 02451-2010 central phone: 781-768-5300 > > > > > > > > > -Original Message- > > > From: Sean Hefty [mailto:[EMAIL PROTECTED] > > > Sent: Tuesday, October 25, 2005 1:08 PM > > > To: Kanevsky, Arkady > > > Cc: Caitlin Bestler; [EMAIL PROTECTED]; > > > openib-general@openib.org; [EMAIL PROTECTED] > > > Subject: Re: [openib-general] RE: [dat-discussions] round 2 - > > > proposal for socket based connection model > > > > > > > > > Kanevsky, Arkady wrote: > > > > Correct. > > > > But this does bring the question how responder CM knows > > > that it need > > > > to parse the private data. I suspect this will be done via > > > new version > > > > of CM. But a suage of some of the CM REQ reserved > fields are also > > > > possible. Anotherwords the current CM version assumes > that CM only > > > > supports one version and there is no need to support more than 1 > > > > version. > > > > > > The responder knows how to parse the private data based on > > > the service ID that > > > they're listening on. This is how it's done today, and how > > > it will still need > > > to be done. What is the motivation to change it? > > > > > > What data is beyond the addressing? How does the responder > > > know how to > > > interpret that? > > > > > > - Sean > > > > > ___ > > openib-general mailing list > > openib-general@openib.org > > http://openib.org/mailman/listinfo/openib-general > > > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib- > > general > ___ openib-general mailing l
[openib-general] FW: [openib-commits] r3881 -gen2/trunk/src/userspace/management/osm/include
Eitan, You should not be checking things into the trunk like this. The procedure is to submit them to the maintainer and the maintainer verifies them and checks them in. -- Hal From: [EMAIL PROTECTED] on behalf of [EMAIL PROTECTED] Sent: Wed 10/26/2005 12:50 PM To: [EMAIL PROTECTED] Subject: [openib-commits] r3881 -gen2/trunk/src/userspace/management/osm/include Author: eitan Date: 2005-10-26 09:50:25 -0700 (Wed, 26 Oct 2005) New Revision: 3881 Modified: gen2/trunk/src/userspace/management/osm/include/Makefile.am Log: Missing osm_console.h in DIST list Modified: gen2/trunk/src/userspace/management/osm/include/Makefile.am === --- gen2/trunk/src/userspace/management/osm/include/Makefile.am 2005-10-26 16:26:15 UTC (rev 3880) +++ gen2/trunk/src/userspace/management/osm/include/Makefile.am 2005-10-26 16:50:25 UTC (rev 3881) @@ -35,6 +35,7 @@ $(srcdir)/opensm/osm_sa_service_record.h \ $(srcdir)/opensm/osm_sa_response.h \ $(srcdir)/opensm/osm_node.h \ + $(srcdir)/opensm/osm_console.h \ $(srcdir)/opensm/osm_sa_slvl_record_ctrl.h \ $(srcdir)/opensm/osm_req.h \ $(srcdir)/opensm/osm_mcm_info.h \ ___ openib-commits mailing list [EMAIL PROTECTED] http://openib.org/mailman/listinfo/openib-commits ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] CMA service ID space versus private data bytes
Roland Dreier wrote: Sean> This results in the CMA reserving 2^40 IDs (about 6% of the Sean> total range). Just to be nitpicky -- 2^40 service IDs is 1 / 2^24 of the total number of service IDs (2^64), which is about .06%. Uhm.. good catch. That makes a difference in how important it is to conserve those IDs. Thanks. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] RE: console patch breaks OpenSM make dist
Hi Eitan, I needed it for SCinet 05 staging. It can easily be backed out. -- Hal From: Eitan Zahavi [mailto:[EMAIL PROTECTED] Sent: Wed 10/26/2005 12:42 PM To: Hal Rosenstock Cc: OPENIB GENERAL Subject: OSM: console patch breaks OpenSM make dist Hi Hal, The last commit you have done breaks the OpenSM make dist. Seems you did not try to make dist. I was also surprised that you have committed OpenSM console patch while it is still under discussion. The following patch resolves that issue: Index: osm/include/Makefile.am === --- osm/include/Makefile.am (revision 3880) +++ osm/include/Makefile.am (working copy) @@ -35,6 +35,7 @@ EXTRA_DIST = \ $(srcdir)/opensm/osm_sa_service_record.h \ $(srcdir)/opensm/osm_sa_response.h \ $(srcdir)/opensm/osm_node.h \ + $(srcdir)/opensm/osm_console.h \ $(srcdir)/opensm/osm_sa_slvl_record_ctrl.h \ $(srcdir)/opensm/osm_req.h \ $(srcdir)/opensm/osm_mcm_info.h \ ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] CMA service ID space versus private data bytes
Sean> This results in the CMA reserving 2^40 IDs (about 6% of the Sean> total range). Just to be nitpicky -- 2^40 service IDs is 1 / 2^24 of the total number of service IDs (2^64), which is about .06%. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] OSM: console patch breaks OpenSM make dist
Hi Hal, The last commit you have done breaks the OpenSM make dist. Seems you did not try to make dist. I was also surprised that you have committed OpenSM console patch while it is still under discussion. The following patch resolves that issue: Index: osm/include/Makefile.am === --- osm/include/Makefile.am (revision 3880) +++ osm/include/Makefile.am (working copy) @@ -35,6 +35,7 @@ EXTRA_DIST = \ $(srcdir)/opensm/osm_sa_service_record.h \ $(srcdir)/opensm/osm_sa_response.h \ $(srcdir)/opensm/osm_node.h \ + $(srcdir)/opensm/osm_console.h \ $(srcdir)/opensm/osm_sa_slvl_record_ctrl.h \ $(srcdir)/opensm/osm_req.h \ $(srcdir)/opensm/osm_mcm_info.h \ ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] RE: Osmtest removal from Gen2 main trunk
Hi Liran, I'm out at SC05 staging. Can this wait until I get back (no later than early next week) ? I want to do a side by side comparison before osmtest is removed from the trunk. -- Hal From: Liran Sorani [mailto:[EMAIL PROTECTED] Sent: Tue 10/25/2005 1:35 AM To: Hal Rosenstock Cc: openib-general@openib.org Subject: Osmtest removal from Gen2 main trunk Hi , Hal . Since now the Osmtest is updated (in all stack flavours) under ibtp repository (https://openib.org/svn/trunk/contrib/mellanox/ibtp/), I'd like to remove it from main trunk : https://openib.org/svn/gen2/trunk/src/userspace/management/osm/osmtest. New updates will be checked into ibtp repository only , thanks . -Original Message- From: Liran Sorani Sent: Sunday, October 23, 2005 9:01 AM To: 'Hal Rosenstock'; Liran Sorani Cc: openib-general@openib.org Subject: RE: [openib-general] InfiniBand Test Project (IBTP) - Update Currently only a minor bug fix in osmt_service flow , and cosmetics changes to fit WinIb stack . -Original Message- From: Hal Rosenstock [mailto:[EMAIL PROTECTED] Sent: Thursday, October 20, 2005 1:01 PM To: Liran Sorani Cc: openib-general@openib.org Subject: RE: [openib-general] InfiniBand Test Project (IBTP) - Update On Thu, 2005-10-20 at 03:49, Liran Sorani wrote: > Hi , Hal . > The Linux & WinIB are the same , except for several cosmetic changes . I was referring to the (differences in the) Linux one in ibtp and the Linux one under gen2/trunk. > Regarding Makefile.in , it's an outcome of autogen , I'll remove it . Thanks. -- Hal > -Original Message- > From: Hal Rosenstock [mailto:[EMAIL PROTECTED] > Sent: Wednesday, October 19, 2005 10:25 PM > To: Liran Sorani > Cc: openib-general@openib.org > Subject: Re: [openib-general] InfiniBand Test Project (IBTP) - Update > > > On Wed, 2005-10-19 at 15:33, Liran Sorani wrote: > > Hi , > > We've updated IBTP tree with Osmtest sources both on ibal (WinIB) > and > > Gen2 stacks : > > > https://openib.org/svn/trunk/contrib/mellanox/ibtp/ibal/ulp/opensm/user/osmtest > > > > > > https://openib.org/svn/trunk/contrib/mellanox/ibtp/gen2/userspace/management/osm/osmtest > > > > > Osmtest is the main verification tool for OpenSM , include various > SA > > (Good / Bad) flows. > > Attached to each directory a short README file for setup and usage > > information. > > How is the Linux one different from osmtest in the trunk ? > > Also, (nit): > I think > https://openib.org/svn/trunk/contrib/mellanox/ibtp/gen2/userspace/management/osm/osmtest/Makefile.in > > is a generated file and should be removed. > > -- Hal > > > > Liran Sorani > > > Mellanox Technologies LTD. > > > mailto:[EMAIL PROTECTED] > > > Phone: +972(4)9097200 Ext: 214 > > > Israel, Yokneam P.O.B 586 ZIP 20692 > > > > > > > > > > > > > > > __ > > > > ___ > > openib-general mailing list > > openib-general@openib.org > > http://openib.org/mailman/listinfo/openib-general > > > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] CMA service ID space versus private data bytes
There's a trade-off between service ID space used by the CMA and the amount of private data available to the user. Currently, the CMA reserves 64k of service ID space and provides 56 bytes of user private data. We can give the user 60 bytes of private data space by shifting 3 bytes (plus 1 reserved byte) from the private data into the service ID. This results in the CMA reserving 2^40 IDs (about 6% of the total range). How important is the private data to people versus the conservation of service IDs? - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] last version for 2.6.9 backport
Robert J Woodruff wrote, >New version of 2.6.9 backport patches committed in svn3854. >woody BTW. I was not able to test the pathscale driver as I do not have any of their H/W, so if someone that has H/W could test it, that would be great. woody ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [swg] RE: [openib-general] round 2 - proposal for socket based connection model
> -Original Message- > From: Caitlin Bestler [mailto:[EMAIL PROTECTED] > Sent: Tuesday, October 25, 2005 6:39 PM > To: Tom Tucker; Kanevsky, Arkady > Cc: [EMAIL PROTECTED]; openib-general@openib.org > Subject: [swg] RE: [openib-general] round 2 - proposal for socket based > connection model > > > > > -Original Message- > > From: [EMAIL PROTECTED] > > [mailto:[EMAIL PROTECTED] On Behalf Of Tom Tucker > > Sent: Tuesday, October 25, 2005 2:56 PM > > To: Kanevsky, Arkady > > Cc: [EMAIL PROTECTED]; openib-general@openib.org > > Subject: RE: [openib-general] round 2 - proposal for socket > > based connection model > > > > Arkady: > > > > I may actually have a constructive comment about the protocol > > (private data format). One thing I noticed is that *almost* > > everything in the private data header is available in the > > native iWARP protocol header except the ZB and SI bits. If > > these bits become part of the canonical private data header, > > then does that require an iWARP transport to use the header > > too even though only two bits are useful? > > > > Sorry if this is a dumb question, > > > > I'm not sure I followed why these were needed myself. I believe ZBTO and Remote Invalidation are mandatory in iWarp, right ? There are two new RDMA features that are available in iWarp, and are new to IB (optional in 1.2 version) A ULP that is supposed to run on both may want to know if the peer supports those, so it can use the correct verbs e.g. if the peer doesn't support remote invalidation the ULP will need to use Send verb, and invalidate the FMR locally, if it does support it, it can use the new "Send with Invalidate" verb which can improve performance and security I don't see why iWarp needs to negotiate it, CMA can just return true on both bits in case its iWarp This is a generic parameters that will be needed by more than one ULP, that wants to make sure what verbs are supported by the RDMA generic layer, that's why its in the generic portion of the header. Yaron ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] RE: [dat-discussions] round 2 - proposal forsocket based connection model
> -Original Message- > From: [EMAIL PROTECTED] [mailto:openib-general- > [EMAIL PROTECTED] On Behalf Of Kanevsky, Arkady > Sent: Tuesday, October 25, 2005 1:26 PM > To: Sean Hefty > Cc: [EMAIL PROTECTED]; openib-general@openib.org; dat- > [EMAIL PROTECTED] > Subject: RE: [openib-general] RE: [dat-discussions] round 2 - proposal > forsocket based connection model > > Think of a single API that supports iWARP and IB (transport independent > API). > To a connection listener it provides the IP 5-tuple + private data. > For IB it means that CM parses REQ and extracts IP 5-tuple as separate > fields from private data. > Listener does not parse the private data encoding of the proposal. > > So CM need to know if it need to encode IP 5-tuple on requestor side > and if need to parse on responder side. > Arkady > Arkady, I agree with Sean you can encode the Dest Port in the ServiceID And if you really want to verify its using that format you can look at the upper 48 bits in the serviceID. We may need to distinguish between Explicit RDMA protocols (iSER, NFS-RDMA, RDP, etc') and Implicit RDMA (SDP, where the Socket application doesn't know it is using RDMA), this can be done in 3 ways: a. port mapper, b. different ServiceID prefix, or c. a bit in the CM REQ Header. Also I'm not sure why we need the Protocol (UDP, TCP, SCTP, ..) since we emulate RDMA we shouldn't care if its TCP or SCTP, and UDP is unconnected and cant drive RDMA anyway Yaron > > Arkady Kanevsky email: [EMAIL PROTECTED] > Network Appliance phone: 781-768-5395 > 375 Totten Pond Rd. Fax: 781-895-1195 > Waltham, MA 02451-2010 central phone: 781-768-5300 > > > > > -Original Message- > > From: Sean Hefty [mailto:[EMAIL PROTECTED] > > Sent: Tuesday, October 25, 2005 1:08 PM > > To: Kanevsky, Arkady > > Cc: Caitlin Bestler; [EMAIL PROTECTED]; > > openib-general@openib.org; [EMAIL PROTECTED] > > Subject: Re: [openib-general] RE: [dat-discussions] round 2 - > > proposal for socket based connection model > > > > > > Kanevsky, Arkady wrote: > > > Correct. > > > But this does bring the question how responder CM knows > > that it need > > > to parse the private data. I suspect this will be done via > > new version > > > of CM. But a suage of some of the CM REQ reserved fields are also > > > possible. Anotherwords the current CM version assumes that CM only > > > supports one version and there is no need to support more than 1 > > > version. > > > > The responder knows how to parse the private data based on > > the service ID that > > they're listening on. This is how it's done today, and how > > it will still need > > to be done. What is the motivation to change it? > > > > What data is beyond the addressing? How does the responder > > know how to > > interpret that? > > > > - Sean > > > ___ > openib-general mailing list > openib-general@openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib- > general ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: RFC userspace CMA
Michael S. Tsirkin wrote: Quoting Sean Hefty <[EMAIL PROTECTED]>: I considered, and continue to consider implementing on top of ucm. The drawbacks are: it requires more kernel modules: one for the CM, one for SA query, and one for address translation. Cant address translation be done with exiting kernel/user interface? There's no kernel/user interface for ib_addr, which is what the kernel CMA uses. To use the ib_at kernel/user interface, ib_at would need to be fixed to avoid crashing the system. ib_addr is based off of the ib_at/sdp implementations, but limited to ARP translation only. It would also require userspace components for other RDMA CMs, such as iWarp. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] RFC userspace CMA
Tom Tucker wrote, +-+ | RDMA CM | +-+-+-+ | | ++ ++ | | +-+ +++ | IB CM | | IW CM | +++ +++ | | +_ +_ +-+|+-+| | IB devs ||| IW devs || +-+ +-+ >I welcome all comments on this especially now that it's early and >there's a lot of options and not much code yet. Looks like the right approach to me and I am glad to see someone start to implement an IW CM under the RDMA CM. This will allow any problems with the current RDMA CM API and implementation to be flushed out early so we can make any needed changes before too many people code to it. woody ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: RFC userspace CMA
Quoting Sean Hefty <[EMAIL PROTECTED]>: > I considered, and continue to consider implementing on top of ucm. The > drawbacks are: it requires more kernel modules: one for the CM, one for SA > query, and one for address translation. Cant address translation be done with exiting kernel/user interface? -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] RFC userspace CMA
Tom Tucker wrote: FYI, I've started writing the iw_cm that sits below the rdma_cm. Here's the general picture I have in mind. +-+ | RDMA CM | +-+-+-+ | | ++ ++ | | +-+ +++ | IB CM | | IW CM | +++ +++ | | +_ +_ +-+|+-+| | IB devs ||| IW devs || +-+ +-+ This is what I was envisioning as well. I am also migrating the current iw_cm.h file to match the interfaces in the rdma_cm more closely. Note that there are still some changes occurring to the rdma_cm to support userspace. I'm concerned about how well these changes map to iWarp, since the changes expose the three-way CM handshake used by IB. In general, the IW CM methods look very much like sockets connect, listen, and accept. There is an iw_cm_id like the ib_cm_id that encapsulates the 5-tuple, a callback for IW CM events and a "provider handle" that represents the adapter "connection cookie". The iw_cm_id is passed to connect, accept, etc... Something that didn't make sense for the kernel rdma_cm running over IB was adding a backlog parameter to the listen request. (The IB CM is callback driven, so there's not really a backlog.) I will probably add this to the userspace API. Does iWarp need a backlog parameter in the kernel? depending on the model. This means that calls like listen with a local port wildcard can't return until the "listen_reply" comes back from the adapter. I didn't quite follow this. Right now, the rdma_cm only tries to support wildcard IP addresses. Are you wanting to support listening on any port as well? What is a listen_reply? - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] round 2 - proposal for socketbased connectionmodel
Arkady wrote, >This is what we are trying to avoid. >ULP should not change regardless whether or not it is running >on IB, iWARP, VIA or any other RDMA transport. The whole point of the CMA is that the ULP can code to an API that is independent of RDMA interconnect. The CMA wire protocol can be documented to allow non-Linux hosts to connect to a Linux box using the same protocol. There is no need to change the existing IB CM protocol to accomplish this. All that is needed is to document that CMA protocol (contained in the private data field of the IB CM requests). woody ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: RFC userspace CMA
Michael S. Tsirkin wrote: Sounds like a lot of work :). I think that it's less work than the alternative. Are there benefits to this approach as opposed to implementing everything in a library on top of ucm/uverbs? I considered, and continue to consider implementing on top of ucm. The drawbacks are: it requires more kernel modules: one for the CM, one for SA query, and one for address translation. It complicates the event model, since the uCMA must now deal with events from three different sources, rather than one. And it duplicates a majority of the kernel CMA code in userspace. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] prototype version of ebus driver
on kernel 2.6.13 and 14 a "ebus" driver is needed to enable the ehca driver on power5. I just uploaded a prototype patch to gen2/users/ehca svn 3879 Christoph R.___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] The miracle.
You've seen it on "60 Minutes" and read the BBC News report -- now find out just what everyone is talking about. # Suppress your appetite and feel full and satisfied all day long # Increase your energy levels # Lose excess weight # Increase your metabolism # Burn body fat # Burn calories # Attack obesity And more.. http://coolhoodia.com/ # Suitable for vegetarians and vegans # MAINTAIN your weight loss # Make losing weight a sure guarantee # Look your best during the summer months http://coolhoodia.com/ Regards, Dr. Robin Talley ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: RFC userspace CMA
Quoting Sean Hefty <[EMAIL PROTECTED]>: > Subject: RFC userspace CMA > > I'm soliciting any comments that anyone might have on the general design for > the > userspace CMA before I get too far into the implementation. > > - The API will match the kernel API for the most part. The exception is that > event handling will match other userspace libraries (get/ack event). > > - There will be a single CMA device exported through /sys/class/infiniband. > > - The kernel CMA will be modified to remove the requirement to use > rdma_create_qp(). Users that want to allocate and manage their own QP states > will be able to specify QP attributes (qpn, qp_type, srq) through the > rdma_conn_param structure. > > - The kernel CMA will expose a new call, rdma_init_qp_attr() to initialize QP > attributes used to modify the state of the QP. The call will be similar to > the > infiniband CM routine. Use of this call is optional. The CMA will > automatically transition QPs created by rdma_create_qp(). > > - The uCMA will open devices for users and return them the device context > with > related events. The uCMA will close the device if there are no rdma_cma_id's > associated with it. > > - To support device add, the uCMA will need a new verb's call: > ibv_open_device_by_guid(). If a connection request occurs for a device that > is > not yet known by the uCMA, it will open the device. > > Comments? > > - Sean Sounds like a lot of work :). Are there benefits to this approach as opposed to implementing everything in a library on top of ucm/uverbs? -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Re: Re: [PATCH] perftest/rdma_bw; add support for RDMA read and starting PSN
Quoting Arlin Davis <[EMAIL PROTECTED]>: > Subject: Re: Re: [PATCH] perftest/rdma_bw;?add support for RDMA read and > starting PSN > > Michael S. Tsirkin wrote: > > >Thanks Arlin. I plan to look into integrating this. > >One question: for which psn values do you see performance drop on 4.6.0 > >FW? > > Any luck isolating this performance problem? I just want to understand > the cause so I know for sure 4.7 FW is a solid fix. Didn't see anything > in the 4.7 release notes that covered this issue. > Hi, Arlin. Unfortunately, I dont have an answer yet. This is not something we have intentionally fixed with 4.7, but it could be fixed as a byproduct of another fix. I plan to follow through and get back to you when we have an answer. Thanks, -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Osmtest Gen2 - Update
Title: [openib-general] Osmtest Gen2 - Update Hi , Gen2 Osmtest now support configuration flag (--with-top-tree=) for the following purposes : 1. Pre Define SOURCE userspace library & header files to compile and link with. 2. Compile Osmtest as part of the Gen2 stack For more info pls refer to README file. Link : https://openib.org/svn/trunk/contrib/mellanox/ibtp/gen2/userspace/management/osm/osmtest Liran Sorani Mellanox Technologies LTD. mailto:[EMAIL PROTECTED] Phone: +972(4)9097200 Ext: 214 Israel, Yokneam P.O.B 586 ZIP 20692 ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] RFC userspace CMA
Sean: FYI, I've started writing the iw_cm that sits below the rdma_cm. Here's the general picture I have in mind. +-+ | RDMA CM | +-+-+-+ | | ++ ++ | | +-+ +++ | IB CM | | IW CM | +++ +++ | | +_ +_ +-+|+-+| | IB devs ||| IW devs || +-+ +-+ The purpose of the IW CM is to abstract the two different connection models used by the iWARP side: offloaded and host integrated, and to act as a shim between device specific connection data structures and the rdma_cm data structures. I am also migrating the current iw_cm.h file to match the interfaces in the rdma_cm more closely. In general, the IW CM methods look very much like sockets connect, listen, and accept. There is an iw_cm_id like the ib_cm_id that encapsulates the 5-tuple, a callback for IW CM events and a "provider handle" that represents the adapter "connection cookie". The iw_cm_id is passed to connect, accept, etc... One big difference between the IW_CM and the IB_CM is that the IW_CM does not implement the connection state machine (three way handshake) like it does in the IB_CM. This greatly simplifies the code. Another big difference is that the IW_CM does not implement the service id database (port number space). This is either in the adapter or native stack depending on the model. This means that calls like listen with a local port wildcard can't return until the "listen_reply" comes back from the adapter. I welcome all comments on this especially now that it's early and there's a lot of options and not much code yet. On Tue, 2005-10-25 at 16:06 -0700, Sean Hefty wrote: > Sean Hefty wrote: > > - The kernel CMA will expose a new call, rdma_init_qp_attr() to > > initialize QP attributes used to modify the state of the QP. The call > > will be similar to the infiniband CM routine. Use of this call is > > optional. The CMA will automatically transition QPs created by > > rdma_create_qp(). > > The changes are more involved than this. To handle the QP transitions in > userspace, the kernel CMA needs to generate another event: CONNECT_RESPONSE. > It > will also need an additional API: rdma_establish(). (We can overload > rdma_accept() in place of rdma_establish().) Basically, the 3-way handshake > used by IB needs to be exposed. > > Use of either of these can be limited to those users that do not associate a > QP > with their rdma_cm_id. Alternatively, the uCMA kernel component can be > integrated with the kernel CMA and make use of private interfaces. > > - Sean > ___ > openib-general mailing list > openib-general@openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] question about poll_cq()
On Tue, 25 Oct 2005 09:19:25 -0700 Roland Dreier <[EMAIL PROTECTED]> wrote: > > A very brief sketch of what happens is that the device-specific > implementation of CQs for Mellanox HCAs allocates a circular buffer in > memory and passes the address to the hardware. The buffer is divided > into fixed-size chunks, each of which represents one completion entry. > Initially the buffer is cleared out, and every time the hardware adds > an entry onto the completion queue, it sets a bit in that chunk to > show that the entry is now valid. The driver polls the CQ by looking > to see if the next chunk has the bit set. If it does, then the driver > translates the entry from hardware format into standard struct ibv_wc > format; if it doesn't, then the driver returns status indicating that > the CQ is empty. > > Completion queues are always located in local system memory. > > - R. > thanks for your reply. that's all i wanted to know. joerg ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
