RE: [openib-general] RFC MPI and app. requirements of OpenIB
On Thu, 2005-12-22 at 16:54, Rimmer, Todd wrote: > Since the SA views multicast operations at a node level, and applications > need to individually operate on multicast groups. It is appropriate for > the core stack to provide some multicast management APIs. These would > multiplex requests from all the applications on a node to make the > appropriate requests to the SA. reference counts would need to be > maintained in the core stack for each MC group so that the node would > remove itself from the group only on last application exit/unregister. > > This would also be a good place to handle the "MC group persistence issues". > Namely rejoining requested groups when ports go up/down, SMs change > (client reregister), etc. Yes, this has been raised in a thread by Eitan started on 11/22 entitled "First Multicast Leave disconnects all other clients" (http://openib.org/pipermail/openib-general/2005-November/014023.html). -- Hal > Todd Rimmer > Chief Systems Architect SilverStorm Technologies > Voice: 610-233-4852 Fax: 610-233-4777 > [EMAIL PROTECTED] www.SilverStorm.com > > > > -Original Message- > > From: amith rajith mamidala [mailto:[EMAIL PROTECTED] > > Sent: Thursday, December 22, 2005 4:39 PM > > To: Sean Hefty > > Cc: openib > > Subject: RE: [openib-general] RFC MPI and app. requirements of OpenIB > > > > > > > > > An alternative is to provide UD and multicast/broadcast > > support in the CMA. I > > > know that the Intel MPI runs over DAPL, which does not > > provide multicast > > > support. Can MPI operate with unreliable multicast > > support? Does MPI plan on > > > using IB multicast? > > > > Yes, the MPI can operate with unreliable multicast support. > > MVAPICH-0.9.6 has this broadcast support over IB multicast. As Hal > > suggested earlier, application processes interact with SA to > > create/join > > multicast groups. > > > > Thanks, > > Amith > > > > > > > > > ___ > > > openib-general mailing list > > > openib-general@openib.org > > > http://openib.org/mailman/listinfo/openib-general > > > > > > To unsubscribe, please visit > > http://openib.org/mailman/listinfo/openib-general > > > > > > > ___ > > openib-general mailing list > > openib-general@openib.org > > http://openib.org/mailman/listinfo/openib-general > > > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > ___ > openib-general mailing list > openib-general@openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] RFC MPI and app. requirements of OpenIB
Since the SA views multicast operations at a node level, and applications need to individually operate on multicast groups. It is appropriate for the core stack to provide some multicast management APIs. These would multiplex requests from all the applications on a node to make the appropriate requests to the SA. reference counts would need to be maintained in the core stack for each MC group so that the node would remove itself from the group only on last application exit/unregister. This would also be a good place to handle the "MC group persistence issues". Namely rejoining requested groups when ports go up/down, SMs change (client reregister), etc. Todd Rimmer Chief Systems Architect SilverStorm Technologies Voice: 610-233-4852 Fax: 610-233-4777 [EMAIL PROTECTED] www.SilverStorm.com > -Original Message- > From: amith rajith mamidala [mailto:[EMAIL PROTECTED] > Sent: Thursday, December 22, 2005 4:39 PM > To: Sean Hefty > Cc: openib > Subject: RE: [openib-general] RFC MPI and app. requirements of OpenIB > > > > > An alternative is to provide UD and multicast/broadcast > support in the CMA. I > > know that the Intel MPI runs over DAPL, which does not > provide multicast > > support. Can MPI operate with unreliable multicast > support? Does MPI plan on > > using IB multicast? > > Yes, the MPI can operate with unreliable multicast support. > MVAPICH-0.9.6 has this broadcast support over IB multicast. As Hal > suggested earlier, application processes interact with SA to > create/join > multicast groups. > > Thanks, > Amith > > > > > > ___ > > openib-general mailing list > > openib-general@openib.org > > http://openib.org/mailman/listinfo/openib-general > > > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > > > > ___ > openib-general mailing list > openib-general@openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] RFC MPI and app. requirements of OpenIB
> An alternative is to provide UD and multicast/broadcast support in the CMA. I > know that the Intel MPI runs over DAPL, which does not provide multicast > support. Can MPI operate with unreliable multicast support? Does MPI plan on > using IB multicast? Yes, the MPI can operate with unreliable multicast support. MVAPICH-0.9.6 has this broadcast support over IB multicast. As Hal suggested earlier, application processes interact with SA to create/join multicast groups. Thanks, Amith > > > ___ > openib-general mailing list > openib-general@openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] RFC MPI and app. requirements of OpenIB
On Thu, Dec 22, 2005 at 12:14:44PM -0800, Sean Hefty wrote: > Can MPI operate with unreliable multicast support? Does MPI plan on > using IB multicast? Given the large number of MPI implementations over IB, I don't think there's a single answer. -- greg ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] RFC MPI and app. requirements of OpenIB
>> To help clarify the trade-offs: >> >> The CMA allows the use of IP addressing for connection establishment and >> abstracts device hotplug. It also operates over any type of RDMA device. >> >> A disadvantage of using the CMA is that it may not select the best set of >paths >> between two or more nodes. > >What defines best ? Is this preference or disjointedness or something >else ? I was intentionally vague here to leave this up to the application developer to define. The application may decide that a particular path or set of paths is better than another based on whatever criteria they choose. The current CMA provides less control over which paths are selected for connections than if the user queried the SA for paths and selected one based on some algorithm. (I'd be surprised if an app actually did this though.) >> The IB CM also permits path failover on a single >> HCA. Use of the IB CM requires that clients also interface with the IB SA to >> obtain path records. > >Note that interaction with the SA will be required for MPI when >multicast groups are to be used. An alternative is to provide UD and multicast/broadcast support in the CMA. I know that the Intel MPI runs over DAPL, which does not provide multicast support. Can MPI operate with unreliable multicast support? Does MPI plan on using IB multicast? >> My personal recommendation would be for applications to use the CMA, but that >> does result in losing some flexibility. > >Would the CMA ultimately support path failover ? Only if there's enough demand. Since IB failover is restricted to a single HCA, I can see where a more robust failover mechanism would be desirable. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] RFC MPI and app. requirements of OpenIB
On Thu, 2005-12-22 at 14:34, Sean Hefty wrote: > >I'm soliciting feedback from the MPI and other application developers > >regarding > >which OpenIB APIs they will be targeting with their implementations. > >Specifically, myself and some of the other IB developers are interesting in > >knowing if userspace applications will be written to the RDMA CMA interface, > >the > >IB CM API, or some other abstraction. > > To help clarify the trade-offs: > > The CMA allows the use of IP addressing for connection establishment and > abstracts device hotplug. It also operates over any type of RDMA device. > > A disadvantage of using the CMA is that it may not select the best set of > paths > between two or more nodes. What defines best ? Is this preference or disjointedness or something else ? Note path selection may be important in subnets when LMC > 0. > The IB CM also permits path failover on a single > HCA. Use of the IB CM requires that clients also interface with the IB SA to > obtain path records. Note that interaction with the SA will be required for MPI when multicast groups are to be used. > My personal recommendation would be for applications to use the CMA, but that > does result in losing some flexibility. Would the CMA ultimately support path failover ? -- Hal ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] RFC MPI and app. requirements of OpenIB
>I'm soliciting feedback from the MPI and other application developers regarding >which OpenIB APIs they will be targeting with their implementations. >Specifically, myself and some of the other IB developers are interesting in >knowing if userspace applications will be written to the RDMA CMA interface, >the >IB CM API, or some other abstraction. To help clarify the trade-offs: The CMA allows the use of IP addressing for connection establishment and abstracts device hotplug. It also operates over any type of RDMA device. A disadvantage of using the CMA is that it may not select the best set of paths between two or more nodes. The IB CM also permits path failover on a single HCA. Use of the IB CM requires that clients also interface with the IB SA to obtain path records. My personal recommendation would be for applications to use the CMA, but that does result in losing some flexibility. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] RFC MPI and app. requirements of OpenIB
I'm soliciting feedback from the MPI and other application developers regarding which OpenIB APIs they will be targeting with their implementations. Specifically, myself and some of the other IB developers are interesting in knowing if userspace applications will be written to the RDMA CMA interface, the IB CM API, or some other abstraction. This will let us focus our priorities on which features to expose to userspace. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
