At the moment, our only "OOB" (out of band) module uses TCP sockets. This can use traditional ethernet or an emulated IP layer, such as IPoIB.
On Aug 24, 2011, at 11:58 AM, Bhargava Ramu Kavati wrote: > Hi Jeff, > Thank you for your prompt response. I have a query related to MPI_INIT here. > What is the underlying transport mechanism does OpenMPI uses to exchange > service-id/lid via MPI_INIT, is it TCP/IP socket ? > > Thanks & Regards, > Ramu > > Message: 2 > Date: Mon, 22 Aug 2011 17:33:19 -0400 > From: Jeff Squyres <jsquy...@cisco.com> > Subject: Re: [OMPI devel] Regarding Connection establishment in > OpenMPI > To: Open MPI Developers <de...@open-mpi.org> > Message-ID: <2399c470-7f91-49d4-a463-a8994691a...@cisco.com> > Content-Type: text/plain; charset=us-ascii > > On Aug 22, 2011, at 9:35 AM, Bhargava Ramu Kavati wrote: > > > I am trying to explore the details of connection establishment in OpenMPI > > using libibcm/librdmacm. > > Note that the IB community has given up on ibcm. Our support of it is > incomplete; I wouldn't look at it as an example. > > > In the code, I could not find how OpenMPI app is getting service-id/lid of > > remote node to which it wants to connect. > > In the normal case, we pass that information during MPI_INIT. It's a global > gather / broadcast operation that we refer to as the "modex" (module > exchange). I.e., each openib BTL module instance publishes its address > information in the modex and sends it. Near the end of MPI_INIT, each MPI > process receives the modex broadcast and caches it. > > During connection establishment, an MPI process will look in its modex cache > to find the connection information for the peer process that it wants to > connect to. > > > Also, I did not see any query in the code related to service_record_get > > from SA. Can you please desribe what is happening OR Am I missing > > something here ? > > IIRC, we don't currently use the SA because of its serialization and other > resource bottlenecks (this is a hand-waving answer; I don't remember the > exact reasons for not using the SA, but there were many discussions between > the MPI and OpenFabrics communities a long time ago. The SA issues were not > resolved to the MPI community's liking, IIRC, but this was a long time ago, > and I don't even work for an IB vendor any more, so I might not be > remembering this correctly...). > > -- > Jeff Squyres > jsquy...@cisco.com > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > > _______________________________________________ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
