Hi Jeff, As per our last discussion, MPI_INIT(..) uses TCP socket to exchange its service-id/lid with other MPI processes. I assume this applies irrespective of underlying library used to establish connection i.e libibcm or librdmacm. Please correct me if I am wrong.
Message: 1 > Date: Wed, 24 Aug 2011 12:06:30 -0400 > From: Jeff Squyres <jsquy...@cisco.com> > Subject: Re: [OMPI devel] Regarding Connection establishment in > OpenMPI (Jeff Squyres) > To: Open MPI Developers <de...@open-mpi.org> > Message-ID: <c730e008-dee5-4444-b2f2-5aa4fdd9a...@cisco.com> > Content-Type: text/plain; charset=us-ascii > > At the moment, our only "OOB" (out of band) module uses TCP sockets. This > can use traditional ethernet or an emulated IP layer, such as IPoIB. > > > On Aug 24, 2011, at 11:58 AM, Bhargava Ramu Kavati wrote: > > > Hi Jeff, > > Thank you for your prompt response. I have a query related to MPI_INIT > here. What is the underlying transport mechanism does OpenMPI uses to > exchange service-id/lid via MPI_INIT, is it TCP/IP socket ? > > > > Thanks & Regards, > > Ramu > > > > Message: 2 > > Date: Mon, 22 Aug 2011 17:33:19 -0400 > > From: Jeff Squyres <jsquy...@cisco.com> > > Subject: Re: [OMPI devel] Regarding Connection establishment in > > OpenMPI > > To: Open MPI Developers <de...@open-mpi.org> > > Message-ID: <2399c470-7f91-49d4-a463-a8994691a...@cisco.com> > > Content-Type: text/plain; charset=us-ascii > > > > On Aug 22, 2011, at 9:35 AM, Bhargava Ramu Kavati wrote: > > > > > I am trying to explore the details of connection establishment in > OpenMPI using libibcm/librdmacm. > > > > Note that the IB community has given up on ibcm. Our support of it is > incomplete; I wouldn't look at it as an example. > > > > > In the code, I could not find how OpenMPI app is getting service-id/lid > of remote node to which it wants to connect. > > > > In the normal case, we pass that information during MPI_INIT. It's a > global gather / broadcast operation that we refer to as the "modex" (module > exchange). I.e., each openib BTL module instance publishes its address > information in the modex and sends it. Near the end of MPI_INIT, each MPI > process receives the modex broadcast and caches it. > > > > During connection establishment, an MPI process will look in its modex > cache to find the connection information for the peer process that it wants > to connect to. > > > > > Also, I did not see any query in the code related to service_record_get > from SA. Can you please desribe what is happening OR Am I missing something > here ? > > > > IIRC, we don't currently use the SA because of its serialization and > other resource bottlenecks (this is a hand-waving answer; I don't remember > the exact reasons for not using the SA, but there were many discussions > between the MPI and OpenFabrics communities a long time ago. The SA issues > were not resolved to the MPI community's liking, IIRC, but this was a long > time ago, and I don't even work for an IB vendor any more, so I might not be > remembering this correctly...). > > > > -- > > Jeff Squyres > > jsquy...@cisco.com > > For corporate legal information go to: > > http://www.cisco.com/web/about/doing_business/legal/cri/ > > > > > > _______________________________________________ > > devel mailing list > > de...@open-mpi.org > > http://www.open-mpi.org/mailman/listinfo.cgi/devel > > > -- > Jeff Squyres > jsquy...@cisco.com > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > > > > ------------------------------ > >
