Re: [OMPI devel] [EXTERNAL] Re: [OMPI svn-full] svn:open-mpi r29703 - in trunk: contrib/platform/iu/odin ompi/mca/btl/openib ompi/mca/btl/openib/connect

>>> 3. Pasha moved the openib/connect to common/ofacm but excluded the rdmacm >>> in that move. Never changed openib to use ofacm/common. >> Pasha: This is not entirely true. I changed openib btl ~3 year ago before >> my departure from Mellanox. (I sent link to the code earlier). >> We

Re: [OMPI devel] [EXTERNAL] Re: [OMPI svn-full] svn:open-mpi r29703 - in trunk: contrib/platform/iu/odin ompi/mca/btl/openib ompi/mca/btl/openib/connect

On 11/14/2013 3:12 PM, Shamis, Pavel wrote: Comments inline. 3. Pasha moved the openib/connect to common/ofacm but excluded the rdmacm in that move. Never changed openib to use ofacm/common. Pasha: This is not entirely true. I changed openib btl ~3 year ago before my departure from

Re: [OMPI devel] [EXTERNAL] Re: [OMPI svn-full] svn:open-mpi r29703 - in trunk: contrib/platform/iu/odin ompi/mca/btl/openib ompi/mca/btl/openib/connect

On Nov 14, 2013, at 7:48 PM, Joshua Ladd wrote: > > The proof of the pudding is that all of the MPI layer has been adapted to the > new async behavior -except- for the openib cpc's. The issue of what to do > with these has been raised several times, especially once the

Re: [OMPI devel] [EXTERNAL] Re: [OMPI svn-full] svn:open-mpi r29703 - in trunk: contrib/platform/iu/odin ompi/mca/btl/openib ompi/mca/btl/openib/connect

The proof of the pudding is that all of the MPI layer has been adapted to the new async behavior -except- for the openib cpc's. The issue of what to do with these has been raised several times, especially once the ofacm code was committed. Unfortunately, lack of time and priorities left this

Re: [OMPI devel] [EXTERNAL] Re: [OMPI svn-full] svn:open-mpi r29703 - in trunk: contrib/platform/iu/odin ompi/mca/btl/openib ompi/mca/btl/openib/connect

On Nov 14, 2013, at 1:16 PM, Shamis, Pavel wrote: >> 1. Ralph made the OOB asynchronous. I pondered this for awhile today, and I just want to correct any misimpression this statement might leave, especially with folks who haven't been around the project that much over the

Re: [OMPI devel] [EXTERNAL] Re: [OMPI svn-full] svn:open-mpi r29703 - in trunk: contrib/platform/iu/odin ompi/mca/btl/openib ompi/mca/btl/openib/connect

Thanks Pasha!! On Nov 14, 2013, at 4:34 PM, Shamis, Pavel wrote: > For Iboffload this should not be an issue since our connection manager is > blocking (I have to double-check ) > > For openib, this should not be such huge change. The code is pretty much > standalone, we

Re: [OMPI devel] [EXTERNAL] Re: [OMPI svn-full] svn:open-mpi r29703 - in trunk: contrib/platform/iu/odin ompi/mca/btl/openib ompi/mca/btl/openib/connect

For Iboffload this should not be an issue since our connection manager is blocking (I have to double-check ) For openib, this should not be such huge change. The code is pretty much standalone, we only have to move it to main thread and add signaling mechanism. I will take a look. Best,

Re: [OMPI devel] [EXTERNAL] Re: [OMPI svn-full] svn:open-mpi r29703 - in trunk: contrib/platform/iu/odin ompi/mca/btl/openib ompi/mca/btl/openib/connect

On Nov 14, 2013, at 4:22 PM, Shamis, Pavel wrote: > Well, this is major change in a behavior. > > Since openib calls communication calls from the callback > it pretty much requires to enable thread safety on openib btl level. Ah, yes - could well be true. Or else separate

Re: [OMPI devel] [EXTERNAL] Re: [OMPI svn-full] svn:open-mpi r29703 - in trunk: contrib/platform/iu/odin ompi/mca/btl/openib ompi/mca/btl/openib/connect

Well, this is major change in a behavior. Since openib calls communication calls from the callback it pretty much requires to enable thread safety on openib btl level. But we may move the queue flush operation from the callback to main thread, so the progress engine will wait on a signal from

Re: [OMPI devel] [EXTERNAL] Re: [OMPI svn-full] svn:open-mpi r29703 - in trunk: contrib/platform/iu/odin ompi/mca/btl/openib ompi/mca/btl/openib/connect

On Nov 14, 2013, at 3:33 PM, Shamis, Pavel wrote: > >> The only change is that the receive callback is now occurring in the ORTE >> event thread, and so perhaps someone needs to look at a way to pass that >> back into the OMPI event base (which I guess is the OPAL event

Re: [OMPI devel] [EXTERNAL] Re: [OMPI svn-full] svn:open-mpi r29703 - in trunk: contrib/platform/iu/odin ompi/mca/btl/openib ompi/mca/btl/openib/connect

> The only change is that the receive callback is now occurring in the ORTE > event thread, and so perhaps someone needs to look at a way to pass that back > into the OMPI event base (which I guess is the OPAL event base)? Just > glancing at the code, it looks like that could be the issue -

Re: [OMPI devel] [EXTERNAL] Re: [OMPI svn-full] svn:open-mpi r29703 - in trunk: contrib/platform/iu/odin ompi/mca/btl/openib ompi/mca/btl/openib/connect

On Nov 14, 2013, at 3:07 PM, Shamis, Pavel wrote: > >> So far as I can tell, the issue is one of blocking. The OOB handshake is now >> async - i.e., you post a non-blocking recv at the beginning of time, and >> then do a non-blocking send to the other side when you want to

Re: [OMPI devel] [EXTERNAL] Re: [OMPI svn-full] svn:open-mpi r29703 - in trunk: contrib/platform/iu/odin ompi/mca/btl/openib ompi/mca/btl/openib/connect

> So far as I can tell, the issue is one of blocking. The OOB handshake is now > async - i.e., you post a non-blocking recv at the beginning of time, and then > do a non-blocking send to the other side when you want to create a > connection. The question is: how do you know when that

Re: [OMPI devel] [EXTERNAL] Re: [OMPI svn-full] svn:open-mpi r29703 - in trunk: contrib/platform/iu/odin ompi/mca/btl/openib ompi/mca/btl/openib/connect

So far as I can tell, the issue is one of blocking. The OOB handshake is now async - i.e., you post a non-blocking recv at the beginning of time, and then do a non-blocking send to the other side when you want to create a connection. The question is: how do you know when that connection is

Re: [OMPI devel] [EXTERNAL] Re: [OMPI svn-full] svn:open-mpi r29703 - in trunk: contrib/platform/iu/odin ompi/mca/btl/openib ompi/mca/btl/openib/connect

> > 1. Ralph made the OOB asynchronous. > Ralph, I'm not familiar with details of the change. If out-of-band communication is supported, it should not be that huge change for XOOB/OOB.

Re: [OMPI devel] [EXTERNAL] Re: [OMPI svn-full] svn:open-mpi r29703 - in trunk: contrib/platform/iu/odin ompi/mca/btl/openib ompi/mca/btl/openib/connect

Comments inline. > > 3. Pasha moved the openib/connect to common/ofacm but excluded the rdmacm in > that move. Never changed openib to use ofacm/common. Pasha: This is not entirely true. I changed openib btl ~3 year ago before my departure from Mellanox. (I sent link to the code earlier).

Re: [OMPI devel] [EXTERNAL] Re: [OMPI svn-full] svn:open-mpi r29703 - in trunk: contrib/platform/iu/odin ompi/mca/btl/openib ompi/mca/btl/openib/connect

n MPI Developers > Cc: Yiftah Shahar; Gilad Shainer > Subject: Re: [OMPI devel] [EXTERNAL] Re: [OMPI svn-full] svn:open-mpi r29703 > - in trunk: contrib/platform/iu/odin ompi/mca/btl/openib > ompi/mca/btl/openib/connect > > > On Nov 14, 2013, at 12:21 PM, Barrett, Br

Re: [OMPI devel] [EXTERNAL] Re: [OMPI svn-full] svn:open-mpi r29703 - in trunk: contrib/platform/iu/odin ompi/mca/btl/openib ompi/mca/btl/openib/connect

, 2013 3:32 PM To: Open MPI Developers Cc: Yiftah Shahar; Gilad Shainer Subject: Re: [OMPI devel] [EXTERNAL] Re: [OMPI svn-full] svn:open-mpi r29703 - in trunk: contrib/platform/iu/odin ompi/mca/btl/openib ompi/mca/btl/openib/connect On Nov 14, 2013, at 12:21 PM, Barrett, Brian W <b

Re: [OMPI devel] [EXTERNAL] Re: [OMPI svn-full] svn:open-mpi r29703 - in trunk: contrib/platform/iu/odin ompi/mca/btl/openib ompi/mca/btl/openib/connect

On Nov 14, 2013, at 12:21 PM, Barrett, Brian W wrote: > On 11/14/13 1:13 PM, "Joshua Ladd" wrote: > >> Let me try to summarize my understanding of the situation: >> >> 1. Ralph made the OOB asynchronous. >> >> 2. OOB cpcs don't work as a result of

Re: [OMPI devel] [EXTERNAL] Re: [OMPI svn-full] svn:open-mpi r29703 - in trunk: contrib/platform/iu/odin ompi/mca/btl/openib ompi/mca/btl/openib/connect

On 11/14/13 1:13 PM, "Joshua Ladd" wrote: >Let me try to summarize my understanding of the situation: > >1. Ralph made the OOB asynchronous. > >2. OOB cpcs don't work as a result of 1, and are thereby "deprecated", >meaning: won't fix. > >3. Pasha moved the openib/connect

Re: [OMPI devel] [EXTERNAL] Re: [OMPI svn-full] svn:open-mpi r29703 - in trunk: contrib/platform/iu/odin ompi/mca/btl/openib ompi/mca/btl/openib/connect

n Behalf Of Shamis, Pavel Sent: Thursday, November 14, 2013 2:08 PM To: Open MPI Developers Subject: Re: [OMPI devel] [EXTERNAL] Re: [OMPI svn-full] svn:open-mpi r29703 - in trunk: contrib/platform/iu/odin ompi/mca/btl/openib ompi/mca/btl/openib/connect There is some confusion in the thread. U

Re: [OMPI devel] [EXTERNAL] Re: [OMPI svn-full] svn:open-mpi r29703 - in trunk: contrib/platform/iu/odin ompi/mca/btl/openib ompi/mca/btl/openib/connect

:58 PM > To: Open MPI Developers > Subject: Re: [OMPI devel] [EXTERNAL] Re: [OMPI svn-full]svn:open-mpi > r29703 - in trunk: contrib/platform/iu/odinompi/mca/btl/openib > ompi/mca/btl/openib/connect > > The key question, though, is: has anyone

Re: [OMPI devel] [EXTERNAL] Re: [OMPI svn-full] svn:open-mpi r29703 - in trunk: contrib/platform/iu/odin ompi/mca/btl/openib ompi/mca/btl/openib/connect

f you know different. >> >> >> Josh >> >> >> -----Original Message- >> From: devel [mailto:devel-boun...@open-mpi.org] On Behalf Of Ralph Castain >> Sent: Thursday, November 14, 2013 1:05 PM >> To: Open MPI Developers >> Subject: Re

Re: [OMPI devel] [EXTERNAL] Re: [OMPI svn-full] svn:open-mpi r29703 - in trunk: contrib/platform/iu/odin ompi/mca/btl/openib ompi/mca/btl/openib/connect

know different. >> >> >> Josh >> >> >> -----Original Message----- >> From: devel [mailto:devel-boun...@open-mpi.org] On Behalf Of Ralph Castain >> Sent: Thursday, November 14, 2013 1:05 PM >> To: Open MPI Developers >> Subject: Re: [OMP

Re: [OMPI devel] [EXTERNAL] Re: [OMPI svn-full] svn:open-mpi r29703 - in trunk: contrib/platform/iu/odin ompi/mca/btl/openib ompi/mca/btl/openib/connect

gt; To: Open MPI Developers > Subject: Re: [OMPI devel] [EXTERNAL] Re: [OMPI svn-full] svn:open-mpi r29703 > - in trunk: contrib/platform/iu/odin ompi/mca/btl/openib > ompi/mca/btl/openib/connect > > > On Nov 14, 2013, at 9:33 AM, Barrett, Brian W <bwba...@sandia.gov> wrote

Re: [OMPI devel] [EXTERNAL] Re: [OMPI svn-full] svn:open-mpi r29703 - in trunk: contrib/platform/iu/odin ompi/mca/btl/openib ompi/mca/btl/openib/connect

n:open-mpi r29703 - in trunk: contrib/platform/iu/odin ompi/mca/btl/openib ompi/mca/btl/openib/connect On Nov 14, 2013, at 9:33 AM, Barrett, Brian W <bwba...@sandia.gov> wrote: > On 11/14/13 9:51 AM, "Jeff Squyres (jsquyres)" <jsquy...@cisco.com> wrote: > >> Does X

Re: [OMPI devel] [EXTERNAL] Re: [OMPI svn-full] svn:open-mpi r29703 - in trunk: contrib/platform/iu/odin ompi/mca/btl/openib ompi/mca/btl/openib/connect

On Nov 14, 2013, at 9:33 AM, Barrett, Brian W wrote: > On 11/14/13 9:51 AM, "Jeff Squyres (jsquyres)" wrote: > >> Does XRC work with the UDCM CPC? >> >> >> On Nov 14, 2013, at 9:35 AM, Ralph Castain wrote: >> >>> I think the

Re: [OMPI devel] [EXTERNAL] Re: [OMPI svn-full] svn:open-mpi r29703 - in trunk: contrib/platform/iu/odin ompi/mca/btl/openib ompi/mca/btl/openib/connect

On 11/14/13 9:51 AM, "Jeff Squyres (jsquyres)" wrote: >Does XRC work with the UDCM CPC? > > >On Nov 14, 2013, at 9:35 AM, Ralph Castain wrote: > >> I think the problems in udcm were fixed by Nathan quite some time ago, >>but never moved to 1.7 as everyone