Re: [OMPI devel] OpenMPI 2.0 and Petsc 3.7.2

2016-07-26 Thread Gilles Gouaillardet 

Nathan and Eric,


there is a know issue of libnbc not correctly retaining datatypes, for 
example, if you start a non blocking collective operation (MPI_Ibcast 
for example) and they MPI_Type_destroy() the datatype *before* the non 
blocking collective completes, then the datatype is free'd by 
MPI_Type_destroy, and is invalid when required to progress the 
collective operation.



a patch for v2.x is available at 
https://github.com/ggouaillardet/ompi-release/commit/cd30056efeff7d37257ab2bc0dbffb2e05a6170c.patch


/* note it only fixes MPI_Ibcast() for now, but if the root cause of the 
petsc crash is this, i can easily fix other non blocking primitives */



/* ideally, this should be done at the ompi level, and not in the libnbc 
module, that is why https://github.com/open-mpi/ompi/pull/1305 has not 
been merged yet */



Cheers,


Gilles


On 7/26/2016 11:46 AM, Nathan Hjelm wrote:

It looks to me like double free on both send and receive requests. The receive 
free is an extra OBJ_RELEASE of MPI_DOUBLE which was not malloced (invalid 
free). The send free is an assert failure in OBJ_RELEASE of an OBJ_NEW() object 
(invalid magic). I plan to look at in in the next couple of days. Let me know 
if you figure it out before I get to it.

-Nathan


On Jul 25, 2016, at 8:38 PM, Gilles Gouaillardet  wrote:

Eric,

where can your test case be downloaded ? how many nodes and tasks do you need 
to reproduce the bug ?

fwiw, currently there are two Open MPI repositories
- https://github.com/open-mpi/ompi
  there is only one branch and is the 'master' branch, today, this can be seen 
as Open MPI 3.0 pre alpha
- https://github.com/open-mpi/ompi-release
  the default branch is 'v2.x', today, this can be seen as Open MPI 2.0.1 pre 
alpha

Cheers,

Gilles

On 7/26/2016 3:33 AM, Eric Chamberland wrote:

Hi,

has someone tried OpenMPI 2.0 with Petsc 3.7.2?

I am having some errors with petsc, maybe someone have them too?

Here are the configure logs for PETSc:

http://www.giref.ulaval.ca/~cmpgiref/dernier_ompi/2016.07.25.01h16m02s_configure.log

http://www.giref.ulaval.ca/~cmpgiref/dernier_ompi/2016.07.25.01h16m02s_RDict.log

And for OpenMPI:
http://www.giref.ulaval.ca/~cmpgiref/dernier_ompi/2016.07.25.01h16m02s_config.log

(in fact, I am testing the ompi-release branch, a sort of petsc-master branch, 
since I need the commit 9ba6678156).

For a set of parallel tests, I have 104 that works on 124 total tests.

And the typical error:
*** Error in 
`/pmi/cmpbib/compilation_BIB_dernier_ompi/COMPILE_AUTO/GIREF/bin/Test.ProblemeGD.dev':
 free(): invalid pointer:
=== Backtrace: =
/lib64/libc.so.6(+0x7277f)[0x7f80eb11677f]
/lib64/libc.so.6(+0x78026)[0x7f80eb11c026]
/lib64/libc.so.6(+0x78d53)[0x7f80eb11cd53]
/opt/openmpi-2.x_opt/lib/libopen-pal.so.20(opal_free+0x1f)[0x7f80ea8f9d60]
/opt/openmpi-2.x_opt/lib/openmpi/mca_pml_ob1.so(+0x16628)[0x7f80df0ea628]
/opt/openmpi-2.x_opt/lib/openmpi/mca_pml_ob1.so(+0x16c50)[0x7f80df0eac50]
/opt/openmpi-2.x_opt/lib/libmpi.so.20(+0x9f9dd)[0x7f80eb7029dd]
/opt/openmpi-2.x_opt/lib/libmpi.so.20(MPI_Request_free+0xf7)[0x7f80eb702ad6]
/opt/petsc-3.7.2_debug_openmpi_2.x/lib/libpetsc.so.3.7(+0x4adc6d)[0x7f80f2fa6c6d]
/opt/petsc-3.7.2_debug_openmpi_2.x/lib/libpetsc.so.3.7(VecScatterDestroy+0x68d)[0x7f80f2fa1c45]
/opt/petsc-3.7.2_debug_openmpi_2.x/lib/libpetsc.so.3.7(+0xa9d0f5)[0x7f80f35960f5]
/opt/petsc-3.7.2_debug_openmpi_2.x/lib/libpetsc.so.3.7(MatDestroy+0x648)[0x7f80f35c2588]
/opt/petsc-3.7.2_debug_openmpi_2.x/lib/libpetsc.so.3.7(+0x10bf0f4)[0x7f80f3bb80f4]
/opt/petsc-3.7.2_debug_openmpi_2.x/lib/libpetsc.so.3.7(PCReset+0x346)[0x7f80f3a796de]
/opt/petsc-3.7.2_debug_openmpi_2.x/lib/libpetsc.so.3.7(KSPReset+0x502)[0x7f80f3d19779]
/opt/petsc-3.7.2_debug_openmpi_2.x/lib/libpetsc.so.3.7(+0x11707f7)[0x7f80f3c697f7]
/opt/petsc-3.7.2_debug_openmpi_2.x/lib/libpetsc.so.3.7(PCReset+0x346)[0x7f80f3a796de]
/opt/petsc-3.7.2_debug_openmpi_2.x/lib/libpetsc.so.3.7(KSPReset+0x502)[0x7f80f3d19779]
/opt/petsc-3.7.2_debug_openmpi_2.x/lib/libpetsc.so.3.7(+0x11707f7)[0x7f80f3c697f7]
/opt/petsc-3.7.2_debug_openmpi_2.x/lib/libpetsc.so.3.7(PCReset+0x346)[0x7f80f3a796de]
/opt/petsc-3.7.2_debug_openmpi_2.x/lib/libpetsc.so.3.7(KSPReset+0x502)[0x7f80f3d19779]
/opt/petsc-3.7.2_debug_openmpi_2.x/lib/libpetsc.so.3.7(+0x11707f7)[0x7f80f3c697f7]
/opt/petsc-3.7.2_debug_openmpi_2.x/lib/libpetsc.so.3.7(PCReset+0x346)[0x7f80f3a796de]
/opt/petsc-3.7.2_debug_openmpi_2.x/lib/libpetsc.so.3.7(PCDestroy+0x5d1)[0x7f80f3a79fd9]
/opt/petsc-3.7.2_debug_openmpi_2.x/lib/libpetsc.so.3.7(KSPDestroy+0x7b6)[0x7f80f3d1a334]

a similar one:
*** Error in 
`/pmi/cmpbib/compilation_BIB_dernier_ompi/COMPILE_AUTO/GIREF/bin/Test.ProbFluideIncompressible.dev':
 free(): invalid pointer: 0x7f382a7c5bc0 ***
=== Backtrace: =
/lib64/libc.so.6(+0x7277f)[0x7f3829f1c77f]
/lib64/libc.so.6(+0x78026)[0x7f3829f22026]
/lib64/libc.so.6(+0x78d53)[0x7f3829f22d53]
/opt/openmpi-2.x_opt/lib/libopen-pal.so.20(opal_free+0x1f)[0x7f38296ffd60]
/opt/open

Re: [OMPI devel] tcp btl rendezvous performance question

2016-07-26 Thread Sreenidhi Bharathkar Ramesh 
hi Howard,

Was this issue resolved ?  If so, what is the solution ?
Please let me know.
Curious to know , since we are also experimenting with these limits.

Thanks,
- Sreenidhi.


On Tue, Jul 19, 2016 at 10:50 AM, Gilles Gouaillardet 
wrote:

> Howard,
>
>
> did you bump both btl_tcp_rndv_eager_limit and btl_tcp_eager_limit ?
>
> you might also need to bump btl_tcp_sndbuf, btl_tcp_rcvbuf and
> btl_tcp_max_send_size to get the max performance out of your 100Gb ethernet
> cards
>
> last but not least, you might also need to bump btl_tcp_links to saturate
> your network (that is likely a good thing when running 1 task per node, but
> that can lead to decreased performance when running several tasks per node)
>
> Cheers,
>
>
> Gilles
>
> On 7/19/2016 6:57 AM, Howard Pritchard wrote:
>
> Hi Folks,
>
> I have a cluster with some 100 Gb ethernet cards
> installed.  What we are noticing if we force Open MPI 1.10.3
> to go through the TCP BTL (rather than yalla)  is that
> the performance of osu_bw once the TCP BTL switches
> from eager to rendezvous (> 32KB)
> falls off a cliff, going from about 1.6 GB/sec to 233 MB/sec
> and stays that way out to 4 MB message lengths.
>
> There's nothing wrong with the IP stack (iperf -P4 gives
> 63 Gb/sec).
>
> So, my questions are
>
> 1) is this performance expected for the TCP BTL when in
> rendezvous mode?
> 2) is there some way to get more like the single socket
> performance obtained with iperf for large messages (~16 Gb/sec).
>
> We tried adjusting the tcp_btl_rendezvous threshold but that doesn't
> appear to actually be adjustable from the mpirun command line.
>
> Thanks for any suggestions,
>
> Howard
>
>
>
>
>
> ___
> devel mailing listde...@open-mpi.org
> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post: 
> http://www.open-mpi.org/community/lists/devel/2016/07/19237.php
>
>
>
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2016/07/19240.php
>

Re: [OMPI devel] OpenMPI 2.0 and Petsc 3.7.2

2016-07-26 Thread Eric Chamberland 

Hi Gilles,


On 25/07/16 10:38 PM, Gilles Gouaillardet wrote:

Eric,

where can your test case be downloaded ? how many nodes and tasks do you
need to reproduce the bug ?


Sadly, it is in our in-house code and it requires to whole source code 
which isn't public... :/


I have this bug with 20 parallel tests from our 124 tests database, 
running with 2 to 10 processes (but 2 for most of them).


The bug is happening at the very end of the execution (FE 
resolution+exports), when everything get destroyed, including PETSc stuff.


Unfortunately, running "make test" and "make testexamples" at the end of 
petsc installation doesn't trigger the bug... :/




fwiw, currently there are two Open MPI repositories
- https://github.com/open-mpi/ompi
  there is only one branch and is the 'master' branch, today, this can
be seen as Open MPI 3.0 pre alpha
- https://github.com/open-mpi/ompi-release
  the default branch is 'v2.x', today, this can be seen as Open MPI
2.0.1 pre alpha


I tested both...  I reported the error also for the "master" of ompi, 
and they seems related to me, see: 
https://github.com/open-mpi/ompi/issues/1875


Thanks,

Eric



Cheers,

Gilles

On 7/26/2016 3:33 AM, Eric Chamberland wrote:

Hi,

has someone tried OpenMPI 2.0 with Petsc 3.7.2?

I am having some errors with petsc, maybe someone have them too?

Here are the configure logs for PETSc:

http://www.giref.ulaval.ca/~cmpgiref/dernier_ompi/2016.07.25.01h16m02s_configure.log


http://www.giref.ulaval.ca/~cmpgiref/dernier_ompi/2016.07.25.01h16m02s_RDict.log


And for OpenMPI:
http://www.giref.ulaval.ca/~cmpgiref/dernier_ompi/2016.07.25.01h16m02s_config.log


(in fact, I am testing the ompi-release branch, a sort of petsc-master
branch, since I need the commit 9ba6678156).

For a set of parallel tests, I have 104 that works on 124 total tests.

And the typical error:
*** Error in
`/pmi/cmpbib/compilation_BIB_dernier_ompi/COMPILE_AUTO/GIREF/bin/Test.ProblemeGD.dev':
free(): invalid pointer:
=== Backtrace: =
/lib64/libc.so.6(+0x7277f)[0x7f80eb11677f]
/lib64/libc.so.6(+0x78026)[0x7f80eb11c026]
/lib64/libc.so.6(+0x78d53)[0x7f80eb11cd53]
/opt/openmpi-2.x_opt/lib/libopen-pal.so.20(opal_free+0x1f)[0x7f80ea8f9d60]

/opt/openmpi-2.x_opt/lib/openmpi/mca_pml_ob1.so(+0x16628)[0x7f80df0ea628]
/opt/openmpi-2.x_opt/lib/openmpi/mca_pml_ob1.so(+0x16c50)[0x7f80df0eac50]
/opt/openmpi-2.x_opt/lib/libmpi.so.20(+0x9f9dd)[0x7f80eb7029dd]
/opt/openmpi-2.x_opt/lib/libmpi.so.20(MPI_Request_free+0xf7)[0x7f80eb702ad6]

/opt/petsc-3.7.2_debug_openmpi_2.x/lib/libpetsc.so.3.7(+0x4adc6d)[0x7f80f2fa6c6d]

/opt/petsc-3.7.2_debug_openmpi_2.x/lib/libpetsc.so.3.7(VecScatterDestroy+0x68d)[0x7f80f2fa1c45]

/opt/petsc-3.7.2_debug_openmpi_2.x/lib/libpetsc.so.3.7(+0xa9d0f5)[0x7f80f35960f5]

/opt/petsc-3.7.2_debug_openmpi_2.x/lib/libpetsc.so.3.7(MatDestroy+0x648)[0x7f80f35c2588]

/opt/petsc-3.7.2_debug_openmpi_2.x/lib/libpetsc.so.3.7(+0x10bf0f4)[0x7f80f3bb80f4]

/opt/petsc-3.7.2_debug_openmpi_2.x/lib/libpetsc.so.3.7(PCReset+0x346)[0x7f80f3a796de]

/opt/petsc-3.7.2_debug_openmpi_2.x/lib/libpetsc.so.3.7(KSPReset+0x502)[0x7f80f3d19779]

/opt/petsc-3.7.2_debug_openmpi_2.x/lib/libpetsc.so.3.7(+0x11707f7)[0x7f80f3c697f7]

/opt/petsc-3.7.2_debug_openmpi_2.x/lib/libpetsc.so.3.7(PCReset+0x346)[0x7f80f3a796de]

/opt/petsc-3.7.2_debug_openmpi_2.x/lib/libpetsc.so.3.7(KSPReset+0x502)[0x7f80f3d19779]

/opt/petsc-3.7.2_debug_openmpi_2.x/lib/libpetsc.so.3.7(+0x11707f7)[0x7f80f3c697f7]

/opt/petsc-3.7.2_debug_openmpi_2.x/lib/libpetsc.so.3.7(PCReset+0x346)[0x7f80f3a796de]

/opt/petsc-3.7.2_debug_openmpi_2.x/lib/libpetsc.so.3.7(KSPReset+0x502)[0x7f80f3d19779]

/opt/petsc-3.7.2_debug_openmpi_2.x/lib/libpetsc.so.3.7(+0x11707f7)[0x7f80f3c697f7]

/opt/petsc-3.7.2_debug_openmpi_2.x/lib/libpetsc.so.3.7(PCReset+0x346)[0x7f80f3a796de]

/opt/petsc-3.7.2_debug_openmpi_2.x/lib/libpetsc.so.3.7(PCDestroy+0x5d1)[0x7f80f3a79fd9]

/opt/petsc-3.7.2_debug_openmpi_2.x/lib/libpetsc.so.3.7(KSPDestroy+0x7b6)[0x7f80f3d1a334]


a similar one:
*** Error in
`/pmi/cmpbib/compilation_BIB_dernier_ompi/COMPILE_AUTO/GIREF/bin/Test.ProbFluideIncompressible.dev':
free(): invalid pointer: 0x7f382a7c5bc0 ***
=== Backtrace: =
/lib64/libc.so.6(+0x7277f)[0x7f3829f1c77f]
/lib64/libc.so.6(+0x78026)[0x7f3829f22026]
/lib64/libc.so.6(+0x78d53)[0x7f3829f22d53]
/opt/openmpi-2.x_opt/lib/libopen-pal.so.20(opal_free+0x1f)[0x7f38296ffd60]

/opt/openmpi-2.x_opt/lib/openmpi/mca_pml_ob1.so(+0x16628)[0x7f381deab628]
/opt/openmpi-2.x_opt/lib/openmpi/mca_pml_ob1.so(+0x16c50)[0x7f381deabc50]
/opt/openmpi-2.x_opt/lib/libmpi.so.20(+0x9f9dd)[0x7f382a5089dd]
/opt/openmpi-2.x_opt/lib/libmpi.so.20(MPI_Request_free+0xf7)[0x7f382a508ad6]

/opt/petsc-3.7.2_debug_openmpi_2.x/lib/libpetsc.so.3.7(+0x4adc6d)[0x7f3831dacc6d]

/opt/petsc-3.7.2_debug_openmpi_2.x/lib/libpetsc.so.3.7(VecScatterDestroy+0x68d)[0x7f3831da7c45]

/opt/petsc-3.7.2_debug_openmpi_2.x/lib/libpetsc.so.3.7(+0x9f4755)[0x7f38322f3755]

/opt/petsc-3.7.2_debug_openmpi_2.x/lib/lib

Re: [OMPI devel] tcp btl rendezvous performance question

2016-07-26 Thread Pritchard Jr., Howard 
Hi Sreenidhi

Only partial resolution.  By pushing out the eager path to 4 MB we were able to 
get around 2GB/sec per socket connection with osu bw test.

The kernel is quite old though - 2.6.x - and being a summer student project 
with a focus on IB vs rout able ROCE we've moved on.

Howard


From: devel  on behalf of Sreenidhi Bharathkar 
Ramesh 
Sent: Tuesday, July 26, 2016 4:11:06 AM
To: Open MPI Developers
Subject: Re: [OMPI devel] tcp btl rendezvous performance question

hi Howard,

Was this issue resolved ?  If so, what is the solution ?
Please let me know.
Curious to know , since we are also experimenting with these limits.

Thanks,
- Sreenidhi.


On Tue, Jul 19, 2016 at 10:50 AM, Gilles Gouaillardet 
mailto:gil...@rist.or.jp>> wrote:

Howard,


did you bump both btl_tcp_rndv_eager_limit and btl_tcp_eager_limit ?

you might also need to bump btl_tcp_sndbuf, btl_tcp_rcvbuf and 
btl_tcp_max_send_size to get the max performance out of your 100Gb ethernet 
cards

last but not least, you might also need to bump btl_tcp_links to saturate your 
network (that is likely a good thing when running 1 task per node, but that can 
lead to decreased performance when running several tasks per node)

Cheers,


Gilles

On 7/19/2016 6:57 AM, Howard Pritchard wrote:
Hi Folks,

I have a cluster with some 100 Gb ethernet cards
installed.  What we are noticing if we force Open MPI 1.10.3
to go through the TCP BTL (rather than yalla)  is that
the performance of osu_bw once the TCP BTL switches
from eager to rendezvous (> 32KB)
falls off a cliff, going from about 1.6 GB/sec to 233 MB/sec
and stays that way out to 4 MB message lengths.

There's nothing wrong with the IP stack (iperf -P4 gives
63 Gb/sec).

So, my questions are

1) is this performance expected for the TCP BTL when in
rendezvous mode?
2) is there some way to get more like the single socket
performance obtained with iperf for large messages (~16 Gb/sec).

We tried adjusting the tcp_btl_rendezvous threshold but that doesn't
appear to actually be adjustable from the mpirun command line.

Thanks for any suggestions,

Howard






___
devel mailing list
de...@open-mpi.org
Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
Link to this post: 
http://www.open-mpi.org/community/lists/devel/2016/07/19237.php


___
devel mailing list
de...@open-mpi.org
Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
Link to this post: 
http://www.open-mpi.org/community/lists/devel/2016/07/19240.php

Re: [OMPI devel] PGI built Open MPI vs GNU built slurm

2016-07-26 Thread Paul Hargrove 
Gilles,

With the additional information you provided about "dependency_libs", I
agree that that either of the fixes you propose sound safe.

-Paul

On Mon, Jul 25, 2016 at 6:26 PM, Gilles Gouaillardet 
wrote:

> Paul,
>
> in my environment, libslurm.la contains
>
> # Linker flags that can not go in dependency_libs.
> inherited_linker_flags=' -pthread'
>
> # Libraries that this one depends upon.
> dependency_libs=' -ldl -lpthread'
>
>
> so bottom line, it invokes the compiler with both -pthread and -lpthread
>
>
> iirc, -pthread does two things :
>
> - invoke the compiler with -D_REENTRANT (so it uses the thread-safe errno
> and so on)
>
> - invoke the linker with -lpthread
>
> OpenMPI has its own way to pass -D_REENTRANT or similar anyway, and
> libslurm.la is used only for for linking.
>
> since -lpthread is pulled anyway from libslurm.la (or it was already set
> by OpenMPI), then yes, discarding -pthread should do the trick.
>
>
> Cheers,
>
>
> Gilles
>
> On 7/26/2016 10:11 AM, Paul Hargrove wrote:
>
> Gilles,
>
> My initial thought is that libslurm probably does require linking
> libpthread, either for for linking pthread_* symbols, or for proper
> *operation* (such as thread-safe versions of functions which override weak
> definitions in libc).
>
> If so, then neither omitting "-pthread" nor telling pgcc not to complain
> about "-pthread" is going to be a good solution.
> Instead the "-pthread" needs to be replaced by "-lpthread", or similar.
>
> -Paul
>
> On Mon, Jul 25, 2016 at 6:03 PM, Gilles Gouaillardet 
> wrote:
>
>> Folks,
>>
>>
>> This is a followup of a thread that initially started at
>> http://www.open-mpi.org/community/lists/users/2016/07/29635.php
>>
>>
>> The user is trying to build Open MPI with PGI compiler and
>> libslurm.la/libpmi.la support, and slurm was built with gcc compiler.
>>
>>
>> At first, it fails because the "-pthread" flag is pulled from
>> libslurm.la/libpmi.la, but this flag is not supported by PGI compilers.
>>
>> A workaround is to pass the -noswitcherror flag to the PGI compiler (so
>> the -pthread flag is discarded and a warning message is issued, but PGI
>> compiler does not fail). Unfortunatly, that does not work because libtool
>> does does not pass this flag to the PGI compiler.
>>
>>
>> Of course, one option is to tell the user to rebuild slurm with PGI, so
>> libslurm.la/libpmi.la do not have the "-pthread" flag.
>>
>> A nicer though arguable option is to hack libtool to silently drop the
>> "-pthread" flag with PGI compiler is used (i made a proof of concept, and
>> this is a two lines patch).
>>
>> An other cleaner option is to hack libtool so it pass -noswitcherror to
>> PGI compiler, but i do not know how to achieve this.
>>
>>
>> Any thoughts ?
>>
>>
>> Cheers
>>
>> ___
>> devel mailing list
>> de...@open-mpi.org
>> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
>> Link to this post:
>> http://www.open-mpi.org/community/lists/devel/2016/07/19278.php
>>
>
>
>
> --
> Paul H. Hargrove  phhargr...@lbl.gov
> Computer Languages & Systems Software (CLaSS) Group
> Computer Science Department   Tel: +1-510-495-2352
> Lawrence Berkeley National Laboratory Fax: +1-510-486-6900
>
>
> ___
> devel mailing listde...@open-mpi.org
> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
>
> Link to this post: 
> http://www.open-mpi.org/community/lists/devel/2016/07/19279.php
>
>
>
> ___
> devel mailing list
> de...@open-mpi.org
> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2016/07/19280.php
>



-- 
Paul H. Hargrove  phhargr...@lbl.gov
Computer Languages & Systems Software (CLaSS) Group
Computer Science Department   Tel: +1-510-495-2352
Lawrence Berkeley National Laboratory Fax: +1-510-486-6900

[OMPI devel] sm BTL performace of the openmpi-2.0.0

2016-07-26 Thread tmishima 

Hi folks,

I saw a performance degradation of openmpi-2.0.0 when I ran our application
on a node (12cores). So I did 4 tests using osu_bw as below:

1: mpirun –np 2 osu_bw  bad(30% of test2)
2: mpirun –np 2 –mca btl self,sm osu_bw good(same as openmpi1.10.3)
3: mpirun –np 2 –mca btl self,sm,openib osu_bw  bad(30% of test2)
4: mpirun –np 2 –mca btl self,openib osu_bw bad(30% of test2)

I  guess openib BTL was used in the test 1 and 3, because these results are
almost  same  as  test  4. I believe that sm BTL should be used even in the
test 1 and 3, because its priority is higher than openib. Unfortunately, at
the  moment,  I couldn’t figure out the root cause. So please someone would
take care of it.

Regards,
Tetsuya Mishima

P.S. Here I attached these test results.

[mishima@manage   OMB-3.1.1-openmpi2.0.0]$   mpirun  -np  2  -bind-to  core
-report-bindings osu_bw
[manage.cluster:13389]  MCW  rank  0  bound  to  socket  0[core  0[hwt 0]]:
[B/././././.][./././././.]
[manage.cluster:13389]  MCW  rank  1  bound  to  socket  0[core  1[hwt 0]]:
[./B/./././.][./././././.]
# OSU MPI Bandwidth Test v3.1.1
# SizeBandwidth (MB/s)
1 1.49
2 3.04
4 6.13
812.23
16   25.01
32   49.96
64   87.07
128 138.87
256 245.97
512 423.30
1024865.85
2048   1279.63
4096264.79
8192473.92
16384   739.27
32768  1030.49
65536  1190.21
131072 1270.77
262144 1238.74
524288 1245.97
10485761260.09
20971521274.53
41943041285.07
[mishima@manage  OMB-3.1.1-openmpi2.0.0]$  mpirun  -np  2  -mca btl self,sm
-bind-to core -report-bindings osu_bw
[manage.cluster:13448]  MCW  rank  0  bound  to  socket  0[core  0[hwt 0]]:
[B/././././.][./././././.]
[manage.cluster:13448]  MCW  rank  1  bound  to  socket  0[core  1[hwt 0]]:
[./B/./././.][./././././.]
# OSU MPI Bandwidth Test v3.1.1
# SizeBandwidth (MB/s)
1 0.51
2 1.01
4 2.03
8 4.08
167.92
32   16.16
64   32.53
128  64.30
256 128.19
512 256.48
1024468.62
2048785.29
4096854.78
8192   1404.51
16384  2249.20
32768  3136.40
65536  3495.84
131072 3436.69
262144 3392.11
524288 3400.07
10485763460.60
20971523488.09
41943043498.45
[mishima@manageOMB-3.1.1-openmpi2.0.0]$   mpirun   -np   2   -mca   btl
self,sm,openib -bind-to core -report-bindings osu_bw
[manage.cluster:13462]  MCW  rank  0  bound  to  socket  0[core  0[hwt 0]]:
[B/././././.][./././././.]
[manage.cluster:13462]  MCW  rank  1  bound  to  socket  0[core  1[hwt 0]]:
[./B/./././.][./././././.]
# OSU MPI Bandwidth Test v3.1.1
# SizeBandwidth (MB/s)
1 0.54
2 1.09
4 2.18
8 4.37
168.75
32   17.37
64   34.67
128  66.66
256 132.55
512 261.52
1024489.51
2048818.38
4096290.48
8192511.64
16384   765.24
32768  1043.28
65536  1180.48
131072 1261.41
262144 1232.86
524288 1245.70
10485761245.69
20971521268.67
41943041281.33
[mishima@manage  OMB-3.1.1-openmpi2.0.0]$ mpirun -np 2 -mca btl self,openib
-bind-to core -report-bindings osu_bw
[manage.cluster:13521]  MCW  rank  0  bound  to  socket  0[core  0[hwt 0]]:
[B/././././.][./././././.]
[manage.cluster:13521]  MCW  rank  1  bound  to  socket  0[core  1[hwt 0]]:
[./B/./././.][./././././.]
# OSU MPI Bandwidth Test v3.1.1
# SizeBandwidth (MB/s)
1 0.54
2 1.08
4 2.16
8 4.34
168.64
32   17.25
64   34.30
128  66.13
256 129.99
512 242.26
1024429.24
2048556.00
4096706.80
8192874.35
16384   762.60
32768  1039.61
65536

Re: [OMPI devel] sm BTL performace of the openmpi-2.0.0

2016-07-26 Thread Gilles Gouaillardet 

Hi,


can you please run again with

--mca pml ob1


if Open MPI was built with mxm support, pml/cm and mtl/mxm are used 
instead of pml/ob1 and btl/openib



Cheers,


Gilles


On 7/27/2016 8:56 AM, tmish...@jcity.maeda.co.jp wrote:

Hi folks,

I saw a performance degradation of openmpi-2.0.0 when I ran our application
on a node (12cores). So I did 4 tests using osu_bw as below:

1: mpirun –np 2 osu_bw  bad(30% of test2)
2: mpirun –np 2 –mca btl self,sm osu_bw good(same as openmpi1.10.3)
3: mpirun –np 2 –mca btl self,sm,openib osu_bw  bad(30% of test2)
4: mpirun –np 2 –mca btl self,openib osu_bw bad(30% of test2)

I  guess openib BTL was used in the test 1 and 3, because these results are
almost  same  as  test  4. I believe that sm BTL should be used even in the
test 1 and 3, because its priority is higher than openib. Unfortunately, at
the  moment,  I couldn’t figure out the root cause. So please someone would
take care of it.

Regards,
Tetsuya Mishima

P.S. Here I attached these test results.

[mishima@manage   OMB-3.1.1-openmpi2.0.0]$   mpirun  -np  2  -bind-to  core
-report-bindings osu_bw
[manage.cluster:13389]  MCW  rank  0  bound  to  socket  0[core  0[hwt 0]]:
[B/././././.][./././././.]
[manage.cluster:13389]  MCW  rank  1  bound  to  socket  0[core  1[hwt 0]]:
[./B/./././.][./././././.]
# OSU MPI Bandwidth Test v3.1.1
# SizeBandwidth (MB/s)
1 1.49
2 3.04
4 6.13
812.23
16   25.01
32   49.96
64   87.07
128 138.87
256 245.97
512 423.30
1024865.85
2048   1279.63
4096264.79
8192473.92
16384   739.27
32768  1030.49
65536  1190.21
131072 1270.77
262144 1238.74
524288 1245.97
10485761260.09
20971521274.53
41943041285.07
[mishima@manage  OMB-3.1.1-openmpi2.0.0]$  mpirun  -np  2  -mca btl self,sm
-bind-to core -report-bindings osu_bw
[manage.cluster:13448]  MCW  rank  0  bound  to  socket  0[core  0[hwt 0]]:
[B/././././.][./././././.]
[manage.cluster:13448]  MCW  rank  1  bound  to  socket  0[core  1[hwt 0]]:
[./B/./././.][./././././.]
# OSU MPI Bandwidth Test v3.1.1
# SizeBandwidth (MB/s)
1 0.51
2 1.01
4 2.03
8 4.08
167.92
32   16.16
64   32.53
128  64.30
256 128.19
512 256.48
1024468.62
2048785.29
4096854.78
8192   1404.51
16384  2249.20
32768  3136.40
65536  3495.84
131072 3436.69
262144 3392.11
524288 3400.07
10485763460.60
20971523488.09
41943043498.45
[mishima@manageOMB-3.1.1-openmpi2.0.0]$   mpirun   -np   2   -mca   btl
self,sm,openib -bind-to core -report-bindings osu_bw
[manage.cluster:13462]  MCW  rank  0  bound  to  socket  0[core  0[hwt 0]]:
[B/././././.][./././././.]
[manage.cluster:13462]  MCW  rank  1  bound  to  socket  0[core  1[hwt 0]]:
[./B/./././.][./././././.]
# OSU MPI Bandwidth Test v3.1.1
# SizeBandwidth (MB/s)
1 0.54
2 1.09
4 2.18
8 4.37
168.75
32   17.37
64   34.67
128  66.66
256 132.55
512 261.52
1024489.51
2048818.38
4096290.48
8192511.64
16384   765.24
32768  1043.28
65536  1180.48
131072 1261.41
262144 1232.86
524288 1245.70
10485761245.69
20971521268.67
41943041281.33
[mishima@manage  OMB-3.1.1-openmpi2.0.0]$ mpirun -np 2 -mca btl self,openib
-bind-to core -report-bindings osu_bw
[manage.cluster:13521]  MCW  rank  0  bound  to  socket  0[core  0[hwt 0]]:
[B/././././.][./././././.]
[manage.cluster:13521]  MCW  rank  1  bound  to  socket  0[core  1[hwt 0]]:
[./B/./././.][./././././.]
# OSU MPI Bandwidth Test v3.1.1
# SizeBandwidth (MB/s)
1 0.54
2 1.08
4 2.16
8 4.34
168.64
32   17.25
64   34.30
128  66.13
256 129.9

Re: [OMPI devel] sm BTL performace of the openmpi-2.0.0

2016-07-26 Thread tmishima 
Hi Gilles,

Thanks. I ran again with --mca pml ob1 but I've got the same results as
below:

[mishima@manage OMB-3.1.1-openmpi2.0.0]$ mpirun -np 2 -mca pml ob1 -bind-to
core -report-bindings osu_bw
[manage.cluster:18142] MCW rank 0 bound to socket 0[core 0[hwt 0]]:
[B/././././.][./././././.]
[manage.cluster:18142] MCW rank 1 bound to socket 0[core 1[hwt 0]]:
[./B/./././.][./././././.]
# OSU MPI Bandwidth Test v3.1.1
# SizeBandwidth (MB/s)
1 1.48
2 3.07
4 6.26
812.53
16   24.33
32   49.03
64   83.46
128 132.60
256 234.96
512 420.86
1024842.37
2048   1231.65
4096264.67
8192472.16
16384   740.42
32768  1030.39
65536  1191.16
131072 1269.45
262144 1238.33
524288 1247.97
10485761257.96
20971521274.74
41943041280.94
[mishima@manage OMB-3.1.1-openmpi2.0.0]$ mpirun -np 2 -mca pml ob1 -mca btl
self,sm -bind-to core -report-bindings osu_b
w
[manage.cluster:18204] MCW rank 0 bound to socket 0[core 0[hwt 0]]:
[B/././././.][./././././.]
[manage.cluster:18204] MCW rank 1 bound to socket 0[core 1[hwt 0]]:
[./B/./././.][./././././.]
# OSU MPI Bandwidth Test v3.1.1
# SizeBandwidth (MB/s)
1 0.52
2 1.05
4 2.08
8 4.18
168.21
32   16.65
64   32.60
128  66.70
256 132.45
512 269.27
1024504.63
2048819.76
4096874.54
8192   1447.11
16384  2263.28
32768  3236.85
65536  3567.34
131072 3555.17
262144 3455.76
524288 3441.80
10485763505.30
20971523534.01
41943043546.94
[mishima@manage OMB-3.1.1-openmpi2.0.0]$ mpirun -np 2 -mca pml ob1 -mca btl
self,sm,openib -bind-to core -report-binding
s osu_bw
[manage.cluster:18218] MCW rank 0 bound to socket 0[core 0[hwt 0]]:
[B/././././.][./././././.]
[manage.cluster:18218] MCW rank 1 bound to socket 0[core 1[hwt 0]]:
[./B/./././.][./././././.]
# OSU MPI Bandwidth Test v3.1.1
# SizeBandwidth (MB/s)
1 0.51
2 1.03
4 2.05
8 4.07
168.14
32   16.32
64   32.98
128  63.70
256 126.66
512 252.61
1024480.22
2048810.54
4096290.61
8192512.49
16384   764.60
32768  1036.81
65536  1182.81
131072 1264.48
262144 1235.82
524288 1246.70
10485761254.66
20971521274.64
41943041280.65
[mishima@manage OMB-3.1.1-openmpi2.0.0]$ mpirun -np 2 -mca pml ob1 -mca btl
self,openib -bind-to core -report-bindings o
su_bw
[manage.cluster:18276] MCW rank 0 bound to socket 0[core 0[hwt 0]]:
[B/././././.][./././././.]
[manage.cluster:18276] MCW rank 1 bound to socket 0[core 1[hwt 0]]:
[./B/./././.][./././././.]
# OSU MPI Bandwidth Test v3.1.1
# SizeBandwidth (MB/s)
1 0.54
2 1.08
4 2.18
8 4.33
168.69
32   17.39
64   34.34
128  66.28
256 130.36
512 241.81
1024429.86
2048553.44
4096707.14
8192879.60
16384   763.02
32768  1042.89
65536  1185.45
131072 1267.56
262144 1227.41
524288 1244.61
10485761255.66
20971521273.55
41943041281.05


2016/07/27 9:02:49、"devel"さんは「Re: [OMPI devel] sm BTL performace of
the openmpi-2.0.0」で書きました
> Hi,
>
>
> can you please run again with
>
> --mca pml ob1
>
>
> if Open MPI was built with mxm support, pml/cm and mtl/mxm are used
> instead of pml/ob1 and btl/openib
>
>
> Cheers,
>
>
> Gilles
>
>
> On 7/27/2016 8:56 AM, tmish...@jcity.maeda.co.jp wrote:
> > Hi folks,
> >
> > I saw a performance degradation of openmpi-2.0.0 when I ran our
application
> > on a node (12cores). So I did 4 tests using osu_bw as below:
> >
> > 1: mpirun –np 2 osu_bw

Re: [OMPI devel] sm BTL performace of the openmpi-2.0.0

2016-07-26 Thread Nathan Hjelm 
sm is deprecated in 2.0.0 and will likely be removed in favor of vader in 2.1.0.

This issue is probably this known issue: 
https://github.com/open-mpi/ompi-release/pull/1250

Please apply those commits and see if it fixes the issue for you.

-Nathan

> On Jul 26, 2016, at 6:17 PM, tmish...@jcity.maeda.co.jp wrote:
> 
> Hi Gilles,
> 
> Thanks. I ran again with --mca pml ob1 but I've got the same results as
> below:
> 
> [mishima@manage OMB-3.1.1-openmpi2.0.0]$ mpirun -np 2 -mca pml ob1 -bind-to
> core -report-bindings osu_bw
> [manage.cluster:18142] MCW rank 0 bound to socket 0[core 0[hwt 0]]:
> [B/././././.][./././././.]
> [manage.cluster:18142] MCW rank 1 bound to socket 0[core 1[hwt 0]]:
> [./B/./././.][./././././.]
> # OSU MPI Bandwidth Test v3.1.1
> # SizeBandwidth (MB/s)
> 1 1.48
> 2 3.07
> 4 6.26
> 812.53
> 16   24.33
> 32   49.03
> 64   83.46
> 128 132.60
> 256 234.96
> 512 420.86
> 1024842.37
> 2048   1231.65
> 4096264.67
> 8192472.16
> 16384   740.42
> 32768  1030.39
> 65536  1191.16
> 131072 1269.45
> 262144 1238.33
> 524288 1247.97
> 10485761257.96
> 20971521274.74
> 41943041280.94
> [mishima@manage OMB-3.1.1-openmpi2.0.0]$ mpirun -np 2 -mca pml ob1 -mca btl
> self,sm -bind-to core -report-bindings osu_b
> w
> [manage.cluster:18204] MCW rank 0 bound to socket 0[core 0[hwt 0]]:
> [B/././././.][./././././.]
> [manage.cluster:18204] MCW rank 1 bound to socket 0[core 1[hwt 0]]:
> [./B/./././.][./././././.]
> # OSU MPI Bandwidth Test v3.1.1
> # SizeBandwidth (MB/s)
> 1 0.52
> 2 1.05
> 4 2.08
> 8 4.18
> 168.21
> 32   16.65
> 64   32.60
> 128  66.70
> 256 132.45
> 512 269.27
> 1024504.63
> 2048819.76
> 4096874.54
> 8192   1447.11
> 16384  2263.28
> 32768  3236.85
> 65536  3567.34
> 131072 3555.17
> 262144 3455.76
> 524288 3441.80
> 10485763505.30
> 20971523534.01
> 41943043546.94
> [mishima@manage OMB-3.1.1-openmpi2.0.0]$ mpirun -np 2 -mca pml ob1 -mca btl
> self,sm,openib -bind-to core -report-binding
> s osu_bw
> [manage.cluster:18218] MCW rank 0 bound to socket 0[core 0[hwt 0]]:
> [B/././././.][./././././.]
> [manage.cluster:18218] MCW rank 1 bound to socket 0[core 1[hwt 0]]:
> [./B/./././.][./././././.]
> # OSU MPI Bandwidth Test v3.1.1
> # SizeBandwidth (MB/s)
> 1 0.51
> 2 1.03
> 4 2.05
> 8 4.07
> 168.14
> 32   16.32
> 64   32.98
> 128  63.70
> 256 126.66
> 512 252.61
> 1024480.22
> 2048810.54
> 4096290.61
> 8192512.49
> 16384   764.60
> 32768  1036.81
> 65536  1182.81
> 131072 1264.48
> 262144 1235.82
> 524288 1246.70
> 10485761254.66
> 20971521274.64
> 41943041280.65
> [mishima@manage OMB-3.1.1-openmpi2.0.0]$ mpirun -np 2 -mca pml ob1 -mca btl
> self,openib -bind-to core -report-bindings o
> su_bw
> [manage.cluster:18276] MCW rank 0 bound to socket 0[core 0[hwt 0]]:
> [B/././././.][./././././.]
> [manage.cluster:18276] MCW rank 1 bound to socket 0[core 1[hwt 0]]:
> [./B/./././.][./././././.]
> # OSU MPI Bandwidth Test v3.1.1
> # SizeBandwidth (MB/s)
> 1 0.54
> 2 1.08
> 4 2.18
> 8 4.33
> 168.69
> 32   17.39
> 64   34.34
> 128  66.28
> 256 130.36
> 512 241.81
> 1024429.86
> 2048553.44
> 4096707.14
> 8192879.60
> 16384   763.02
> 32768  1042.89
> 65536  1185.45
> 131072 1267.56
> 262144 1227.41
> 524288 1244.61
> 10485761255.66
> 20971521273.55
> 41943

Re: [OMPI devel] sm BTL performace of the openmpi-2.0.0

2016-07-26 Thread Gilles Gouaillardet 
Also, btl/vader has a higher exclusivity than btl/sm, so if you do not 
manually specify any btl, vader should be used.



you can run with

--mca btl_base_verbose 10

to confirm which btl is used


Cheers,


Gilles


On 7/27/2016 9:20 AM, Nathan Hjelm wrote:

sm is deprecated in 2.0.0 and will likely be removed in favor of vader in 2.1.0.

This issue is probably this known issue: 
https://github.com/open-mpi/ompi-release/pull/1250

Please apply those commits and see if it fixes the issue for you.

-Nathan


On Jul 26, 2016, at 6:17 PM, tmish...@jcity.maeda.co.jp wrote:

Hi Gilles,

Thanks. I ran again with --mca pml ob1 but I've got the same results as
below:

[mishima@manage OMB-3.1.1-openmpi2.0.0]$ mpirun -np 2 -mca pml ob1 -bind-to
core -report-bindings osu_bw
[manage.cluster:18142] MCW rank 0 bound to socket 0[core 0[hwt 0]]:
[B/././././.][./././././.]
[manage.cluster:18142] MCW rank 1 bound to socket 0[core 1[hwt 0]]:
[./B/./././.][./././././.]
# OSU MPI Bandwidth Test v3.1.1
# SizeBandwidth (MB/s)
1 1.48
2 3.07
4 6.26
812.53
16   24.33
32   49.03
64   83.46
128 132.60
256 234.96
512 420.86
1024842.37
2048   1231.65
4096264.67
8192472.16
16384   740.42
32768  1030.39
65536  1191.16
131072 1269.45
262144 1238.33
524288 1247.97
10485761257.96
20971521274.74
41943041280.94
[mishima@manage OMB-3.1.1-openmpi2.0.0]$ mpirun -np 2 -mca pml ob1 -mca btl
self,sm -bind-to core -report-bindings osu_b
w
[manage.cluster:18204] MCW rank 0 bound to socket 0[core 0[hwt 0]]:
[B/././././.][./././././.]
[manage.cluster:18204] MCW rank 1 bound to socket 0[core 1[hwt 0]]:
[./B/./././.][./././././.]
# OSU MPI Bandwidth Test v3.1.1
# SizeBandwidth (MB/s)
1 0.52
2 1.05
4 2.08
8 4.18
168.21
32   16.65
64   32.60
128  66.70
256 132.45
512 269.27
1024504.63
2048819.76
4096874.54
8192   1447.11
16384  2263.28
32768  3236.85
65536  3567.34
131072 3555.17
262144 3455.76
524288 3441.80
10485763505.30
20971523534.01
41943043546.94
[mishima@manage OMB-3.1.1-openmpi2.0.0]$ mpirun -np 2 -mca pml ob1 -mca btl
self,sm,openib -bind-to core -report-binding
s osu_bw
[manage.cluster:18218] MCW rank 0 bound to socket 0[core 0[hwt 0]]:
[B/././././.][./././././.]
[manage.cluster:18218] MCW rank 1 bound to socket 0[core 1[hwt 0]]:
[./B/./././.][./././././.]
# OSU MPI Bandwidth Test v3.1.1
# SizeBandwidth (MB/s)
1 0.51
2 1.03
4 2.05
8 4.07
168.14
32   16.32
64   32.98
128  63.70
256 126.66
512 252.61
1024480.22
2048810.54
4096290.61
8192512.49
16384   764.60
32768  1036.81
65536  1182.81
131072 1264.48
262144 1235.82
524288 1246.70
10485761254.66
20971521274.64
41943041280.65
[mishima@manage OMB-3.1.1-openmpi2.0.0]$ mpirun -np 2 -mca pml ob1 -mca btl
self,openib -bind-to core -report-bindings o
su_bw
[manage.cluster:18276] MCW rank 0 bound to socket 0[core 0[hwt 0]]:
[B/././././.][./././././.]
[manage.cluster:18276] MCW rank 1 bound to socket 0[core 1[hwt 0]]:
[./B/./././.][./././././.]
# OSU MPI Bandwidth Test v3.1.1
# SizeBandwidth (MB/s)
1 0.54
2 1.08
4 2.18
8 4.33
168.69
32   17.39
64   34.34
128  66.28
256 130.36
512 241.81
1024429.86
2048553.44
4096707.14
8192879.60
16384   763.02
32768  1042.89
65536  1185.45
131072 1267.56
262144 1227.41
524288 1244.61
10485761255.66
20971521273.55
4194304

Re: [OMPI devel] sm BTL performace of the openmpi-2.0.0

2016-07-26 Thread tmishima 
Hi,

Thanks. I will try it and report later.

Tetsuya Mishima


2016/07/27 9:20:28、"devel"さんは「Re: [OMPI devel] sm BTL performace of
the openmpi-2.0.0」で書きました
> sm is deprecated in 2.0.0 and will likely be removed in favor of vader in
2.1.0.
>
> This issue is probably this known issue:
https://github.com/open-mpi/ompi-release/pull/1250
>
> Please apply those commits and see if it fixes the issue for you.
>
> -Nathan
>
> > On Jul 26, 2016, at 6:17 PM, tmish...@jcity.maeda.co.jp wrote:
> >
> > Hi Gilles,
> >
> > Thanks. I ran again with --mca pml ob1 but I've got the same results as
> > below:
> >
> > [mishima@manage OMB-3.1.1-openmpi2.0.0]$ mpirun -np 2 -mca pml ob1
-bind-to
> > core -report-bindings osu_bw
> > [manage.cluster:18142] MCW rank 0 bound to socket 0[core 0[hwt 0]]:
> > [B/././././.][./././././.]
> > [manage.cluster:18142] MCW rank 1 bound to socket 0[core 1[hwt 0]]:
> > [./B/./././.][./././././.]
> > # OSU MPI Bandwidth Test v3.1.1
> > # SizeBandwidth (MB/s)
> > 1 1.48
> > 2 3.07
> > 4 6.26
> > 812.53
> > 16   24.33
> > 32   49.03
> > 64   83.46
> > 128 132.60
> > 256 234.96
> > 512 420.86
> > 1024842.37
> > 2048   1231.65
> > 4096264.67
> > 8192472.16
> > 16384   740.42
> > 32768  1030.39
> > 65536  1191.16
> > 131072 1269.45
> > 262144 1238.33
> > 524288 1247.97
> > 10485761257.96
> > 20971521274.74
> > 41943041280.94
> > [mishima@manage OMB-3.1.1-openmpi2.0.0]$ mpirun -np 2 -mca pml ob1 -mca
btl
> > self,sm -bind-to core -report-bindings osu_b
> > w
> > [manage.cluster:18204] MCW rank 0 bound to socket 0[core 0[hwt 0]]:
> > [B/././././.][./././././.]
> > [manage.cluster:18204] MCW rank 1 bound to socket 0[core 1[hwt 0]]:
> > [./B/./././.][./././././.]
> > # OSU MPI Bandwidth Test v3.1.1
> > # SizeBandwidth (MB/s)
> > 1 0.52
> > 2 1.05
> > 4 2.08
> > 8 4.18
> > 168.21
> > 32   16.65
> > 64   32.60
> > 128  66.70
> > 256 132.45
> > 512 269.27
> > 1024504.63
> > 2048819.76
> > 4096874.54
> > 8192   1447.11
> > 16384  2263.28
> > 32768  3236.85
> > 65536  3567.34
> > 131072 3555.17
> > 262144 3455.76
> > 524288 3441.80
> > 10485763505.30
> > 20971523534.01
> > 41943043546.94
> > [mishima@manage OMB-3.1.1-openmpi2.0.0]$ mpirun -np 2 -mca pml ob1 -mca
btl
> > self,sm,openib -bind-to core -report-binding
> > s osu_bw
> > [manage.cluster:18218] MCW rank 0 bound to socket 0[core 0[hwt 0]]:
> > [B/././././.][./././././.]
> > [manage.cluster:18218] MCW rank 1 bound to socket 0[core 1[hwt 0]]:
> > [./B/./././.][./././././.]
> > # OSU MPI Bandwidth Test v3.1.1
> > # SizeBandwidth (MB/s)
> > 1 0.51
> > 2 1.03
> > 4 2.05
> > 8 4.07
> > 168.14
> > 32   16.32
> > 64   32.98
> > 128  63.70
> > 256 126.66
> > 512 252.61
> > 1024480.22
> > 2048810.54
> > 4096290.61
> > 8192512.49
> > 16384   764.60
> > 32768  1036.81
> > 65536  1182.81
> > 131072 1264.48
> > 262144 1235.82
> > 524288 1246.70
> > 10485761254.66
> > 20971521274.64
> > 41943041280.65
> > [mishima@manage OMB-3.1.1-openmpi2.0.0]$ mpirun -np 2 -mca pml ob1 -mca
btl
> > self,openib -bind-to core -report-bindings o
> > su_bw
> > [manage.cluster:18276] MCW rank 0 bound to socket 0[core 0[hwt 0]]:
> > [B/././././.][./././././.]
> > [manage.cluster:18276] MCW rank 1 bound to socket 0[core 1[hwt 0]]:
> > [./B/./././.][./././././.]
> > # OSU MPI Bandwidth Test v3.1.1
> > # SizeBandwidth (MB/s)
> > 1 0.54
> > 2 1.08
> > 4 2.18
> > 8 4.33
> > 168.69
> > 32   17.39
> > 64   34.34
> > 128  66.28
> > 256 130.36
> > 512 241

Re: [OMPI devel] sm BTL performace of the openmpi-2.0.0

2016-07-26 Thread tmishima 
Hi Gilles,

I confirmed the vader is used when I don't specify any BTL as you pointed
out!

Regards,
Tetsuya Mishima

[mishima@manage OMB-3.1.1-openmpi2.0.0]$ mpirun -np 2 --mca
btl_base_verbose 10 -bind-to core -report-bindings osu_bw
[manage.cluster:20006] MCW rank 0 bound to socket 0[core 0[hwt 0]]:
[B/././././.][./././././.]
[manage.cluster:20006] MCW rank 1 bound to socket 0[core 1[hwt 0]]:
[./B/./././.][./././././.]
[manage.cluster:20011] mca: base: components_register: registering
framework btl components
[manage.cluster:20011] mca: base: components_register: found loaded
component self
[manage.cluster:20011] mca: base: components_register: component self
register function successful
[manage.cluster:20011] mca: base: components_register: found loaded
component vader
[manage.cluster:20011] mca: base: components_register: component vader
register function successful
[manage.cluster:20011] mca: base: components_register: found loaded
component tcp
[manage.cluster:20011] mca: base: components_register: component tcp
register function successful
[manage.cluster:20011] mca: base: components_register: found loaded
component sm
[manage.cluster:20011] mca: base: components_register: component sm
register function successful
[manage.cluster:20011] mca: base: components_register: found loaded
component openib
[manage.cluster:20011] mca: base: components_register: component openib
register function successful
[manage.cluster:20011] mca: base: components_open: opening btl components
[manage.cluster:20011] mca: base: components_open: found loaded component
self
[manage.cluster:20011] mca: base: components_open: component self open
function successful
[manage.cluster:20011] mca: base: components_open: found loaded component
vader
[manage.cluster:20011] mca: base: components_open: component vader open
function successful
[manage.cluster:20011] mca: base: components_open: found loaded component
tcp
[manage.cluster:20011] mca: base: components_open: component tcp open
function successful
[manage.cluster:20011] mca: base: components_open: found loaded component
sm
[manage.cluster:20011] mca: base: components_open: component sm open
function successful
[manage.cluster:20011] mca: base: components_open: found loaded component
openib
[manage.cluster:20011] mca: base: components_open: component openib open
function successful
[manage.cluster:20011] select: initializing btl component self
[manage.cluster:20011] select: init of component self returned success
[manage.cluster:20011] select: initializing btl component vader
[manage.cluster:20011] select: init of component vader returned success
[manage.cluster:20011] select: initializing btl component tcp
[manage.cluster:20011] select: init of component tcp returned success
[manage.cluster:20011] select: initializing btl component sm
[manage.cluster:20011] select: init of component sm returned success
[manage.cluster:20011] select: initializing btl component openib
[manage.cluster:20011] Checking distance from this process to device=mthca0
[manage.cluster:20011] hwloc_distances->nbobjs=2
[manage.cluster:20011] hwloc_distances->latency[0]=1.00
[manage.cluster:20011] hwloc_distances->latency[1]=1.60
[manage.cluster:20011] hwloc_distances->latency[2]=1.60
[manage.cluster:20011] hwloc_distances->latency[3]=1.00
[manage.cluster:20011] ibv_obj->type set to NULL
[manage.cluster:20011] Process is bound: distance to device is 0.00
[manage.cluster:20012] mca: base: components_register: registering
framework btl components
[manage.cluster:20012] mca: base: components_register: found loaded
component self
[manage.cluster:20012] mca: base: components_register: component self
register function successful
[manage.cluster:20012] mca: base: components_register: found loaded
component vader
[manage.cluster:20012] mca: base: components_register: component vader
register function successful
[manage.cluster:20012] mca: base: components_register: found loaded
component tcp
[manage.cluster:20012] mca: base: components_register: component tcp
register function successful
[manage.cluster:20012] mca: base: components_register: found loaded
component sm
[manage.cluster:20012] mca: base: components_register: component sm
register function successful
[manage.cluster:20012] mca: base: components_register: found loaded
component openib
[manage.cluster:20012] mca: base: components_register: component openib
register function successful
[manage.cluster:20012] mca: base: components_open: opening btl components
[manage.cluster:20012] mca: base: components_open: found loaded component
self
[manage.cluster:20012] mca: base: components_open: component self open
function successful
[manage.cluster:20012] mca: base: components_open: found loaded component
vader
[manage.cluster:20012] mca: base: components_open: component vader open
function successful
[manage.cluster:20012] mca: base: components_open: found loaded component
tcp
[manage.cluster:20012] mca: base: components_open: component tcp ope