Hi Tobias,
Thanks for the reply. I tried both "export OMPI_MCA_mpi_leave_pinned=0" and
"mpirun -mca mpi_leave_pinned 0" but still got the same behavior. Our OpenMPI
version is 2.0.1. Repo version is v2.0.0-257-gee86e07. We have Intel Qlogic and
OPA networks on the same cluster.
Below is our configuration flags:
./configure --prefix=$PREFIX \
--with-hwloc=internal \
--enable-mpirun-prefix-by-default \
--with-slurm \
--with-verbs \
--with-psm \
--with-psm2 \
--disable-openib-connectx-xrc \
--with-knem=/opt/knem-1.1.2.90mlnx1 \
--with-cma
So the question remains why OpenMPI choose openib over self,sm for single node
jobs? Isn't there a mechanism to differentiate btl networks for
single/multi-node jobs?
Thanks,
Jingchao
________________________________
From: users <users-boun...@lists.open-mpi.org> on behalf of Tobias Kloeffel
<tobias.kloef...@fau.de>
Sent: Tuesday, February 7, 2017 2:54:46 AM
To: Open MPI Users
Subject: Re: [OMPI users] openmpi single node jobs using btl openib
Hello Jingchao,
try to use -mca mpi_leave_pinned 0, also for multinode jobs.
kind regards,
Tobias Klöffel
On 02/06/2017 09:38 PM, Jingchao Zhang wrote:
Hi,
We recently noticed openmpi is using btl openib over self,sm for single node
jobs, which has caused performance degradation for some applications, e.g.
'cp2k'. For opempi version 2.0.1, our test shows single node 'cp2k' job using
openib is ~25% slower than using self,sm. We advise users do '--mca
btl_base_exclude openib' as a temporary fix. I need to point out that not all
applications are affected by this feature. Many of them have the same
single-node performance with/without openib. Why doesn't openmpi use self,sm by
default for single node jobs? Is this the intended behavior?
Thanks,
Jingchao
_______________________________________________
users mailing list
users@lists.open-mpi.org<mailto:users@lists.open-mpi.org>
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
--
M.Sc. Tobias Klöffel
=======================================================
Interdisciplinary Center for Molecular Materials (ICMM)
and Computer-Chemistry-Center (CCC)
Department Chemie und Pharmazie
Friedrich-Alexander-Universität Erlangen-Nürnberg
Nägelsbachstr. 25
D-91052 Erlangen, Germany
Room: 2.305
Phone: +49 (0) 9131 / 85 - 20423
Fax: +49 (0) 9131 / 85 - 26565
=======================================================
E-mail: tobias.kloef...@fau.de<mailto:tobias.kloef...@fau.de>
_______________________________________________
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
