That backtrace shows we are registering MPI_Alloc_mem memory with verbs. This is expected behavior but it doesn’t show the openib btl being used for any communication. I am looking into a issue on an OmniPath system where just initializing the openib btl causes performance problems even if it is never used. It would be best to open an issue on this.
-Nathan > On Feb 7, 2017, at 2:50 PM, Jingchao Zhang <zh...@unl.edu> wrote: > > Hi Jeff, > > I just installed Open MPI: 2.0.2 (repo revision: v2.0.1-348-ge291d0e; release > date: Jan 31, 2017) but have the same problem. > > Attached please find two gdb backtraces on any write of a file descriptor > returned from opening /dev/infiniband/uverbs in the cp2k.popt process. > > Thanks, > Jingchao > From: users <users-boun...@lists.open-mpi.org> on behalf of Jeff Squyres > (jsquyres) <jsquy...@cisco.com> > Sent: Tuesday, February 7, 2017 2:14:40 PM > To: Open MPI User's List > Subject: Re: [OMPI users] openmpi single node jobs using btl openib > > Can you try upgrading to Open MPI v2.0.2? We just released that last week > with a bunch of bug fixes. > > > > On Feb 7, 2017, at 3:07 PM, Jingchao Zhang <zh...@unl.edu> wrote: > > > > Hi Tobias, > > > > Thanks for the reply. I tried both "export OMPI_MCA_mpi_leave_pinned=0" and > > "mpirun -mca mpi_leave_pinned 0" but still got the same behavior. Our > > OpenMPI version is 2.0.1. Repo version is v2.0.0-257-gee86e07. We have > > Intel Qlogic and OPA networks on the same cluster. > > > > Below is our configuration flags: > > ./configure --prefix=$PREFIX \ > > --with-hwloc=internal \ > > --enable-mpirun-prefix-by-default \ > > --with-slurm \ > > --with-verbs \ > > --with-psm \ > > --with-psm2 \ > > --disable-openib-connectx-xrc \ > > --with-knem=/opt/knem-1.1.2.90mlnx1 \ > > --with-cma > > > > So the question remains why OpenMPI choose openib over self,sm for single > > node jobs? Isn't there a mechanism to differentiate btl networks for > > single/multi-node jobs? > > > > Thanks, > > Jingchao > > From: users <users-boun...@lists.open-mpi.org> on behalf of Tobias Kloeffel > > <tobias.kloef...@fau.de> > > Sent: Tuesday, February 7, 2017 2:54:46 AM > > To: Open MPI Users > > Subject: Re: [OMPI users] openmpi single node jobs using btl openib > > > > Hello Jingchao, > > try to use -mca mpi_leave_pinned 0, also for multinode jobs. > > > > kind regards, > > Tobias Klöffel > > > > On 02/06/2017 09:38 PM, Jingchao Zhang wrote: > >> Hi, > >> > >> We recently noticed openmpi is using btl openib over self,sm for single > >> node jobs, which has caused performance degradation for some applications, > >> e.g. 'cp2k'. For opempi version 2.0.1, our test shows single node 'cp2k' > >> job using openib is ~25% slower than using self,sm. We advise users do > >> '--mca btl_base_exclude openib' as a temporary fix. I need to point out > >> that not all applications are affected by this feature. Many of them have > >> the same single-node performance with/without openib. Why doesn't openmpi > >> use self,sm by default for single node jobs? Is this the intended behavior? > >> > >> Thanks, > >> Jingchao > >> > >> > >> _______________________________________________ > >> users mailing list > >> > >> users@lists.open-mpi.org > >> https://rfd.newmexicoconsortium.org/mailman/listinfo/users > > > > > > -- > > M.Sc. Tobias Klöffel > > ======================================================= > > Interdisciplinary Center for Molecular Materials (ICMM) > > and Computer-Chemistry-Center (CCC) > > Department Chemie und Pharmazie > > Friedrich-Alexander-Universität Erlangen-Nürnberg > > Nägelsbachstr. 25 > > D-91052 Erlangen, Germany > > > > Room: 2.305 > > Phone: +49 (0) 9131 / 85 - 20423 > > Fax: +49 (0) 9131 / 85 - 26565 > > > > ======================================================= > > > > E-mail: > > tobias.kloef...@fau.de > > _______________________________________________ > > users mailing list > > users@lists.open-mpi.org > > https://rfd.newmexicoconsortium.org/mailman/listinfo/users > > > -- > Jeff Squyres > jsquy...@cisco.com > > _______________________________________________ > users mailing list > users@lists.open-mpi.org > https://rfd.newmexicoconsortium.org/mailman/listinfo/users > <ompi_2.0.2_bt.txt>_______________________________________________ > users mailing list > users@lists.open-mpi.org > https://rfd.newmexicoconsortium.org/mailman/listinfo/users _______________________________________________ users mailing list users@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/users
