Ronald, the fix I mentioned landed into the v1.10 branch https://github.com/open-mpi/ompi-release/commit/c376994b81030cfa380c29d5b8f60c3e53d3df62
can you please post your configure command line ? you can also try to mpirun --mca btl self,vader,openib ... to make sure your run will abort instead of falling back to tcp then you can mpirun ... grep Cpus_allowed_list /proc/self/status to confirm your tasks do not end up bound to the same cores when running on two nodes. is your application known to scale on infiniband network ? or did you naively hope it would scale ? at first, I recommend you run standard benchmark to make sure you get the performance you expect from your infiniband network (for example IMB or OSU benchmark) and run this test in the same environment than your app (e.g. via a batch manager if applicable) if you do not get the performance you expect, then I suggest you try the stock gcc compiler shipped with your distro and see if it helps. Cheers, Gilles On Wednesday, March 23, 2016, Ronald Cohen <recoh...@gmail.com> wrote: > Thank you! Here are the answers: > > I did not try a previous release of gcc. > I built from a tarball. > What should I do about the iirc issue--how should I check? > Are there any flags I should be using for infiniband? Is this a > problem with latency? > > Ron > > > --- > Ron Cohen > recoh...@gmail.com <javascript:;> > skypename: ronaldcohen > twitter: @recohen3 > > > On Wed, Mar 23, 2016 at 8:13 AM, Gilles Gouaillardet > <gilles.gouaillar...@gmail.com <javascript:;>> wrote: > > Ronald, > > > > did you try to build openmpi with a previous gcc release ? > > if yes, what about the performance ? > > > > did you build openmpi from a tarball or from git ? > > if from git and without VPATH, then you need to > > configure with --disable-debug > > > > iirc, one issue was identified previously > > (gcc optimization that prevents the memory wrapper from behaving as > > expected) and I am not sure the fix landed in v1.10 branch nor master ... > > > > thanks for the info about gcc 6.0.0 > > now this is supported on a free compiler > > (cray and intel already support that, but they are commercial compilers), > > I will resume my work on supporting this > > > > Cheers, > > > > Gilles > > > > On Wednesday, March 23, 2016, Ronald Cohen <recoh...@gmail.com > <javascript:;>> wrote: > >> > >> I get 100 GFLOPS for 16 cores on one node, but 1 GFLOP running 8 cores > >> on two nodes. It seems that quad-infiniband should do better than > >> this. I built openmpi-1.10.2g with gcc version 6.0.0 20160317 . Any > >> ideas of what to do to get usable performance? Thank you! > >> > >> bstatus > >> Infiniband device 'mlx4_0' port 1 status: > >> default gid: fe80:0000:0000:0000:0002:c903:00ec:9301 > >> base lid: 0x1 > >> sm lid: 0x1 > >> state: 4: ACTIVE > >> phys state: 5: LinkUp > >> rate: 56 Gb/sec (4X FDR) > >> link_layer: InfiniBand > >> > >> Ron > >> -- > >> > >> Professor Dr. Ronald Cohen > >> Ludwig Maximilians Universität > >> Theresienstrasse 41 Room 207 > >> Department für Geo- und Umweltwissenschaften > >> München > >> 80333 > >> Deutschland > >> > >> > >> ronald.co...@min.uni-muenchen.de <javascript:;> > >> skype: ronaldcohen > >> +49 (0) 89 74567980 > >> --- > >> Ronald Cohen > >> Geophysical Laboratory > >> Carnegie Institution > >> 5251 Broad Branch Rd., N.W. > >> Washington, D.C. 20015 > >> rco...@carnegiescience.edu <javascript:;> > >> office: 202-478-8937 > >> skype: ronaldcohen > >> https://twitter.com/recohen3 > >> https://www.linkedin.com/profile/view?id=163327727 > >> > >> > >> --- > >> Ron Cohen > >> recoh...@gmail.com <javascript:;> > >> skypename: ronaldcohen > >> twitter: @recohen3 > >> _______________________________________________ > >> users mailing list > >> us...@open-mpi.org <javascript:;> > >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > >> Link to this post: > >> http://www.open-mpi.org/community/lists/users/2016/03/28791.php > > > > > > _______________________________________________ > > users mailing list > > us...@open-mpi.org <javascript:;> > > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > > Link to this post: > > http://www.open-mpi.org/community/lists/users/2016/03/28793.php > _______________________________________________ > users mailing list > us...@open-mpi.org <javascript:;> > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2016/03/28794.php
