The configure line was simply: ./configure --prefix=/home/rcohen
when I run: mpirun --mca btl self,vader,openib ... I get the same lousy results: 1.5 GFLOPS The output of the grep is: Cpus_allowed_list: 0-7 Cpus_allowed_list: 8-15 Cpus_allowed_list: 0-7 Cpus_allowed_list: 8-15 Cpus_allowed_list: 0-7 Cpus_allowed_list: 8-15 Cpus_allowed_list: 0-7 Cpus_allowed_list: 8-15 Cpus_allowed_list: 0-7 Cpus_allowed_list: 8-15 Cpus_allowed_list: 0-7 Cpus_allowed_list: 8-15 Cpus_allowed_list: 0-7 Cpus_allowed_list: 8-15 Cpus_allowed_list: 0-7 Cpus_allowed_list: 8-15 linpack *HPL) certainly is known to scale fine. I am running a standard benchmark--HPL--linpack. I think it is not the compiler, but I could try that. Ron --- Ron Cohen recoh...@gmail.com skypename: ronaldcohen twitter: @recohen3 On Wed, Mar 23, 2016 at 9:32 AM, Gilles Gouaillardet <gilles.gouaillar...@gmail.com> wrote: > Ronald, > > the fix I mentioned landed into the v1.10 branch > https://github.com/open-mpi/ompi-release/commit/c376994b81030cfa380c29d5b8f60c3e53d3df62 > > can you please post your configure command line ? > > you can also try to > mpirun --mca btl self,vader,openib ... > to make sure your run will abort instead of falling back to tcp > > then you can > mpirun ... grep Cpus_allowed_list /proc/self/status > to confirm your tasks do not end up bound to the same cores when running on > two nodes. > > is your application known to scale on infiniband network ? > or did you naively hope it would scale ? > > at first, I recommend you run standard benchmark to make sure you get the > performance you expect from your infiniband network > (for example IMB or OSU benchmark) > and run this test in the same environment than your app (e.g. via a batch > manager if applicable) > > if you do not get the performance you expect, then I suggest you try the > stock gcc compiler shipped with your distro and see if it helps. > > Cheers, > > Gilles > > On Wednesday, March 23, 2016, Ronald Cohen <recoh...@gmail.com> wrote: >> >> Thank you! Here are the answers: >> >> I did not try a previous release of gcc. >> I built from a tarball. >> What should I do about the iirc issue--how should I check? >> Are there any flags I should be using for infiniband? Is this a >> problem with latency? >> >> Ron >> >> >> --- >> Ron Cohen >> recoh...@gmail.com >> skypename: ronaldcohen >> twitter: @recohen3 >> >> >> On Wed, Mar 23, 2016 at 8:13 AM, Gilles Gouaillardet >> <gilles.gouaillar...@gmail.com> wrote: >> > Ronald, >> > >> > did you try to build openmpi with a previous gcc release ? >> > if yes, what about the performance ? >> > >> > did you build openmpi from a tarball or from git ? >> > if from git and without VPATH, then you need to >> > configure with --disable-debug >> > >> > iirc, one issue was identified previously >> > (gcc optimization that prevents the memory wrapper from behaving as >> > expected) and I am not sure the fix landed in v1.10 branch nor master >> > ... >> > >> > thanks for the info about gcc 6.0.0 >> > now this is supported on a free compiler >> > (cray and intel already support that, but they are commercial >> > compilers), >> > I will resume my work on supporting this >> > >> > Cheers, >> > >> > Gilles >> > >> > On Wednesday, March 23, 2016, Ronald Cohen <recoh...@gmail.com> wrote: >> >> >> >> I get 100 GFLOPS for 16 cores on one node, but 1 GFLOP running 8 cores >> >> on two nodes. It seems that quad-infiniband should do better than >> >> this. I built openmpi-1.10.2g with gcc version 6.0.0 20160317 . Any >> >> ideas of what to do to get usable performance? Thank you! >> >> >> >> bstatus >> >> Infiniband device 'mlx4_0' port 1 status: >> >> default gid: fe80:0000:0000:0000:0002:c903:00ec:9301 >> >> base lid: 0x1 >> >> sm lid: 0x1 >> >> state: 4: ACTIVE >> >> phys state: 5: LinkUp >> >> rate: 56 Gb/sec (4X FDR) >> >> link_layer: InfiniBand >> >> >> >> Ron >> >> -- >> >> >> >> Professor Dr. Ronald Cohen >> >> Ludwig Maximilians Universität >> >> Theresienstrasse 41 Room 207 >> >> Department für Geo- und Umweltwissenschaften >> >> München >> >> 80333 >> >> Deutschland >> >> >> >> >> >> ronald.co...@min.uni-muenchen.de >> >> skype: ronaldcohen >> >> +49 (0) 89 74567980 >> >> --- >> >> Ronald Cohen >> >> Geophysical Laboratory >> >> Carnegie Institution >> >> 5251 Broad Branch Rd., N.W. >> >> Washington, D.C. 20015 >> >> rco...@carnegiescience.edu >> >> office: 202-478-8937 >> >> skype: ronaldcohen >> >> https://twitter.com/recohen3 >> >> https://www.linkedin.com/profile/view?id=163327727 >> >> >> >> >> >> --- >> >> Ron Cohen >> >> recoh...@gmail.com >> >> skypename: ronaldcohen >> >> twitter: @recohen3 >> >> _______________________________________________ >> >> users mailing list >> >> us...@open-mpi.org >> >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >> >> Link to this post: >> >> http://www.open-mpi.org/community/lists/users/2016/03/28791.php >> > >> > >> > _______________________________________________ >> > users mailing list >> > us...@open-mpi.org >> > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >> > Link to this post: >> > http://www.open-mpi.org/community/lists/users/2016/03/28793.php >> _______________________________________________ >> users mailing list >> us...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >> Link to this post: >> http://www.open-mpi.org/community/lists/users/2016/03/28794.php > > > _______________________________________________ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2016/03/28796.php
