Ralph, I believe these nodes might have both an Ethernet and Infiniband port where the Ethernet port is not the one to use. Is there a way to tell Open MPI to ignore any ethernet devices it sees? I've tried:
--mca btl sm,openib,self and (based on the advice of the much more intelligent support at NAS): --mca btl openib,self --mca btl_openib_if_include mlx4_0,mlx4_1 But neither worked. Matt On Thu, Sep 24, 2015 at 11:41 AM, Ralph Castain <r...@open-mpi.org> wrote: > Starting in the 1.7 series, OMPI by default launches daemons on all nodes > in the allocation during startup. This is done so we can “probe” the > topology of the nodes and use that info during the process mapping > procedure - e.g., if you want to map-by NUMA regions. > > What is happening here is that some of the nodes in your allocation aren’t > allowing those daemons to callback to mpirun. Either a firewall is in the > way, or something is preventing it. > > If you don’t want to launch on those other nodes, you could just add —novm > to your cmd line, or use the —host option to restrict us to your local > node. However, I imagine you got the bigger allocation so you could use it > :-) > > In which case, you need to remove the obstacle. You might check for > firewall, or check to see if multiple NICs are on the non-maia nodes (this > can sometimes confuse things, especially if someone put the NICs on the > same IP subnet) > > HTH > Ralph > > > > On Sep 24, 2015, at 8:18 AM, Matt Thompson <fort...@gmail.com> wrote: > > Open MPI Users, > > I'm hoping someone here can help. I built Open MPI 1.10.0 with PGI 15.7 > using this configure string: > > ./configure --disable-vt --with-tm=/PBS --with-verbs > --disable-wrapper-rpath \ > CC=pgcc CXX=pgCC FC=pgf90 F77=pgf77 CFLAGS='-fpic -m64' \ > CXXFLAGS='-fpic -m64' FCFLAGS='-fpic -m64' FFLAGS='-fpic -m64' \ > --prefix=/nobackup/gmao_SIteam/MPI/pgi_15.7-openmpi_1.10.0 |& tee > configure.pgi15.7.log > > It seemed to pass 'make check'. > > I'm working at pleiades at NAS, and there they have both Sandy Bridge > nodes with GPUs (maia) and regular Sandy Bridge compute nodes (here after > called Sandy) without. To be extra careful (since PGI compiles to the > architecture you build on) I took a Westmere node and built Open MPI there > just in case. > > So, as I said, all seems to work with a test. I now grab a maia node, > maia1, of an allocation of 4 I had: > > (102) $ mpicc -tp=px-64 -o helloWorld.x helloWorld.c > (103) $ mpirun -np 2 ./helloWorld.x > Process 0 of 2 is on maia1 > Process 1 of 2 is on maia1 > > Good. Now, let's go to a Sandy Bridge (non-GPU) node, r321i7n16, of an > allocation of 8 I had: > > (49) $ mpicc -tp=px-64 -o helloWorld.x helloWorld.c > (50) $ mpirun -np 2 ./helloWorld.x > [r323i5n11:13063] [[62995,0],7] tcp_peer_send_blocking: send() to socket 9 > failed: Broken pipe (32) > [r323i5n6:57417] [[62995,0],2] tcp_peer_send_blocking: send() to socket 9 > failed: Broken pipe (32) > [r323i5n7:67287] [[62995,0],3] tcp_peer_send_blocking: send() to socket 9 > failed: Broken pipe (32) > [r323i5n8:57429] [[62995,0],4] tcp_peer_send_blocking: send() to socket 9 > failed: Broken pipe (32) > [r323i5n10:35329] [[62995,0],6] tcp_peer_send_blocking: send() to socket 9 > failed: Broken pipe (32) > [r323i5n9:13456] [[62995,0],5] tcp_peer_send_blocking: send() to socket 9 > failed: Broken pipe (32) > > Hmm. Let's try turning off tcp (often my first thought when on an > Infiniband system): > > (51) $ mpirun --mca btl sm,openib,self -np 2 ./helloWorld.x > [r323i5n6:57420] [[62996,0],2] tcp_peer_send_blocking: send() to socket 9 > failed: Broken pipe (32) > [r323i5n9:13459] [[62996,0],5] tcp_peer_send_blocking: send() to socket 9 > failed: Broken pipe (32) > [r323i5n8:57432] [[62996,0],4] tcp_peer_send_blocking: send() to socket 9 > failed: Broken pipe (32) > [r323i5n7:67290] [[62996,0],3] tcp_peer_send_blocking: send() to socket 9 > failed: Broken pipe (32) > [r323i5n11:13066] [[62996,0],7] tcp_peer_send_blocking: send() to socket 9 > failed: Broken pipe (32) > [r323i5n10:35332] [[62996,0],6] tcp_peer_send_blocking: send() to socket 9 > failed: Broken pipe (32) > > Now, the nodes reporting the issue seem to be the "other" nodes on the > allocation that are in a different rack: > > (52) $ cat $PBS_NODEFILE | uniq > r321i7n16 > r321i7n17 > r323i5n6 > r323i5n7 > r323i5n8 > r323i5n9 > r323i5n10 > r323i5n11 > > Maybe that's a clue? I didn't think this would matter if I only ran two > processes...and it works on the multi-node maia allocation. > > I've tried searching the web, but the only place I've seen > tcp_peer_send_blocking is in a PDF where they say it's an error that can be > seen: > > > http://www.hpc.mcgill.ca/downloads/checkpointing_workshop/20150326%20-%20McGill%20-%20Checkpointing%20Techniques.pdf > > Any ideas for what this error can mean? > > -- > Matt Thompson > > Man Among Men > Fulcrum of History > > _______________________________________________ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2015/09/27669.php > > > > _______________________________________________ > users mailing list > us...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users > Link to this post: > http://www.open-mpi.org/community/lists/users/2015/09/27670.php > -- Matt Thompson Man Among Men Fulcrum of History
