subject:"\[OMPI users\] MPI hangs on multiple nodes"

Re: [OMPI users] MPI hangs on multiple nodes

2011-09-25 Thread Ole Nielsen

Dear all - thanks to everyone for all the hints. Problem has been solved.
To summarise for the benefit of anyone else who may encounter the issue of
MPI hanging when trying to connect through the network:


   1. Followed Rolf's advice and also hints from
   http://www.open-mpi.org/faq/?category=tcp) and ran MPI with the option
   btl_base_verbose.
   2. This revealed that MPI was trying to connect through the wrong IP
   addresses for some nodes:
   http://www.aifdr.org/projects/system_administration/ticket/17#comment:4
   3. This in turn, was due to our cluster was using different eth* to mac
   address mappings leading DHCP to issues the wrong addresses:
   http://www.aifdr.org/projects/system_administration/ticket/9#comment:10 -
   Interestingly, we had no problem with ping and ssh, so maybe MPI defaults to
   using eth0?
   4. After much fiddling with udev/rules.d we gave up on that and just
   assigned static addresses to eth0 (I am now convinced udev/rules.d doesn't
   work :-))

After that everything works beautifully.
The morale is KISS - Keep everything as simple as you can.

Onto some more earthquake and tsunami modelling.

Cheers
Ole



On Tue, Sep 20, 2011 at 9:44 PM, <users-requ...@open-mpi.org> wrote:

> Send users mailing list submissions to
>us...@open-mpi.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>http://www.open-mpi.org/mailman/listinfo.cgi/users
> or, via email, send a message with subject or body 'help' to
>users-requ...@open-mpi.org
>
> You can reach the person managing the list at
>users-ow...@open-mpi.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of users digest..."
>
>
> Today's Topics:
>
>   1. Re: RE :  MPI hangs on multiple nodes (Gus Correa)
>   2. Typo in MPI_Cart_coords man page (Jeremiah Willcock)
>   3. Re: RE :  MPI hangs on multiple nodes (Gus Correa)
>   4. How could OpenMPI (or MVAPICH) affect floating-point  results?
>  (Blosch, Edwin L)
>   5. MPI hangs on multiple nodes (Ole Nielsen)
>   6. MPI hangs on multiple nodes (Ole Nielsen)
>   7. Re: How could OpenMPI (or MVAPICH) affect floating-point
>  results? (Reuti)
>   8. Re: How could OpenMPI (or MVAPICH) affect floating-point
>  results? (Tim Prince)
>   9. Re: How could OpenMPI (or MVAPICH) affect floating-point
>  results? (Jeff Squyres)
>  10. Re: MPI hangs on multiple nodes (Jeff Squyres)
>  11. Re: Latency of 250 microseconds with Open-MPI 1.4.3, Mellanox
>  Infiniband and 256 MPI ranks (Yevgeny Kliteynik)
>  12. Re: How could OpenMPI (or MVAPICH) affect floating-point
>  results? (Reuti)
>  13. Re: MPI hangs on multiple nodes (Rolf vandeVaart)
>  14. Re: Open MPI and Objective C (Barrett, Brian W)
>  15. Re: How could OpenMPI (or MVAPICH) affect floating-point
>  results? (Samuel K. Gutierrez)
>
>
> ------
>
> Message: 1
> Date: Mon, 19 Sep 2011 13:13:08 -0400
> From: Gus Correa <g...@ldeo.columbia.edu>
> Subject: Re: [OMPI users] RE :  MPI hangs on multiple nodes
> To: Open MPI Users <us...@open-mpi.org>
> Message-ID: <4e777824.60...@ldeo.columbia.edu>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
> Hi Eugene
>
> You're right, it is blocking send, buffers can be reused after MPI_Send
> returns.
> My bad, I only read your answer to Sebastien and Ole
> after I posted mine.
>
> Could MPI run out of [internal] buffers to hold the messages, perhaps?
> The messages aren't that big anyway [5000 doubles].
> Could MPI behave differently regarding internal
> buffering when communication is intra-node vs. across the network?
> [It works intra-node, according to Ole's posting.]
>
> I suppose Ole rebuilt OpenMPI on his newly installed Ubuntu.
>
> Gus Correa
>
>
> Eugene Loh wrote:
> > I'm missing the point on the buffer re-use.  It seems to me that the
> > sample program passes some buffer around in a ring.  Each process
> > receives the buffer with a blocking receive and then forwards it with a
> > blocking send.  The blocking send does not return until the send buffer
> > is safe to reuse.
> >
> > On 9/19/2011 7:37 AM, Gus Correa wrote:
> >> You could try the examples/connectivity.c program in the
> >> OpenMPI source tree, to test if everything is alright.
> >> It also hints how to solve the buffer re-use issue
> >> that Sebastien [rightfully] pointed out [i.e., declare separate
> >> buffers for MPI_Send and MPI_Recv].
> >>
> >> S?bastien Boisvert wrote:
> >>> Is it safe to re-use the same buffer (variable A) for MPI_Send and
> >>> MPI

Re: [OMPI users] MPI hangs on multiple nodes

2011-09-20 Thread Gus Correa


Ole Nielsen wrote:
Thanks for your suggestion Gus, we need a way of debugging what is going 
on. I am pretty sure the problem lies with our cluster configuration. I 
know MPI simply relies on the underlying network. However, we can ping 
and ssh to all nodes (and in between and pair as well) so it is 
currently a mystery why MPI doesn't communicate across nodes on our cluster.

Two further questions for the group

   1. I would love to run the test program connectivity.c, but cannot
  find it anywhere. Can anyone help please?


If you downloaded the OpenMPI tarball, it is in examples/connectivity.c
wherever you untarred it [now where you installed].



   2. After having left the job hanging over night we got the message
  
[node5][[9454,1],1][../../../../../../ompi/mca/btl/tcp/btl_tcp_frag.c:216:mca_btl_tcp_frag_recv]
  mca_btl_tcp_frag_recv: readv failed: Connection timed out (110).
  Does anyone know what this means?


Cheers and thanks
Ole
PS - I don't see how separate buffers would help. Recall that the test 
program I use works fine on other installations and indeed when run on 
one the cores of one Node.




It probably won't help, as Eugene explained.
Your program works here, worked also for Davendra Rai.
If you were using MPI_ISend [non-blocking],
then you would need separate buffers.

For large amounts of data and many processes,
I would rather use non-blocking communication [and separate
buffers], specially if you do work between send and recv.
But that's not what hangs your program.

Gus Correa






Message: 11
Date: Mon, 19 Sep 2011 10:37:02 -0400
From: Gus Correa <g...@ldeo.columbia.edu <mailto:g...@ldeo.columbia.edu>>
Subject: Re: [OMPI users] RE :  MPI hangs on multiple nodes
To: Open MPI Users <us...@open-mpi.org <mailto:us...@open-mpi.org>>
Message-ID: <4e77538e.3070...@ldeo.columbia.edu 
<mailto:4e77538e.3070...@ldeo.columbia.edu>>

Content-Type: text/plain; charset=iso-8859-1; format=flowed

Hi Ole

You could try the examples/connectivity.c program in the
OpenMPI source tree, to test if everything is alright.
It also hints how to solve the buffer re-use issue
that Sebastien [rightfully] pointed out [i.e., declare separate
buffers for MPI_Send and MPI_Recv].

Gus Correa




___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] MPI hangs on multiple nodes

2011-09-20 Thread Rolf vandeVaart


>> 1: After a reboot of two nodes I ran again, and the inter-node freeze didn't
>happen until the third iteration. I take that to mean that the basic
>communication works, but that something is saturating. Is there some notion
>of buffer size somewhere in the MPI system that could explain this?
>
>Hmm.  This is not a good sign; it somewhat indicates a problem with your OS.
>Based on this email and your prior emails, I'm guessing you're using TCP for
>communication, and that the problem is based on inter-node communication
>(e.g., the problem would occur even if you only run 1 process per machine,
>but does not occur if you run all N processes on a single machine, per your #4,
>below).
>

I agree with Jeff here.  Open MPI uses lazy connections to establish 
connections and round robins through the interfaces.
So, the first few communications could work as they are using interfaces that 
could communicate between the nodes, but the third iteration uses an interface 
that for some reason cannot establish the connection.

One flag you can use that may help is --mca btl_base_verbose 20, like this;

mpirun --mca btl_base_verbose 20 connectivity_c

It will dump out a bunch of stuff, but there will be a few lines that look like 
this:

[...snip...]
[dt:09880] btl: tcp: attempting to connect() to [[58627,1],1] address 
10.20.14.101 on port 1025
[...snip...]

Rolf


---
This email message is for the sole use of the intended recipient(s) and may 
contain
confidential information.  Any unauthorized review, use, disclosure or 
distribution
is prohibited.  If you are not the intended recipient, please contact the 
sender by
reply email and destroy all copies of the original message.
---

Re: [OMPI users] MPI hangs on multiple nodes

2011-09-20 Thread Jeff Squyres

On Sep 19, 2011, at 10:23 PM, Ole Nielsen wrote:

> Hi all - and sorry for the multiple postings, but I have more information.

+1 on Eugene's comments.  The test program looks fine to me.

FWIW, you don't need -lmpi to compile your program; OMPI's wrapper compiler 
allows you to just:

mpicc mpi_test.c -o mpi_test -Wall

> 1: After a reboot of two nodes I ran again, and the inter-node freeze didn't 
> happen until the third iteration. I take that to mean that the basic 
> communication works, but that something is saturating. Is there some notion 
> of buffer size somewhere in the MPI system that could explain this?

Hmm.  This is not a good sign; it somewhat indicates a problem with your OS.  
Based on this email and your prior emails, I'm guessing you're using TCP for 
communication, and that the problem is based on inter-node communication (e.g., 
the problem would occur even if you only run 1 process per machine, but does 
not occur if you run all N processes on a single machine, per your #4, below).

> 2: The nodes have 4 ethernet cards each. Could the mapping be a problem?

Shouldn't be.  If it runs at all, then it should run fine.

Do you have all your ethernet cards on a single subnet, or multiple subnets?  I 
have heard of problems when you have multiple ethernet cards on the same subnet 
-- I believe there's some non-determinism in than case in what wire/NIC a 
packet will actually go out, which may be problematic for OMPI.

> 3: The cpus are running at a 100% for all processes involved in the freeze

That's probably right.  OMPI aggressively polls for progress as a way to 
decrease latency.  So all processes are trying to make progress, and therefore 
are aggressively polling, eating up 100% of the CPU.

> 4: The same test program 
> (http://code.google.com/p/pypar/source/browse/source/mpi_test.c) works fine 
> when run within one node so the problem must be with MPI and/or our network. 

This helps identify the issue as the TCP communication, not the shared memory 
communication.

> 5: The network and ssh works otherwise fine.

Good.

> Again many thanks for any hint that can get us going again. The main thing we 
> need is some diagnostics that may point to what causes this problem for MPI.

If you are running with multiple NICs on the same subnet, change them to 
multiple subnets and see if it starts working fine.

If they're on different subnets, try using the btl_tcp_if_include / 
btl_tcp_if_exclude MCA parameters to exclude certain networks and see if 
they're the problematic ones.  Keep in mind that ..._include and ..._exclude 
are mutually exclusive; you should only specify one.  And if you specify 
exclude, be sure to exclude loopback.  E.g:

  mpirun --mca btl_if_include eth0,eth1 -np 16 --hostfile hostfile mpi_test
or
  mpirun --mca btl_if_exclude lo0,eth1 -np 16 --hostfile hostfile mpi_test

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/

[OMPI users] MPI hangs on multiple nodes

2011-09-19 Thread Ole Nielsen

Hi all - and sorry for the multiple postings, but I have more information.

1: After a reboot of two nodes I ran again, and the inter-node freeze didn't
happen until the third iteration. I take that to mean that the basic
communication works, but that something is saturating. Is there some notion
of buffer size somewhere in the MPI system that could explain this?
2: The nodes have 4 ethernet cards each. Could the mapping be a problem?
3: The cpus are running at a 100% for all processes involved in the freeze
4: The same test program (
http://code.google.com/p/pypar/source/browse/source/mpi_test.c) works fine
when run within one node so the problem must be with MPI and/or our network.

5: The network and ssh works otherwise fine.


Again many thanks for any hint that can get us going again. The main thing
we need is some diagnostics that may point to what causes this problem for
MPI.
Cheers
Ole Nielsen

--

Here's the output which shows the freeze in the third iteration:

nielso@alamba:~/sandpit/pypar/source$ mpirun --hostfile /etc/mpihosts --host
node5,node6 --npernode 2 a.out
Number of processes = 4
Test repeated 3 times for reliability
I am process 2 on node node6
P2: Waiting to receive from to P1
P2: Sending to to P3
I am process 3 on node node6
P3: Waiting to receive from to P2
I am process 1 on node node5
P1: Waiting to receive from to P0
P1: Sending to to P2
P1: Waiting to receive from to P0
I am process 0 on node node5
Run 1 of 3
P0: Sending to P1
P0: Waiting to receive from P3
P2: Waiting to receive from to P1
P3: Sending to to P0
P3: Waiting to receive from to P2
P1: Sending to to P2
P0: Received from to P3
Run 2 of 3
P0: Sending to P1
P0: Waiting to receive from P3
P1: Waiting to receive from to P0

[OMPI users] MPI hangs on multiple nodes

2011-09-19 Thread Ole Nielsen

Thanks for your suggestion Gus, we need a way of debugging what is going on.
I am pretty sure the problem lies with our cluster configuration. I know MPI
simply relies on the underlying network. However, we can ping and ssh to all
nodes (and in between and pair as well) so it is currently a mystery why MPI
doesn't communicate across nodes on our cluster.
Two further questions for the group

   1. I would love to run the test program connectivity.c, but cannot find
   it anywhere. Can anyone help please?
   2. After having left the job hanging over night we got the message
   
[node5][[9454,1],1][../../../../../../ompi/mca/btl/tcp/btl_tcp_frag.c:216:mca_btl_tcp_frag_recv]
   mca_btl_tcp_frag_recv: readv failed: Connection timed out (110). Does anyone
   know what this means?


Cheers and thanks
Ole
PS - I don't see how separate buffers would help. Recall that the test
program I use works fine on other installations and indeed when run on one
the cores of one Node.




Message: 11
List-Post: users@lists.open-mpi.org
Date: Mon, 19 Sep 2011 10:37:02 -0400
From: Gus Correa <g...@ldeo.columbia.edu>
Subject: Re: [OMPI users] RE :  MPI hangs on multiple nodes
To: Open MPI Users <us...@open-mpi.org>
Message-ID: <4e77538e.3070...@ldeo.columbia.edu>
Content-Type: text/plain; charset=iso-8859-1; format=flowed

Hi Ole

You could try the examples/connectivity.c program in the
OpenMPI source tree, to test if everything is alright.
It also hints how to solve the buffer re-use issue
that Sebastien [rightfully] pointed out [i.e., declare separate
buffers for MPI_Send and MPI_Recv].

Gus Correa

Re: [OMPI users] MPI hangs on multiple nodes

2011-09-19 Thread devendra rai

Hello Ole

I ran your program on open-mpi-1.4.2  five times, and all five times, it 
finished successfully.

So, I think the problem was with the version of mpi.

Output from your program is attached. I ran on 3 nodes:

$home/OpenMPI-1.4.2/bin/mpirun -np 3 -v --output-filename mpi_testfile 
./mpi_test

So, maybe this helps you.

Best,

Devendra Rai




From: Ole Nielsen <ole.moller.niel...@gmail.com>
To: us...@open-mpi.org
Sent: Monday, 19 September 2011, 10:59
Subject: [OMPI users] MPI hangs on multiple nodes


The test program is available here:
http://code.google.com/p/pypar/source/browse/source/mpi_test.c

Hopefully, someone can help us troubleshoot why communications stop when 
multiple nodes are involved and CPU usage goes to 100% for as long as we leave 
the program running.

Many thanks
Ole Nielsen



-- Forwarded message --
From: Ole Nielsen <ole.moller.niel...@gmail.com>
Date: Mon, Sep 19, 2011 at 3:39 PM
Subject: Re: MPI hangs on multiple nodes
To: us...@open-mpi.org


Further to the posting below, I can report that the test program (attached - 
this time correctly) is chewing up CPU time on both compute nodes for as long 
as I care to let it continue.
It would appear that MPI_Receive which is the next command after the print 
statements in the test program.

Has anyone else seen this behavior or can anyone give me a hint on how to 
troubleshoot.

Cheers and thanks
Ole Nielsen

Output:

nielso@alamba:~/sandpit/pypar/source$ mpirun --hostfile /etc/mpihosts --host 
node17,node18 --npernode 2 a.out 
Number of processes = 4
Test repeated 3 times for reliability

I am process 2 on node node18
P2: Waiting to receive from to P1

I am process 0 on node node17
Run 1 of 3
P0: Sending to P1
I am process 1 on node node17
P1: Waiting to receive from to P0

I am process 3 on node node18
P3: Waiting to receive from to P2
P0: Waiting to receive from P3

P1: Sending to to P2

P1: Waiting to receive from to P0
P2: Sending to to P3

P0: Received from to P3
Run 2 of 3
P0: Sending to P1
P3: Sending to to P0

P3: Waiting to receive from to P2

P2: Waiting to receive from to P1

P1: Sending to to P2
P0: Waiting to receive from P3







 


On Mon, Sep 19, 2011 at 11:04 AM, Ole Nielsen <ole.moller.niel...@gmail.com> 
wrote:


>
>Hi all
>
>We have been using OpenMPI for many years with Ubuntu on our 20-node cluster. 
>Each node has 2 quad cores, so we usually run up to 8 processes on each node 
>up to a maximum of 160 processes.
>
>However, we just upgraded the cluster to Ubuntu 11.04 with Open MPI 1.4.3 and 
>and have come across a strange behavior where mpi programs run perfectly well 
>when confined to one node but hangs during communication across multiple 
>nodes. We have no idea why and would like some help in debugging this. A small 
>MPI test program is attached and typical output shown below.
>
>Hope someone can help us
>Cheers and thanks
>Ole Nielsen
>
> Test output across two nodes (This one hangs) 
>--
>nielso@alamba:~/sandpit/pypar/source$ mpirun --hostfile /etc/mpihosts --host 
>node17,node18 --npernode 2 a.out 
>Number of processes = 4
>Test repeated 3 times for reliability
>I am process 1 on node node17
>P1: Waiting to receive from to P0
>I am process 0 on node node17
>Run 1 of 3
>P0: Sending to P1
>I am process 2 on node node18
>P2: Waiting to receive from to P1
>I am process 3 on node node18
>P3: Waiting to receive from to P2
>P1: Sending to to P2
>
>
> Test output within one node (This one is OK) 
>--
>nielso@alamba:~/sandpit/pypar/source$ mpirun --hostfile /etc/mpihosts --host 
>node17 --npernode 4 a.out 
>Number of processes = 4
>Test repeated 3 times for reliability
>I am process 2 on node node17
>P2: Waiting to receive from to P1
>I am process 0 on node node17
>Run 1 of 3
>P0: Sending to P1
>I am process 1 on node node17
>P1: Waiting to receive from to P0
>I am process 3 on node node17
>P3: Waiting to receive from to P2
>P1: Sending to to P2
>P2: Sending to to P3
>P1: Waiting to receive from to P0
>P2: Waiting to receive from to P1
>P3: Sending to to P0
>P0: Received from to P3
>Run 2 of 3
>P0: Sending to P1
>P3: Waiting to receive from to P2
>P1: Sending to to P2
>P2: Sending to to P3
>P1: Waiting to receive from to P0
>P3: Sending to to P0
>P2: Waiting to receive from to P1
>P0: Received from to P3
>Run 3 of 3
>P0: Sending to P1
>P3: Waiting to receive from to P2
>P1: Sending to to P2
>P2: Sending to to P3
>P1: Done
>P2: Done
>P3: Sending to to P0
>P0: Received from to P3
>P0: Done
>P3: Done
>
>
>
>


___
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users

mpi_testfile.1
Description: Binary data


mpi_testfile.2
Description: Binary data


mpi_testfile.0
Description: Binary data

[OMPI users] MPI hangs on multiple nodes

2011-09-19 Thread Ole Nielsen

The test program is available here:
http://code.google.com/p/pypar/source/browse/source/mpi_test.c

Hopefully, someone can help us troubleshoot why communications stop when
multiple nodes are involved and CPU usage goes to 100% for as long as we
leave the program running.

Many thanks
Ole Nielsen


-- Forwarded message --
From: Ole Nielsen 
List-Post: users@lists.open-mpi.org
Date: Mon, Sep 19, 2011 at 3:39 PM
Subject: Re: MPI hangs on multiple nodes
To: us...@open-mpi.org


Further to the posting below, I can report that the test program (attached -
this time correctly) is chewing up CPU time on both compute nodes for as
long as I care to let it continue.
It would appear that MPI_Receive which is the next command after the print
statements in the test program.

Has anyone else seen this behavior or can anyone give me a hint on how to
troubleshoot.

Cheers and thanks
Ole Nielsen

Output:

nielso@alamba:~/sandpit/pypar/source$ mpirun --hostfile /etc/mpihosts --host
node17,node18 --npernode 2 a.out
Number of processes = 4
Test repeated 3 times for reliability
I am process 2 on node node18
P2: Waiting to receive from to P1
I am process 0 on node node17
Run 1 of 3
P0: Sending to P1
I am process 1 on node node17
P1: Waiting to receive from to P0
I am process 3 on node node18
P3: Waiting to receive from to P2
P0: Waiting to receive from P3

P1: Sending to to P2
P1: Waiting to receive from to P0
P2: Sending to to P3

P0: Received from to P3
Run 2 of 3
P0: Sending to P1
P3: Sending to to P0

P3: Waiting to receive from to P2
P2: Waiting to receive from to P1
P1: Sending to to P2
P0: Waiting to receive from P3









On Mon, Sep 19, 2011 at 11:04 AM, Ole Nielsen
wrote:

>
> Hi all
>
> We have been using OpenMPI for many years with Ubuntu on our 20-node
> cluster. Each node has 2 quad cores, so we usually run up to 8 processes on
> each node up to a maximum of 160 processes.
>
> However, we just upgraded the cluster to Ubuntu 11.04 with Open MPI 1.4.3
> and and have come across a strange behavior where mpi programs run perfectly
> well when confined to one node but hangs during communication across
> multiple nodes. We have no idea why and would like some help in debugging
> this. A small MPI test program is attached and typical output shown below.
>
> Hope someone can help us
> Cheers and thanks
> Ole Nielsen
>
>  Test output across two nodes (This one hangs)
> --
> nielso@alamba:~/sandpit/pypar/source$ mpirun --hostfile /etc/mpihosts
> --host node17,node18 --npernode 2 a.out
> Number of processes = 4
> Test repeated 3 times for reliability
> I am process 1 on node node17
> P1: Waiting to receive from to P0
> I am process 0 on node node17
> Run 1 of 3
> P0: Sending to P1
> I am process 2 on node node18
> P2: Waiting to receive from to P1
> I am process 3 on node node18
> P3: Waiting to receive from to P2
> P1: Sending to to P2
>
>
>  Test output within one node (This one is OK)
> --
> nielso@alamba:~/sandpit/pypar/source$ mpirun --hostfile /etc/mpihosts
> --host node17 --npernode 4 a.out
> Number of processes = 4
> Test repeated 3 times for reliability
> I am process 2 on node node17
> P2: Waiting to receive from to P1
> I am process 0 on node node17
> Run 1 of 3
> P0: Sending to P1
> I am process 1 on node node17
> P1: Waiting to receive from to P0
> I am process 3 on node node17
> P3: Waiting to receive from to P2
> P1: Sending to to P2
> P2: Sending to to P3
> P1: Waiting to receive from to P0
> P2: Waiting to receive from to P1
> P3: Sending to to P0
> P0: Received from to P3
> Run 2 of 3
> P0: Sending to P1
> P3: Waiting to receive from to P2
> P1: Sending to to P2
> P2: Sending to to P3
> P1: Waiting to receive from to P0
> P3: Sending to to P0
> P2: Waiting to receive from to P1
> P0: Received from to P3
> Run 3 of 3
> P0: Sending to P1
> P3: Waiting to receive from to P2
> P1: Sending to to P2
> P2: Sending to to P3
> P1: Done
> P2: Done
> P3: Sending to to P0
> P0: Received from to P3
> P0: Done
> P3: Done
>
>
>
>

Re: [OMPI users] MPI hangs on multiple nodes

2011-09-19 Thread Ole Nielsen

Further to the posting below, I can report that the test program (attached -
this time correctly) is chewing up CPU time on both compute nodes for as
long as I care to let it continue.
It would appear that MPI_Receive which is the next command after the print
statements in the test program.

Has anyone else seen this behavior or can anyone give me a hint on how to
troubleshoot.
Cheers and thanks
Ole Nielsen

Output:
nielso@alamba:~/sandpit/pypar/source$ mpirun --hostfile /etc/mpihosts --host
node17,node18 --npernode 2 a.out
Number of processes = 4
Test repeated 3 times for reliability
I am process 2 on node node18
P2: Waiting to receive from to P1
I am process 0 on node node17
Run 1 of 3
P0: Sending to P1
I am process 1 on node node17
P1: Waiting to receive from to P0
I am process 3 on node node18
P3: Waiting to receive from to P2
P0: Waiting to receive from P3
P1: Sending to to P2
P1: Waiting to receive from to P0
P2: Sending to to P3
P0: Received from to P3
Run 2 of 3
P0: Sending to P1
P3: Sending to to P0
P3: Waiting to receive from to P2
P2: Waiting to receive from to P1
P1: Sending to to P2
P0: Waiting to receive from P3








On Mon, Sep 19, 2011 at 11:04 AM, Ole Nielsen
wrote:

>
> Hi all
>
> We have been using OpenMPI for many years with Ubuntu on our 20-node
> cluster. Each node has 2 quad cores, so we usually run up to 8 processes on
> each node up to a maximum of 160 processes.
>
> However, we just upgraded the cluster to Ubuntu 11.04 with Open MPI 1.4.3
> and and have come across a strange behavior where mpi programs run perfectly
> well when confined to one node but hangs during communication across
> multiple nodes. We have no idea why and would like some help in debugging
> this. A small MPI test program is attached and typical output shown below.
>
> Hope someone can help us
> Cheers and thanks
> Ole Nielsen
>
>  Test output across two nodes (This one hangs)
> --
> nielso@alamba:~/sandpit/pypar/source$ mpirun --hostfile /etc/mpihosts
> --host node17,node18 --npernode 2 a.out
> Number of processes = 4
> Test repeated 3 times for reliability
> I am process 1 on node node17
> P1: Waiting to receive from to P0
> I am process 0 on node node17
> Run 1 of 3
> P0: Sending to P1
> I am process 2 on node node18
> P2: Waiting to receive from to P1
> I am process 3 on node node18
> P3: Waiting to receive from to P2
> P1: Sending to to P2
>
>
>  Test output within one node (This one is OK)
> --
> nielso@alamba:~/sandpit/pypar/source$ mpirun --hostfile /etc/mpihosts
> --host node17 --npernode 4 a.out
> Number of processes = 4
> Test repeated 3 times for reliability
> I am process 2 on node node17
> P2: Waiting to receive from to P1
> I am process 0 on node node17
> Run 1 of 3
> P0: Sending to P1
> I am process 1 on node node17
> P1: Waiting to receive from to P0
> I am process 3 on node node17
> P3: Waiting to receive from to P2
> P1: Sending to to P2
> P2: Sending to to P3
> P1: Waiting to receive from to P0
> P2: Waiting to receive from to P1
> P3: Sending to to P0
> P0: Received from to P3
> Run 2 of 3
> P0: Sending to P1
> P3: Waiting to receive from to P2
> P1: Sending to to P2
> P2: Sending to to P3
> P1: Waiting to receive from to P0
> P3: Sending to to P0
> P2: Waiting to receive from to P1
> P0: Received from to P3
> Run 3 of 3
> P0: Sending to P1
> P3: Waiting to receive from to P2
> P1: Sending to to P2
> P2: Sending to to P3
> P1: Done
> P2: Done
> P3: Sending to to P0
> P0: Received from to P3
> P0: Done
> P3: Done
>
>
>
>
/*
  Simple MPI communication test.
  Ole Moller Nielsen - 2011
*/

#include 
#include 
#include 


#define M  5000 /* Data size */


int main(int argc, char **argv) {
   int repeats = 3, msgid = 0;
   int myid, procs;
   int j, k;

   double A[M];

   int  namelen;
   char processor_name[MPI_MAX_PROCESSOR_NAME];

   MPI_Status stat;


   /* Initialize */
   MPI_Init(, );
   MPI_Comm_size(MPI_COMM_WORLD, );
   MPI_Comm_rank(MPI_COMM_WORLD, );
   MPI_Get_processor_name(processor_name, );

   if (myid == 0) {
 printf("Number of processes = %d\n", procs);
 printf("Test repeated %d times for reliability\n", repeats);
   }

   if (procs < 2) {
 printf("Program needs at least two processors - aborting\n");
 MPI_Abort(MPI_COMM_WORLD,999);
   }

   /* Create the data */
   for (j=0; j

[OMPI users] MPI hangs on multiple nodes

2011-09-19 Thread Ole Nielsen

Hi all

We have been using OpenMPI for many years with Ubuntu on our 20-node
cluster. Each node has 2 quad cores, so we usually run up to 8 processes on
each node up to a maximum of 160 processes.

However, we just upgraded the cluster to Ubuntu 11.04 with Open MPI 1.4.3
and and have come across a strange behavior where mpi programs run perfectly
well when confined to one node but hangs during communication across
multiple nodes. We have no idea why and would like some help in debugging
this. A small MPI test program is attached and typical output shown below.

Hope someone can help us
Cheers and thanks
Ole Nielsen

 Test output across two nodes (This one hangs)
--
nielso@alamba:~/sandpit/pypar/source$ mpirun --hostfile /etc/mpihosts --host
node17,node18 --npernode 2 a.out
Number of processes = 4
Test repeated 3 times for reliability
I am process 1 on node node17
P1: Waiting to receive from to P0
I am process 0 on node node17
Run 1 of 3
P0: Sending to P1
I am process 2 on node node18
P2: Waiting to receive from to P1
I am process 3 on node node18
P3: Waiting to receive from to P2
P1: Sending to to P2


 Test output within one node (This one is OK)
--
nielso@alamba:~/sandpit/pypar/source$ mpirun --hostfile /etc/mpihosts --host
node17 --npernode 4 a.out
Number of processes = 4
Test repeated 3 times for reliability
I am process 2 on node node17
P2: Waiting to receive from to P1
I am process 0 on node node17
Run 1 of 3
P0: Sending to P1
I am process 1 on node node17
P1: Waiting to receive from to P0
I am process 3 on node node17
P3: Waiting to receive from to P2
P1: Sending to to P2
P2: Sending to to P3
P1: Waiting to receive from to P0
P2: Waiting to receive from to P1
P3: Sending to to P0
P0: Received from to P3
Run 2 of 3
P0: Sending to P1
P3: Waiting to receive from to P2
P1: Sending to to P2
P2: Sending to to P3
P1: Waiting to receive from to P0
P3: Sending to to P0
P2: Waiting to receive from to P1
P0: Received from to P3
Run 3 of 3
P0: Sending to P1
P3: Waiting to receive from to P2
P1: Sending to to P2
P2: Sending to to P3
P1: Done
P2: Done
P3: Sending to to P0
P0: Received from to P3
P0: Done
P3: Done







 http://www.gstatic.com/codesite/ph/images/phosting.ico;>
 
 
 
 
 
 
 
 var codesite_token = "37184c5d629ad5b3864d886a93928b17";
 
 
 var CS_env = {"profileUrl":["/u/@UhFXRVJWBRJGVwN8/"],"token":"37184c5d629ad5b3864d886a93928b17","assetHostPath":"http://www.gstatic.com/codesite/ph","domainName":null,"assetVersionPath":"http://www.gstatic.com/codesite/ph/6432456761710229989","projectHomeUrl":"/p/pypar","relativeBaseUrl":"","projectName":"pypar","loggedInUserEmail":"ole.moller.niel...@gmail.com"};
 var _gaq = _gaq || [];
 _gaq.push(
 ['siteTracker._setAccount', 'UA-18071-1'],
 ['siteTracker._trackPageview']);
 
 _gaq.push(
 ['projectTracker._setAccount', 'UA-19946867-1'],
 ['projectTracker._trackPageview']);
 
 
 
 
 
 mpi_test.c - 
 pypar -
 
 
 Parallel programming for Python - Google Project Hosting
 
 
 
 
 
 
 http://www.gstatic.com/codesite/ph/6432456761710229989/css/core.css;>
 
 http://www.gstatic.com/codesite/ph/6432456761710229989/css/ph_detail.css; >
 
 
 http://www.gstatic.com/codesite/ph/6432456761710229989/css/d_sb.css; >
 
 
 

 
 .menuIcon.off { background: no-repeat url(http://www.gstatic.com/codesite/ph/images/dropdown_sprite.gif) 0 -42px }
 .menuIcon.on { background: no-repeat url(http://www.gstatic.com/codesite/ph/images/dropdown_sprite.gif) 0 -28px }
 .menuIcon.down { background: no-repeat url(http://www.gstatic.com/codesite/ph/images/dropdown_sprite.gif) 0 0; }
 
 
 
  tr.inline_comment {
 background: #fff;
 vertical-align: top;
 }
 div.draft, div.published {
 padding: .3em;
 border: 1px solid #999; 
 margin-bottom: .1em;
 font-family: arial, sans-serif;
 max-width: 60em;
 }
 div.draft {
 background: #ffa;
 } 
 div.published {
 background: #e5ecf9;
 }
 div.published .body, div.draft .body {
 padding: .5em .1em .1em .1em;
 max-width: 60em;
 white-space: pre-wrap;
 white-space: -moz-pre-wrap;
 white-space: -pre-wrap;
 white-space: -o-pre-wrap;
 word-wrap: break-word;
 font-size: 1em;
 }
 div.draft .actions {
 margin-left: 1em;
 font-size: 90%;
 }
 div.draft form {
 padding: .5em .5em .5em 0;
 }
 div.draft textarea, div.published textarea {
 width: 95%;
 height: 10em;
 font-family: arial, sans-serif;
 margin-bottom: .5em;
 }

 
 .nocursor,

Re: [OMPI users] MPI hangs on multiple nodes

Re: [OMPI users] MPI hangs on multiple nodes

Re: [OMPI users] MPI hangs on multiple nodes

Re: [OMPI users] MPI hangs on multiple nodes

[OMPI users] MPI hangs on multiple nodes

[OMPI users] MPI hangs on multiple nodes

Re: [OMPI users] MPI hangs on multiple nodes

[OMPI users] MPI hangs on multiple nodes

Re: [OMPI users] MPI hangs on multiple nodes

[OMPI users] MPI hangs on multiple nodes

10 matches

Site Navigation

Mail list logo

Footer information