Re: [OMPI users] example program "ring" hangs when running across multiple hardware nodes (SOLVED)
Dear Gus, Thanks for your help - your clue solved my problem! The ultimate solution was to limit mpi communications to the local, unrouted subnet. I made this the default behavior of all users of my cluster by adding the following line to the bottom of my $prefix/etc/openmpi-mca-params.conf file btl_tcp_if_include = 10.0.0.0/8 Thanks again - what a relief! Jed On Fri, Jul 5, 2013, at 01:25 AM, Gustavo Correa wrote: > Hi Jed > > You could try to select only ethernet interface that match your node's IP > addresses, > which seems to be en2. > > The en1 interface seems to be an external IP. > Not sure about en3, but it is awkward that it has a > different IP than en2, but in the same subnet. > I wonder if this may be the reason for the program hanging. > > You may need to search all nodes ifconfig for a consistent set of > interfaces/IP addresses, > and tailor your mpiexec command line and your hostfile accordingly. > > Say, something like this: > > mpiexec -mca btl_tcp_if_include en2 -hostfile your_hostfile -np 43 > ./ring_c > > See this FAQ (actually, all of them are very informative): > http://www.open-mpi.org/faq/?category=tcp#tcp-selection > > I hope this helps, > Gus Correa
Re: [OMPI users] example program "ring" hangs when running across multiple hardware nodes
You also might want to check that you don't have any firewalls between those nodes. This is a typical cause of what you describe. On Jul 4, 2013, at 4:25 PM, Gustavo Correa wrote: > Hi Jed > > You could try to select only ethernet interface that match your node's IP > addresses, > which seems to be en2. > > The en1 interface seems to be an external IP. > Not sure about en3, but it is awkward that it has a > different IP than en2, but in the same subnet. > I wonder if this may be the reason for the program hanging. > > You may need to search all nodes ifconfig for a consistent set of > interfaces/IP addresses, > and tailor your mpiexec command line and your hostfile accordingly. > > Say, something like this: > > mpiexec -mca btl_tcp_if_include en2 -hostfile your_hostfile -np 43 ./ring_c > > See this FAQ (actually, all of them are very informative): > http://www.open-mpi.org/faq/?category=tcp#tcp-selection > > I hope this helps, > Gus Correa > > > > On Jul 4, 2013, at 6:37 PM, Jed O. Kaplan wrote: > >> Dear openmpi gurus, >> >> I am running openmpi 1.7.2 on a homogenous cluster of Apple XServes >> running OS X 10.6.8. My hardware nodes are connected through four >> gigabit ethernet connections; I have no infiniband or other high-speed >> interconnect. The problem I describe below is the same if I use openmpi >> 1.6.5. My openmpi installation is compiled with Intel icc and ifort. See >> the attached result of ompi_info --all for more details on my >> installation and runtime parameters, and other diagnostic information >> below >> >> My problem is that I noticed that inter-hardware communication hangs in >> one of my own programs; I thought this was the fault of my own bad >> programming, so I tried some of the example programs that are >> distributed with the openmpi source code. In the program "ring_*" using >> whichever of the APIs (c, cxx, fortran etc.), I have the same faulty >> behavior that I noticed in my own program: if I run the program on a >> single hardware node (with multiple processes) it works fine. As soon as >> I run the program across hardware nodes, it hangs. Below you will find >> an example of the program output and other diagnostic information. >> >> This problem has really frustrated me. Unfortunately I am not >> experienced enough with openmpi to get more into the debugging. >> >> Thank you in advance for any help you can give me! >> >> Jed Kaplan >> >> --- DETAILS OF MY PROBLEM --- >> >> -- this run works because it is only on one hardware node -- >> >> jkaplan@grkapsrv2:~/openmpi_examples > mpirun --prefix /usr/local >> --hostfile arvehosts.txt -np 3 ring_c >> Process 0 sending 10 to 1, tag 201 (3 processes in ring) >> Process 0 sent to 1 >> Process 0 decremented value: 9 >> Process 0 decremented value: 8 >> Process 0 decremented value: 7 >> Process 0 decremented value: 6 >> Process 0 decremented value: 5 >> Process 0 decremented value: 4 >> Process 0 decremented value: 3 >> Process 0 decremented value: 2 >> Process 0 decremented value: 1 >> Process 0 decremented value: 0 >> Process 0 exiting >> Process 1 exiting >> Process 2 exiting >> >> -- this run hangs when running over two hardware nodes -- >> >> jkaplan@grkapsrv2:~/openmpi_examples > mpirun --prefix /usr/local >> --hostfile arvehosts.txt -np 4 ring_c >> Process 0 sending 10 to 1, tag 201 (4 processes in ring) >> Process 0 sent to 1 >> Process 0 decremented value: 9 >> Process 0 decremented value: 8 >> ... hangs forever ... >> ^CKilled by signal 2. >> >> -- here is what my hostfile looks like -- >> >> jkaplan@grkapsrv2:~/openmpi_examples > cat arvehosts.txt >> #host file for ARVE group mac servers >> >> 10.0.0.21 slots=3 >> 10.0.0.31 slots=8 >> 10.0.0.41 slots=8 >> 10.0.0.51 slots=8 >> 10.0.0.61 slots=8 >> 10.0.0.71 slots=8 >> >> -- results of ifconfig - this looks pretty much the same on all of my >> servers, with different ip addresses of course -- >> >> jkaplan@grkapsrv2:~/openmpi_examples > ifconfig >> lo0: flags=8049 mtu 16384 >> inet6 ::1 prefixlen 128 >> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 >> inet 127.0.0.1 netmask 0xff00 >> gif0: flags=8010 mtu 1280 >> stf0: flags=0<> mtu 1280 >> en0: flags=8863 mtu 1500 >> ether 00:24:36:f3:dc:fc >> inet6 fe80::224:36ff:fef3:dcfc%en0 prefixlen 64 scopeid 0x4 >> inet 128.178.107.85 netmask 0xff00 broadcast 128.178.107.255 >> media: autoselect (1000baseT ) >> status: active >> en1: flags=8863 mtu 1500 >> ether 00:24:36:f3:dc:fa >> inet6 fe80::224:36ff:fef3:dcfa%en1 prefixlen 64 scopeid 0x5 >> inet 10.0.0.2 netmask 0xff00 broadcast 10.255.255.255 >> media: autoselect (1000baseT ) >> status: active >> en2: flags=8863 mtu 1500 >> ether 00:24:36:f5:ba:4e >> inet6 fe80::224:36ff:fef5:ba4e%en2 prefixlen 64 scopeid 0x6 >> inet 10.0.0.21 netmask 0xff00 broadcast 10.255.255.255 >> media: autoselect (1000baseT ) >> status: active >> en3:
Re: [OMPI users] example program "ring" hangs when running across multiple hardware nodes
Hi Jed You could try to select only ethernet interface that match your node's IP addresses, which seems to be en2. The en1 interface seems to be an external IP. Not sure about en3, but it is awkward that it has a different IP than en2, but in the same subnet. I wonder if this may be the reason for the program hanging. You may need to search all nodes ifconfig for a consistent set of interfaces/IP addresses, and tailor your mpiexec command line and your hostfile accordingly. Say, something like this: mpiexec -mca btl_tcp_if_include en2 -hostfile your_hostfile -np 43 ./ring_c See this FAQ (actually, all of them are very informative): http://www.open-mpi.org/faq/?category=tcp#tcp-selection I hope this helps, Gus Correa On Jul 4, 2013, at 6:37 PM, Jed O. Kaplan wrote: > Dear openmpi gurus, > > I am running openmpi 1.7.2 on a homogenous cluster of Apple XServes > running OS X 10.6.8. My hardware nodes are connected through four > gigabit ethernet connections; I have no infiniband or other high-speed > interconnect. The problem I describe below is the same if I use openmpi > 1.6.5. My openmpi installation is compiled with Intel icc and ifort. See > the attached result of ompi_info --all for more details on my > installation and runtime parameters, and other diagnostic information > below > > My problem is that I noticed that inter-hardware communication hangs in > one of my own programs; I thought this was the fault of my own bad > programming, so I tried some of the example programs that are > distributed with the openmpi source code. In the program "ring_*" using > whichever of the APIs (c, cxx, fortran etc.), I have the same faulty > behavior that I noticed in my own program: if I run the program on a > single hardware node (with multiple processes) it works fine. As soon as > I run the program across hardware nodes, it hangs. Below you will find > an example of the program output and other diagnostic information. > > This problem has really frustrated me. Unfortunately I am not > experienced enough with openmpi to get more into the debugging. > > Thank you in advance for any help you can give me! > > Jed Kaplan > > --- DETAILS OF MY PROBLEM --- > > -- this run works because it is only on one hardware node -- > > jkaplan@grkapsrv2:~/openmpi_examples > mpirun --prefix /usr/local > --hostfile arvehosts.txt -np 3 ring_c > Process 0 sending 10 to 1, tag 201 (3 processes in ring) > Process 0 sent to 1 > Process 0 decremented value: 9 > Process 0 decremented value: 8 > Process 0 decremented value: 7 > Process 0 decremented value: 6 > Process 0 decremented value: 5 > Process 0 decremented value: 4 > Process 0 decremented value: 3 > Process 0 decremented value: 2 > Process 0 decremented value: 1 > Process 0 decremented value: 0 > Process 0 exiting > Process 1 exiting > Process 2 exiting > > -- this run hangs when running over two hardware nodes -- > > jkaplan@grkapsrv2:~/openmpi_examples > mpirun --prefix /usr/local > --hostfile arvehosts.txt -np 4 ring_c > Process 0 sending 10 to 1, tag 201 (4 processes in ring) > Process 0 sent to 1 > Process 0 decremented value: 9 > Process 0 decremented value: 8 > ... hangs forever ... > ^CKilled by signal 2. > > -- here is what my hostfile looks like -- > > jkaplan@grkapsrv2:~/openmpi_examples > cat arvehosts.txt > #host file for ARVE group mac servers > > 10.0.0.21 slots=3 > 10.0.0.31 slots=8 > 10.0.0.41 slots=8 > 10.0.0.51 slots=8 > 10.0.0.61 slots=8 > 10.0.0.71 slots=8 > > -- results of ifconfig - this looks pretty much the same on all of my > servers, with different ip addresses of course -- > > jkaplan@grkapsrv2:~/openmpi_examples > ifconfig > lo0: flags=8049 mtu 16384 > inet6 ::1 prefixlen 128 > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 > inet 127.0.0.1 netmask 0xff00 > gif0: flags=8010 mtu 1280 > stf0: flags=0<> mtu 1280 > en0: flags=8863 mtu 1500 > ether 00:24:36:f3:dc:fc > inet6 fe80::224:36ff:fef3:dcfc%en0 prefixlen 64 scopeid 0x4 > inet 128.178.107.85 netmask 0xff00 broadcast 128.178.107.255 > media: autoselect (1000baseT ) > status: active > en1: flags=8863 mtu 1500 > ether 00:24:36:f3:dc:fa > inet6 fe80::224:36ff:fef3:dcfa%en1 prefixlen 64 scopeid 0x5 > inet 10.0.0.2 netmask 0xff00 broadcast 10.255.255.255 > media: autoselect (1000baseT ) > status: active > en2: flags=8863 mtu 1500 > ether 00:24:36:f5:ba:4e > inet6 fe80::224:36ff:fef5:ba4e%en2 prefixlen 64 scopeid 0x6 > inet 10.0.0.21 netmask 0xff00 broadcast 10.255.255.255 > media: autoselect (1000baseT ) > status: active > en3: flags=8863 mtu 1500 > ether 00:24:36:f5:ba:4f > inet6 fe80::224:36ff:fef5:ba4f%en3 prefixlen 64 scopeid 0x7 > inet 10.0.0.22 netmask 0xff00 broadcast 10.255.255.255 > media: autoselect (1000baseT ) > status: active > fw0: flags=8822 mtu 4078 > lladdr 04:1e:64:ff:fe:f8:aa:d2 >
[OMPI users] example program "ring" hangs when running across multiple hardware nodes
Dear openmpi gurus, I am running openmpi 1.7.2 on a homogenous cluster of Apple XServes running OS X 10.6.8. My hardware nodes are connected through four gigabit ethernet connections; I have no infiniband or other high-speed interconnect. The problem I describe below is the same if I use openmpi 1.6.5. My openmpi installation is compiled with Intel icc and ifort. See the attached result of ompi_info --all for more details on my installation and runtime parameters, and other diagnostic information below My problem is that I noticed that inter-hardware communication hangs in one of my own programs; I thought this was the fault of my own bad programming, so I tried some of the example programs that are distributed with the openmpi source code. In the program "ring_*" using whichever of the APIs (c, cxx, fortran etc.), I have the same faulty behavior that I noticed in my own program: if I run the program on a single hardware node (with multiple processes) it works fine. As soon as I run the program across hardware nodes, it hangs. Below you will find an example of the program output and other diagnostic information. This problem has really frustrated me. Unfortunately I am not experienced enough with openmpi to get more into the debugging. Thank you in advance for any help you can give me! Jed Kaplan --- DETAILS OF MY PROBLEM --- -- this run works because it is only on one hardware node -- jkaplan@grkapsrv2:~/openmpi_examples > mpirun --prefix /usr/local --hostfile arvehosts.txt -np 3 ring_c Process 0 sending 10 to 1, tag 201 (3 processes in ring) Process 0 sent to 1 Process 0 decremented value: 9 Process 0 decremented value: 8 Process 0 decremented value: 7 Process 0 decremented value: 6 Process 0 decremented value: 5 Process 0 decremented value: 4 Process 0 decremented value: 3 Process 0 decremented value: 2 Process 0 decremented value: 1 Process 0 decremented value: 0 Process 0 exiting Process 1 exiting Process 2 exiting -- this run hangs when running over two hardware nodes -- jkaplan@grkapsrv2:~/openmpi_examples > mpirun --prefix /usr/local --hostfile arvehosts.txt -np 4 ring_c Process 0 sending 10 to 1, tag 201 (4 processes in ring) Process 0 sent to 1 Process 0 decremented value: 9 Process 0 decremented value: 8 ... hangs forever ... ^CKilled by signal 2. -- here is what my hostfile looks like -- jkaplan@grkapsrv2:~/openmpi_examples > cat arvehosts.txt #host file for ARVE group mac servers 10.0.0.21 slots=3 10.0.0.31 slots=8 10.0.0.41 slots=8 10.0.0.51 slots=8 10.0.0.61 slots=8 10.0.0.71 slots=8 -- results of ifconfig - this looks pretty much the same on all of my servers, with different ip addresses of course -- jkaplan@grkapsrv2:~/openmpi_examples > ifconfig lo0: flags=8049 mtu 16384 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 inet 127.0.0.1 netmask 0xff00 gif0: flags=8010 mtu 1280 stf0: flags=0<> mtu 1280 en0: flags=8863 mtu 1500 ether 00:24:36:f3:dc:fc inet6 fe80::224:36ff:fef3:dcfc%en0 prefixlen 64 scopeid 0x4 inet 128.178.107.85 netmask 0xff00 broadcast 128.178.107.255 media: autoselect (1000baseT ) status: active en1: flags=8863 mtu 1500 ether 00:24:36:f3:dc:fa inet6 fe80::224:36ff:fef3:dcfa%en1 prefixlen 64 scopeid 0x5 inet 10.0.0.2 netmask 0xff00 broadcast 10.255.255.255 media: autoselect (1000baseT ) status: active en2: flags=8863 mtu 1500 ether 00:24:36:f5:ba:4e inet6 fe80::224:36ff:fef5:ba4e%en2 prefixlen 64 scopeid 0x6 inet 10.0.0.21 netmask 0xff00 broadcast 10.255.255.255 media: autoselect (1000baseT ) status: active en3: flags=8863 mtu 1500 ether 00:24:36:f5:ba:4f inet6 fe80::224:36ff:fef5:ba4f%en3 prefixlen 64 scopeid 0x7 inet 10.0.0.22 netmask 0xff00 broadcast 10.255.255.255 media: autoselect (1000baseT ) status: active fw0: flags=8822 mtu 4078 lladdr 04:1e:64:ff:fe:f8:aa:d2 media: autoselect status: inactive jkaplan@grkapsrv2:~/science_projects/mpitest/openmpi_examples > ompi_info --all Prefix: /usr/local Exec_prefix: /usr/local Bindir: /usr/local/bin Sbindir: /usr/local/sbin Libdir: /usr/local/lib Incdir: /usr/local/include Mandir: /usr/local/share/man Pkglibdir: /usr/local/lib/openmpi Libexecdir: /usr/local/libexec Datarootdir: /usr/local/share Datadir: /usr/local/share Sysconfdir: /usr/local/etc Sharedstatedir: /usr/local/com Localstatedir: /usr/local/var Infodir: /usr/local/share/info Pkgdatadir: /usr/local/share/openmpi Pkglibdir: /usr/local/lib/openmpi Pkgincludedir: /usr/local/include/openmpi Configured architecture: x86_64-apple-darw
