Best guess I can offer is that they are blocking loopback on those networks - i.e., they are configured such that you can use them to connect to a remote machine, but not to a process on your local machine. I'll take a look at the connection logic and see if I can get it to failover to the loopback device in that case. I believe we disable use of the loopback if an active TCP network is available as we expect it to include loopback capability.
Meantime, you might want to talk to your IT folks and see if that is correct and intentional - and if so, why. On Tue, Dec 3, 2013 at 5:04 AM, Meredith, Karl <karl.mered...@fmglobal.com>wrote: > I disconnected for our corporate network (ethernet connection) and tried > running again: same result, it stalls. > > Then, I also disconnected from our local wifi network and tried running > again: it worked! > > bash-4.2$ mpirun -np 2 --mca btl sm,self hello_c > Hello, world, I am 0 of 2, (Open MPI v1.7.4a1, package: Open MPI > meredi...@meredithk-mac.corp.fmglobal.com<mailto: > meredi...@meredithk-mac.corp.fmglobal.com> Distribution, ident: > 1.7.4a1r29784, repo rev: r29784, Dec 02, 2013 (nightly snapshot tarball), > 173) > Hello, world, I am 1 of 2, (Open MPI v1.7.4a1, package: Open MPI > meredi...@meredithk-mac.corp.fmglobal.com<mailto: > meredi...@meredithk-mac.corp.fmglobal.com> Distribution, ident: > 1.7.4a1r29784, repo rev: r29784, Dec 02, 2013 (nightly snapshot tarball), > 173) > bash-4.2$ mpirun -np 2 hello_c > Hello, world, I am 0 of 2, (Open MPI v1.7.4a1, package: Open MPI > meredi...@meredithk-mac.corp.fmglobal.com<mailto: > meredi...@meredithk-mac.corp.fmglobal.com> Distribution, ident: > 1.7.4a1r29784, repo rev: r29784, Dec 02, 2013 (nightly snapshot tarball), > 173) > Hello, world, I am 1 of 2, (Open MPI v1.7.4a1, package: Open MPI > meredi...@meredithk-mac.corp.fmglobal.com<mailto: > meredi...@meredithk-mac.corp.fmglobal.com> Distribution, ident: > 1.7.4a1r29784, repo rev: r29784, Dec 02, 2013 (nightly snapshot tarball), > 173) > > Why? What would be causing the network to be interfering with mpirun? Do > you have any insight? > > Karl > > > > On Dec 3, 2013, at 7:54 AM, Ralph Castain <r...@open-mpi.org<mailto: > r...@open-mpi.org>> wrote: > > Hmmm...are you connected to a network, or at least have a network active, > when you do this? It looks a little like the system is trying to open a > port between the process and mpirun, but is failing to do so. > > > > On Tue, Dec 3, 2013 at 4:51 AM, Meredith, Karl <karl.mered...@fmglobal.com > <mailto:karl.mered...@fmglobal.com>> wrote: > Using openmpi-1.7.4, no macports, only apple compilers/tools: > > mpirun -np 2 --mca btl sm,self hello_c > > This hangs, also in MPI_Init(). > > Here’s the back trace from the debugger: > > bash-4.2$ lldb -p 4517 > Attaching to process with: > process attach -p 4517 > Process 4517 stopped > Executable module set to > "/Users/meredithk/tools/openmpi-1.7.4a1r29784/examples/hello_c". > Architecture set to: x86_64-apple-macosx. > (lldb) bt > * thread #1: tid = 0x57efb, 0x00007fff8c991a3a > libsystem_kernel.dylib`__semwait_signal + 10, queue = > 'com.apple.main-thread, stop reason = signal SIGSTOP > frame #0: 0x00007fff8c991a3a libsystem_kernel.dylib`__semwait_signal + > 10 > frame #1: 0x00007fff8ade4e60 libsystem_c.dylib`nanosleep + 200 > frame #2: 0x0000000108d668e3 > libopen-rte.6.dylib`orte_routed_base_register_sync(setup=true) + 2435 at > routed_base_fns.c:344 > frame #3: 0x000000010904e3a7 > mca_routed_binomial.so`init_routes(job=1294401537, ndat=0x0000000000000000) > + 2759 at routed_binomial.c:708 > frame #4: 0x0000000108d1b84d > libopen-rte.6.dylib`orte_ess_base_app_setup(db_restrict_local=true) + 2109 > at ess_base_std_app.c:233 > frame #5: 0x0000000108fbc442 mca_ess_env.so`rte_init + 418 at > ess_env_module.c:146 > frame #6: 0x0000000108cd6cfe > libopen-rte.6.dylib`orte_init(pargc=0x0000000000000000, > pargv=0x0000000000000000, flags=32) + 718 at orte_init.c:158 > frame #7: 0x0000000108a3b3c8 libmpi.1.dylib`ompi_mpi_init(argc=1, > argv=0x00007fff57200508, requested=0, provided=0x00007fff57200360) + 616 at > ompi_mpi_init.c:451 > frame #8: 0x0000000108a895a0 > libmpi.1.dylib`MPI_Init(argc=0x00007fff572004d0, argv=0x00007fff572004c8) + > 480 at init.c:84 > frame #9: 0x00000001089ffe4a hello_c`main(argc=1, > argv=0x00007fff57200508) + 58 at hello_c.c:18 > frame #10: 0x00007fff8d5df5fd libdyld.dylib`start + 1 > frame #11: 0x00007fff8d5df5fd libdyld.dylib`start + 1 > > > On Dec 2, 2013, at 2:11 PM, Jeff Squyres (jsquyres) <jsquy...@cisco.com > <mailto:jsquy...@cisco.com>> wrote: > > > Karl -- > > > > Can you force the use of just the shared memory transport -- i.e., > disable the TCP BTL? For example: > > > > mpirun -np 2 --mca btl sm,self hello_c > > > > If that also hangs, can you attach a debugger and see *where* it is > hanging inside MPI_Init()? (In OMPI, MPI::Init() simply invokes MPI_Init()) > > > > > > On Nov 27, 2013, at 2:56 PM, "Meredith, Karl" < > karl.mered...@fmglobal.com<mailto:karl.mered...@fmglobal.com>> wrote: > > > >> /opt/trunk/apple-only/bin/ompi_info --param oob tcp --level 9 > >> MCA oob: parameter "oob_tcp_verbose" (current value: > "0", data source: default, level: 9 dev/all, type: int) > >> Verbose level for the OOB tcp component > >> MCA oob: parameter "oob_tcp_peer_limit" (current value: > "-1", data source: default, level: 9 dev/all, type: int) > >> Maximum number of peer connections to > simultaneously maintain (-1 = infinite) > >> MCA oob: parameter "oob_tcp_peer_retries" (current > value: "60", data source: default, level: 9 dev/all, type: int) > >> Number of times to try shutting down a > connection before giving up > >> MCA oob: parameter "oob_tcp_debug" (current value: "0", > data source: default, level: 9 dev/all, type: int) > >> Enable (1) / disable (0) debugging output for > this component > >> MCA oob: parameter "oob_tcp_sndbuf" (current value: > "131072", data source: default, level: 9 dev/all, type: int) > >> TCP socket send buffering size (in bytes) > >> MCA oob: parameter "oob_tcp_rcvbuf" (current value: > "131072", data source: default, level: 9 dev/all, type: int) > >> TCP socket receive buffering size (in bytes) > >> MCA oob: parameter "oob_tcp_if_include" (current value: > "", data source: default, level: 9 dev/all, type: string, synonyms: > oob_tcp_include) > >> Comma-delimited list of devices and/or CIDR > notation of networks to use for Open MPI bootstrap communication (e.g., > "eth0,192.168.0.0/16<http://192.168.0.0/16>"). Mutually exclusive with > oob_tcp_if_exclude. > >> MCA oob: parameter "oob_tcp_if_exclude" (current value: > "", data source: default, level: 9 dev/all, type: string, synonyms: > oob_tcp_exclude) > >> Comma-delimited list of devices and/or CIDR > notation of networks to NOT use for Open MPI bootstrap communication -- all > devices not matching these specifications will be used (e.g., "eth0, > 192.168.0.0/16<http://192.168.0.0/16>"). If set to a non-default value, > it is mutually exclusive with oob_tcp_if_include. > >> MCA oob: parameter "oob_tcp_connect_sleep" (current > value: "1", data source: default, level: 9 dev/all, type: int) > >> Enable (1) / disable (0) random sleep for > connection wireup. > >> MCA oob: parameter "oob_tcp_listen_mode" (current value: > "event", data source: default, level: 9 dev/all, type: int) > >> Mode for HNP to accept incoming connections: > event, listen_thread. > >> Valid values: 0:"event", 1:"listen_thread" > >> MCA oob: parameter "oob_tcp_listen_thread_max_queue" > (current value: "10", data source: default, level: 9 dev/all, type: int) > >> High water mark for queued accepted socket list > size. Used only when listen_mode is listen_thread. > >> MCA oob: parameter "oob_tcp_listen_thread_wait_time" > (current value: "10", data source: default, level: 9 dev/all, type: int) > >> Time in milliseconds to wait before actively > checking for new connections when listen_mode is listen_thread. > >> MCA oob: parameter "oob_tcp_static_ports" (current > value: "", data source: default, level: 9 dev/all, type: string) > >> Static ports for daemons and procs (IPv4) > >> MCA oob: parameter "oob_tcp_dynamic_ports" (current > value: "", data source: default, level: 9 dev/all, type: string) > >> Range of ports to be dynamically used by > daemons and procs (IPv4) > >> MCA oob: parameter "oob_tcp_disable_family" (current > value: "none", data source: default, level: 9 dev/all, type: int) > >> Disable IPv4 (4) or IPv6 (6) > >> Valid values: 0:"none", 4:"IPv4", 6:"IPv6" > >> > >> /opt/trunk/apple-only/bin/ompi_info --param btl tcp --level 9 > >> MCA btl: parameter "btl_tcp_links" (current value: "1", > data source: default, level: 4 tuner/basic, type: unsigned) > >> MCA btl: parameter "btl_tcp_if_include" (current value: > "", data source: default, level: 1 user/basic, type: string) > >> Comma-delimited list of devices and/or CIDR > notation of networks to use for MPI communication (e.g., "eth0, > 192.168.0.0/16<http://192.168.0.0/16>"). Mutually exclusive with > btl_tcp_if_exclude. > >> MCA btl: parameter "btl_tcp_if_exclude" (current value: " > 127.0.0.1/8,sppp<http://127.0.0.1/8,sppp>", data source: default, level: > 1 user/basic, type: string) > >> Comma-delimited list of devices and/or CIDR > notation of networks to NOT use for MPI communication -- all devices not > matching these specifications will be used (e.g., "eth0,192.168.0.0/16< > http://192.168.0.0/16>"). If set to a non-default value, it is mutually > exclusive with btl_tcp_if_include. > >> MCA btl: parameter "btl_tcp_free_list_num" (current > value: "8", data source: default, level: 5 tuner/detail, type: int) > >> MCA btl: parameter "btl_tcp_free_list_max" (current > value: "-1", data source: default, level: 5 tuner/detail, type: int) > >> MCA btl: parameter "btl_tcp_free_list_inc" (current > value: "32", data source: default, level: 5 tuner/detail, type: int) > >> MCA btl: parameter "btl_tcp_sndbuf" (current value: > "131072", data source: default, level: 4 tuner/basic, type: int) > >> MCA btl: parameter "btl_tcp_rcvbuf" (current value: > "131072", data source: default, level: 4 tuner/basic, type: int) > >> MCA btl: parameter "btl_tcp_endpoint_cache" (current > value: "30720", data source: default, level: 4 tuner/basic, type: int) > >> The size of the internal cache for each TCP > connection. This cache is used to reduce the number of syscalls, by > replacing them with memcpy. Every read will read the expected data plus the > amount of the endpoint_cache > >> MCA btl: parameter "btl_tcp_use_nagle" (current value: > "0", data source: default, level: 4 tuner/basic, type: int) > >> Whether to use Nagle's algorithm or not (using > Nagle's algorithm may increase short message latency) > >> MCA btl: parameter "btl_tcp_port_min_v4" (current value: > "1024", data source: default, level: 2 user/detail, type: int) > >> The minimum port where the TCP BTL will try to > bind (default 1024) > >> MCA btl: parameter "btl_tcp_port_range_v4" (current > value: "64511", data source: default, level: 2 user/detail, type: int) > >> The number of ports where the TCP BTL will try > to bind (default 64511). This parameter together with the port min, define > a range of ports where Open MPI will open sockets. > >> MCA btl: parameter "btl_tcp_exclusivity" (current value: > "100", data source: default, level: 7 dev/basic, type: unsigned) > >> BTL exclusivity (must be >= 0) > >> MCA btl: parameter "btl_tcp_flags" (current value: > "314", data source: default, level: 5 tuner/detail, type: unsigned) > >> BTL bit flags (general flags: SEND=1, PUT=2, > GET=4, SEND_INPLACE=8, RDMA_MATCHED=64, HETEROGENEOUS_RDMA=256; flags only > used by the "dr" PML (ignored by others): ACK=16, CHECKSUM=32, > RDMA_COMPLETION=128; flags only used by the "bfo" PML (ignored by others): > FAILOVER_SUPPORT=512) > >> MCA btl: parameter "btl_tcp_rndv_eager_limit" (current > value: "65536", data source: default, level: 4 tuner/basic, type: size_t) > >> Size (in bytes, including header) of "phase 1" > fragment sent for all large messages (must be >= 0 and <= eager_limit) > >> MCA btl: parameter "btl_tcp_eager_limit" (current value: > "65536", data source: default, level: 4 tuner/basic, type: size_t) > >> Maximum size (in bytes, including header) of > "short" messages (must be >= 1). > >> MCA btl: parameter "btl_tcp_max_send_size" (current > value: "131072", data source: default, level: 4 tuner/basic, type: size_t) > >> Maximum size (in bytes) of a single "phase 2" > fragment of a long message when using the pipeline protocol (must be >= 1) > >> MCA btl: parameter "btl_tcp_rdma_pipeline_send_length" > (current value: "131072", data source: default, level: 4 tuner/basic, type: > size_t) > >> Length of the "phase 2" portion of a large > message (in bytes) when using the pipeline protocol. This part of the > message will be split into fragments of size max_send_size and sent using > send/receive semantics (must be >= 0; only relevant when the PUT flag is > set) > >> MCA btl: parameter "btl_tcp_rdma_pipeline_frag_size" > (current value: "2147483647", data source: default, level: 4 tuner/basic, > type: size_t) > >> Maximum size (in bytes) of a single "phase 3" > fragment from a long message when using the pipeline protocol. These > fragments will be sent using RDMA semantics (must be >= 1; only relevant > when the PUT flag is set) > >> MCA btl: parameter "btl_tcp_min_rdma_pipeline_size" > (current value: "196608", data source: default, level: 4 tuner/basic, type: > size_t) > >> Messages smaller than this size (in bytes) will > not use the RDMA pipeline protocol. Instead, they will be split into > fragments of max_send_size and sent using send/receive semantics (must be > >=0, and is automatically adjusted up to at least > (eager_limit+btl_rdma_pipeline_send_length); only relevant when the PUT > flag is set) > >> MCA btl: parameter "btl_tcp_bandwidth" (current value: > "100", data source: default, level: 5 tuner/detail, type: unsigned) > >> Approximate maximum bandwidth of interconnect > (0 = auto-detect value at run-time [not supported in all BTL modules], >= 1 > = bandwidth in Mbps) > >> MCA btl: parameter "btl_tcp_disable_family" (current > value: "0", data source: default, level: 2 user/detail, type: int) > >> MCA btl: parameter "btl_tcp_if_seq" (current value: "", > data source: default, level: 9 dev/all, type: string) > >> If specified, a comma-delimited list of TCP > interfaces. Interfaces will be assigned, one to each MPI process, in a > round-robin fashion on each server. For example, if the list is > "eth0,eth1" and four MPI processes are run on a single server, then local > ranks 0 and 2 will use eth0 and local ranks 1 and 3 will use eth1. > >> > >> > >> On Nov 27, 2013, at 2:41 PM, George Bosilca <bosi...@icl.utk.edu > <mailto:bosi...@icl.utk.edu><mailto:bosi...@icl.utk.edu<mailto: > bosi...@icl.utk.edu>>> wrote: > >> > >> ompi_info —param oob tcp —level 9 > >> ompi_info —param btl tcp —level 9 > >> > >> _______________________________________________ > >> users mailing list > >> us...@open-mpi.org<mailto:us...@open-mpi.org> > >> http://www.open-mpi.org/mailman/listinfo.cgi/users > > > > > > -- > > Jeff Squyres > > jsquy...@cisco.com<mailto:jsquy...@cisco.com> > > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > > > _______________________________________________ > > users mailing list > > us...@open-mpi.org<mailto:us...@open-mpi.org> > > http://www.open-mpi.org/mailman/listinfo.cgi/users > > _______________________________________________ > users mailing list > us...@open-mpi.org<mailto:us...@open-mpi.org> > http://www.open-mpi.org/mailman/listinfo.cgi/users > > _______________________________________________ > users mailing list > us...@open-mpi.org<mailto:us...@open-mpi.org> > http://www.open-mpi.org/mailman/listinfo.cgi/users > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users >
