I'm trying to use OpenMPI 1.10.4 to communicate between two Docker
containers running on two different physical machines. Docker doesn't have
much to do with my question (unless someone has a suggestion for a better
way to do what I'm trying to :o) )... each Docker container is running an
OpenSSH server which shows up as 172.17.0.1 on the physical hosts:
$ ifconfig docker0
docker0 Link encap:Ethernet HWaddr 02:42:8E:07:05:A0
inet addr:172.17.0.1 Bcast:0.0.0.0 Mask:255.255.0.0
inet6 addr: fe80::42:8eff:fe07:5a0/64 Scope:Link
The Docker container's ssh port is published on the physical host as port
32768.
The Docker container has a user 'mpirun' which I have public/private ssh
keys set up for.
Let's call the physical hosts host1 and host2; each host is running a
Docker container I'll refer to as docker1 and docker2 respectively. So,
this means I can...
1. ssh From host1 into docker1:
ssh mpirun@172.17.0.1 -i ssh/id_rsa -p 32768
2. Set up an ssh tunnel from inside docker1, through host2, into docker2,
on local port 4334 (ec2-user is the login to host2)
ssh -f -N -q -o "TCPKeepAlive yes" -o "ServerAliveInterval 60" -L 4334:
172.17.0.1:32768 -l ec2-user host2
3. Update my ~/.ssh/config file to name this host 'docker2':
StrictHostKeyChecking no
Host docker2
HostName 127.0.0.1
Port 4334
User mpirun
4. I can now do 'ssh docker2' and ssh into it without issues.
Here's where I get stuck. I'd read that OpenMPI's mpirun didn't support
ssh'ing on a non-standard port, so I thought I could just do step 3 above
and then list the hosts when I run mpirun from docker1:
mpirun --prefix /usr/local -n 2 -H localhost,docker2
/home/mpirun/mpi_hello_world
However, I get:
[3524ae84a26b:00197] [[55635,0],1] tcp_peer_send_blocking: send() to socket
9 failed: Broken pipe (32)
--------------------------------------------------------------------------
ORTE was unable to reliably start one or more daemons.
This usually is caused by:
* not finding the required libraries and/or binaries on
one or more nodes. Please check your PATH and LD_LIBRARY_PATH
settings, or configure OMPI with --enable-orterun-prefix-by-default
* lack of authority to execute on one or more specified nodes.
Please verify your allocation and authorities.
* the inability to write startup files into /tmp
(--tmpdir/orte_tmpdir_base).
Please check with your sys admin to determine the correct location to use.
* compilation of the orted with dynamic libraries when static are required
(e.g., on Cray). Please check your configure cmd line and consider using
one of the contrib/platform definitions for your system type.
* an inability to create a connection back to mpirun due to a
lack of common network interfaces and/or no route found between
them. Please check network connectivity (including firewalls
and network routing requirements).
--------------------------------------------------------------------------
I'm guessing that something's going wrong when docker2 tries to communicate
back to docker1. However, I'm not sure what additional tunneling to set up
to support this. My understanding of ssh tunnels is relatively basic... I
can of course create a tunnel on docker2 back to docker1 but I don't know
how ssh/mpi will "find" it. I've read a bit about reverse ssh tunneling
but it's not clear enough to me what this is doing to apply it here.
Any help is much appreciated!
-Adam
_______________________________________________
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
