Hi Rong, SSH is safe when properly implemented. That being said, some sites does not allow endusers to directly SSH into compute nodes because they do not want them to do anything without the resource manager knowing about it. What is your concern with SSH ?
You can run a resource manager (such as SLURM) in each container, so SSH is no more needed. Not sure that will address your concerns though. An other option is to run a PMIx server in each container. By default, Open MPI uses the rsh PLM in order to start a job. Basically, mpirun ssh orted on each compute nodes (this is a tree spawn also by default), and then orted fork&exec the MPI task. Yet an other option is to use three agents to alter this behavior mpirun --mca plm_rsh_agent /.../rsh --mca orte_launch_agent /.../launcher --mca orte_fork_agent /.../forker ... mpirun will /.../rsh /.../launcher instead of ssh orted, and then orted will fork&exec /.../forker a.out instead of fork&exec a.out with a bit of creativity, you should be able to plug docker via some adhoc agents. Cheers, Gilles On Fri, Mar 16, 2018 at 10:30 PM, Rong Ou <rong...@gmail.com> wrote: > Hi, > > I've implemented a Kubernetes operator for running OpenMPI jobs there. An > operator is just a Custom Resource Definition (CRD) for a new type of > mpi-aware batch job, and a custom controller that manages the states of > those jobs. Unfortunately the code can't be open sourced just yet, but I've > put together a demo that roughly matches the current design > (https://github.com/rongou/k8s-openmpi). > > The basic steps are: > > Create a pair of one-time-use ssh keys and store them as a Kubernetes > secret. > Launch a set of worker pods (in this context, a pod is just a container), > and start sshd on them. > Find the IP addresses of the worker pods (OpenMPI doesn't seem to like > Kubernetes' DNS names "foo.bar.my-namespace.svc.cluster.local"). > Pass the list of worker IPs to a launcher job that invokes `mpirun`. > > This seems to work pretty well. I was able to run the tensorflow benchmark > on 32 GPUs across 4 nodes, on both AWS and an on-prem cluster. > > Although I don't think it's an issue, some people are nervous about running > sshd in a container in a shared cluster environment. So I have two > questions: > > From your operational experience, how do you deal with security concerns > with regard to allowing ssh traffic? > If we want to do this without using ssh, what's the best option? Kubernetes > has a remote execution api similar to `docker exec`, would it be possible to > launch all the worker pods first, and exec some command on all of them? > Similar to what SLURM does? > > I'm pretty new to OpenMPI and haven't dug around the codebase much yet, so > I'm at a bit of a loss on how to proceed. Any pointer would be greatly > appreciated! > > Thanks, > > Rong > > _______________________________________________ > devel mailing list > devel@lists.open-mpi.org > https://lists.open-mpi.org/mailman/listinfo/devel _______________________________________________ devel mailing list devel@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/devel
