[
https://issues.apache.org/jira/browse/SPARK-24723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16550182#comment-16550182
]
Saisai Shao commented on SPARK-24723:
-------------------------------------
Hi [~mengxr], I don't think YARN has such feature to configure password-less
SSH on all containers. YARN itself doesn't rely on SSH, and in our deployment
(Ambari), we don't have use password-less ssh.
{quote}And does container by default run sshd? If not, which process is
responsible for starting/terminating the daemon?
{quote}
If the container is is not dockerized, so it will share with system's sshd, it
is system's responsibility to start/terminate this daemon.
If the container is dockerized, I think the docker container should be
responsible for starting sshd (IIUC).
Maybe we should check if sshd is started before starting MPI job, if sshd is
not started, simply we cannot run MPI job no matter who is responsible for sshd
daemon.
[~leftnoteasy] might have some thoughts, since he is the originator of
mpich2-yarn.
> Discuss necessary info and access in barrier mode + YARN
> --------------------------------------------------------
>
> Key: SPARK-24723
> URL: https://issues.apache.org/jira/browse/SPARK-24723
> Project: Spark
> Issue Type: Story
> Components: ML, Spark Core
> Affects Versions: 3.0.0
> Reporter: Xiangrui Meng
> Assignee: Saisai Shao
> Priority: Major
>
> In barrier mode, to run hybrid distributed DL training jobs, we need to
> provide users sufficient info and access so they can set up a hybrid
> distributed training job, e.g., using MPI.
> This ticket limits the scope of discussion to Spark + YARN. There were some
> past attempts from the Hadoop community. So we should find someone with good
> knowledge to lead the discussion here.
>
> Requirements:
> * understand how to set up YARN to run MPI job as a YARN application
> * figure out how to do it with Spark w/ Barrier
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
