[
https://issues.apache.org/jira/browse/SPARK-24723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16550048#comment-16550048
]
Xiangrui Meng commented on SPARK-24723:
---------------------------------------
[~jerryshao] Does YARN have the feature that will by default configure
passwordless SSH on all containers (or per application)? If Spark generates the
key files in barrier mode on YARN, it might break this feature provided by
YARN. And does container by default run sshd? If not, which process is
responsible for starting/terminating the daemon?
> Discuss necessary info and access in barrier mode + YARN
> --------------------------------------------------------
>
> Key: SPARK-24723
> URL: https://issues.apache.org/jira/browse/SPARK-24723
> Project: Spark
> Issue Type: Story
> Components: ML, Spark Core
> Affects Versions: 3.0.0
> Reporter: Xiangrui Meng
> Priority: Major
>
> In barrier mode, to run hybrid distributed DL training jobs, we need to
> provide users sufficient info and access so they can set up a hybrid
> distributed training job, e.g., using MPI.
> This ticket limits the scope of discussion to Spark + YARN. There were some
> past attempts from the Hadoop community. So we should find someone with good
> knowledge to lead the discussion here.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
