[
https://issues.apache.org/jira/browse/SPARK-10288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14734161#comment-14734161
]
Saisai Shao commented on SPARK-10288:
-------------------------------------
Hi Marcelo,
Yes I agree with you that rest client is not only just replacing RPC to rest
API, it also requires some other things.
1. HDFS communication. Spark yarn-client will do some preparation works like
creating directories, uploading jars, etc. I think this HDFS communication
might be achieved by HDFS rest API (WebHDFS).
2. Configurations. As you mentioned configurations on the submission node maybe
different from cluster.
3. Security related problems, like token getting, SPNEGO authentication.
I think these things can be fixed through careful design and document, but the
problem is that it will separate from current yarn-client a lot, so still need
to think a lot.
> Add a rest client for Spark on Yarn
> -----------------------------------
>
> Key: SPARK-10288
> URL: https://issues.apache.org/jira/browse/SPARK-10288
> Project: Spark
> Issue Type: New Feature
> Components: YARN
> Reporter: Saisai Shao
>
> This is a proposal to add rest client for Spark on Yarn. Rest API offers a
> convenient addition to let user to submit application through rest client,
> people will easily achieve long haul submission, build their own submission
> gateway through rest client.
> Here is the design doc
> (https://docs.google.com/document/d/1m_P-4olXrp0tJ3kEOLZh1rwrjTfAat7P3fAVPR5GTmg/edit?usp=sharing).
> Currently I'm working on it, working branch is
> (https://github.com/jerryshao/apache-spark/tree/yarn-rest-support), the major
> part is already finished.
> Any comment is greatly appreciated, thanks a lot.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
