[
https://issues.apache.org/jira/browse/SPARK-5388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14292874#comment-14292874
]
Andrew Or commented on SPARK-5388:
----------------------------------
Hi Dale, thank you for your comments. Yes, in the design doc I used REST
roughly interchangeably with HTTP/JSON. But the goal is not to provide a
mechanism for other entities to communicate with the Master as you suggested;
it is simply to provide a stable mechanism for Spark to work across multiple
versions. For instance, you might have a long-running Master that outlives
multiple Spark versions, in which case we want to guarantee that newer versions
of Spark will still be able to submit to the long-running Master.
I think your proposal to make this more REST-like is potentially a great idea.
However, I find the alternative of simply putting the action in the JSON itself
easier to reason about. This also allows us to add other messages in the future
that are not strictly limited to the semantics of GET, POST, and DELETE. That
said, my proposal is also not set in stone yet so if there is a reason
compelling enough to change it then I will do so.
Also, a first-cut implementation of my design is now posted at:
https://github.com/apache/spark/pull/4216. Please take a look if you feel
inclined.
> Provide a stable application submission gateway
> -----------------------------------------------
>
> Key: SPARK-5388
> URL: https://issues.apache.org/jira/browse/SPARK-5388
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.2.0
> Reporter: Andrew Or
> Assignee: Andrew Or
> Priority: Blocker
> Attachments: Stable Spark Standalone Submission.pdf
>
>
> The existing submission gateway in standalone mode is not compatible across
> Spark versions. If you have a newer version of Spark submitting to an older
> version of the standalone Master, it is currently not guaranteed to work. The
> goal is to provide a stable REST interface to replace this channel.
> The first cut implementation will target standalone cluster mode because
> there are very few messages exchanged. The design, however, will be general
> enough to eventually support this for other cluster managers too. Note that
> this is not necessarily required in YARN because we already use YARN's stable
> interface to submit applications there.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
