[
https://issues.apache.org/jira/browse/SPARK-5388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14308260#comment-14308260
]
Marcelo Vanzin commented on SPARK-5388:
---------------------------------------
Hi [~andrewor14],
Thanks for updating the spec! This one looks much, much better. I think most of
my concerns have been addressed. Adherence to "RESTfulness" is not super
important since this is an internal API, although I really would suggest
picking a better name for the Scala package (e.g.
"org.apache.spark.deploy.proto" or something, instead of "rest").
A few questions:
- is the "action" field required? Since you have different URIs handling
different messages, it seems redundant now. And responses having an "action" is
kinda weird.
- what is the "protocolVersion" field in ErrorResponse? From the spec, it
sounds like the maximum protocol version supported by the server. If that's the
case, can the property be renamed to "maxProtocolVersion"?
- the message definitions use strings for all data, is that intentional? It
would feel more natural to have proper types, e.g.: "jars" : [ "one.jar",
"two.jar" ], driverCores: 8, superviseDriver: false.
- The spec says the server should report unknown fields back to the client.
There's nothing in the response type that supports that; is the server expected
to embed that information in the "message" field? Feels like it would be better
to have an explicit field for that.
- Is the "kill" endpoint protected in any way? Right now it seems like anyone
can post to that and kill a driver, if they know (or guess) the submission ID.
If there's no special protection, I'd say in the spec that the submission ID
should be, at least, cryptographically secure. At that point, as long as the
server has SSL enabled, it should be hard enough to kill a random driver.
> Provide a stable application submission gateway in standalone cluster mode
> --------------------------------------------------------------------------
>
> Key: SPARK-5388
> URL: https://issues.apache.org/jira/browse/SPARK-5388
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.2.0
> Reporter: Andrew Or
> Assignee: Andrew Or
> Priority: Blocker
> Attachments: stable-spark-submit-in-standalone-mode-2-4-15.pdf
>
>
> The existing submission gateway in standalone mode is not compatible across
> Spark versions. If you have a newer version of Spark submitting to an older
> version of the standalone Master, it is currently not guaranteed to work. The
> goal is to provide a stable REST interface to replace this channel.
> For more detail, please see the most recent design doc attached.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
