[ https://issues.apache.org/jira/browse/SPARK-5388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14304010#comment-14304010 ]
Patrick Wendell commented on SPARK-5388: ---------------------------------------- The intention for this is really just to take single RPC that was using Akka and add a stable version of it that we are okay supporting long term. It doesn't preclude moving to avro or some other RPC as a general thing we use across all of Spark. However, that design choice was intentionally excluded from this decision given all the complexities you bring up. Doing some basic message dispatching on our own - there is only a small and very straightforward code related to this. Adopting Avro would be overkill for this. In the current implementation the client and server exchange Spark versions, so this is the basis of reasoning about version changes - maybe it wasn't in the design doc. In terms of evolvability, the way you do this is that you only add new functionality over time, and you never remove fields from messages. This is similar to the API contract of the history logs with the history server. So the idea is that newer clients would implement a super set of messages and fields as older ones. Adding v1 seems like a good idea in case this evolves into something public or more well specified over time. It would just be good to define precisely what it means to advance that version identifier. That all matters a lot more if we want it to be something others interact with. > Provide a stable application submission gateway in standalone cluster mode > -------------------------------------------------------------------------- > > Key: SPARK-5388 > URL: https://issues.apache.org/jira/browse/SPARK-5388 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 1.2.0 > Reporter: Andrew Or > Assignee: Andrew Or > Priority: Blocker > Attachments: Stable Spark Standalone Submission.pdf > > > The existing submission gateway in standalone mode is not compatible across > Spark versions. If you have a newer version of Spark submitting to an older > version of the standalone Master, it is currently not guaranteed to work. The > goal is to provide a stable REST interface to replace this channel. > The first cut implementation will target standalone cluster mode because > there are very few messages exchanged. The design, however, should be general > enough to potentially support this for other cluster managers too. Note that > this is not necessarily required in YARN because we already use YARN's stable > interface to submit applications there. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org