[ https://issues.apache.org/jira/browse/SPARK-5388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14290847#comment-14290847 ]
Dale Richardson edited comment on SPARK-5388 at 1/26/15 5:30 AM: ----------------------------------------------------------------- Hi Andrew, I think the idea is well worth considering. In response to the requirement of making it easier for client<-> master communications to pass through restrictive fire-walling, have you considered just using Akka's REST gateway ( http://doc.akka.io/docs/akka/1.3.1/scala/http.html )? I also have a question if there is an intention for other entities (such as job servers) to communicate with the master at all? If so then the proposed gateway is semantically defined at a fairly low level (just RPC over JSON/HTTP). This is fine if the interface is not going to be exposed to anybody who is not a spark developer with detailed knowledge of spark internals. Did you use the term “REST” to simply mean RPC over JSON/HTTP? Creating a REST interface is more then a HTTP RPC gateway. If the interface is going to be exposed to 3rd parties (such as developers of Job servers and web notebooks etc) then there is a benefit to simplifying some of the exposed application semantics, and exposing an API that is more integrated with HTTP’s protocol semantics which most people are already familiar with - this is what a true REST interface does and if you are defining an endpoint for others to use it is a very powerful concept that allows other people to quickly grasp how to properly use the exposed interface. A rough sketch of a more “REST”ed version of the API would be: *Submit_driver_request* HTTP POST JSON body of request http://host:port/SparkMaster?SubmitDriver Responds with standard HTTP Response including allocated DRIVER_ID if driver submission ok, http error codes with spark specific error if not. *Get status of DRIVER* HTTP GET http://host:port/SparkMaster/Drivers/<DRIVER_ID> Responds with JSON body containing information on driver execution. If no record of driver_id, then http error code 404 (Not found) returned. *Kill Driver request* HTTP DELETE http://host:port/SparkMaster/Drivers/<DRIVER_ID> Responds with JSON body containing information on driver kill request, or http error code if an error occurs. I would be happy to prototype something like this up to test the concept out for you if you are looking for something more than just RPC over JSON/HTTP. was (Author: tigerquoll): Hi Andrew, I think the idea is well worth considering. In response to the requirement of making it easier for client<-> master communications to pass through restrictive fire-walling, have you considered just using Akka's REST gateway (http://doc.akka.io/docs/akka/1.3.1/scala/http.html)? I also have a question if there is an intention for other entities (such as job servers) to communicate with the master at all? If so then the proposed gateway is semantically defined at a fairly low level (just RPC over JSON/HTTP). This is fine if the interface is not going to be exposed to anybody who is not a spark developer with detailed knowledge of spark internals. Did you use the term “REST” to simply mean RPC over JSON/HTTP? Creating a REST interface is more then a HTTP RPC gateway. If the interface is going to be exposed to 3rd parties (such as developers of Job servers and web notebooks etc) then there is a benefit to simplifying some of the exposed application semantics, and exposing an API that is more integrated with HTTP’s protocol semantics which most people are already familiar with - this is what a true REST interface does and if you are defining an endpoint for others to use it is a very powerful concept that allows other people to quickly grasp how to properly use the exposed interface. A rough sketch of a more “REST”ed version of the API would be: *Submit_driver_request* HTTP POST JSON body of request http://host:port/SparkMaster?SubmitDriver Responds with standard HTTP Response including allocated DRIVER_ID if driver submission ok, http error codes with spark specific error if not. *Get status of DRIVER* HTTP GET http://host:port/SparkMaster/Drivers/<DRIVER_ID> Responds with JSON body containing information on driver execution. If no record of driver_id, then http error code 404 (Not found) returned. *Kill Driver request* HTTP DELETE http://host:port/SparkMaster/Drivers/<DRIVER_ID> Responds with JSON body containing information on driver kill request, or http error code if an error occurs. I would be happy to prototype something like this up to test the concept out for you if you are looking for something more than just RPC over JSON/HTTP. > Provide a stable application submission gateway > ----------------------------------------------- > > Key: SPARK-5388 > URL: https://issues.apache.org/jira/browse/SPARK-5388 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 1.2.0 > Reporter: Andrew Or > Assignee: Andrew Or > Priority: Blocker > Attachments: Stable Spark Standalone Submission.pdf > > > The existing submission gateway in standalone mode is not compatible across > Spark versions. If you have a newer version of Spark submitting to an older > version of the standalone Master, it is currently not guaranteed to work. The > goal is to provide a stable REST interface to replace this channel. > The first cut implementation will target standalone cluster mode because > there are very few messages exchanged. The design, however, will be general > enough to eventually support this for other cluster managers too. Note that > this is not necessarily required in YARN because we already use YARN's stable > interface to submit applications there. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org