This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new a694a8a0be5 [SPARK-46095][DOCS] Document `REST API` for Spark Standalone Cluster a694a8a0be5 is described below commit a694a8a0be540e5d60d7f462e0761c4ba3b8b3e6 Author: Dongjoon Hyun <dh...@apple.com> AuthorDate: Fri Nov 24 17:31:22 2023 -0800 [SPARK-46095][DOCS] Document `REST API` for Spark Standalone Cluster ### What changes were proposed in this pull request? This PR aims to document `REST API` for Spark Standalone Cluster. ### Why are the changes needed? To help the users to understand Apache Spark features. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manual review. `REST API` Section is added newly. **AFTER** <img width="704" alt="Screenshot 2023-11-24 at 4 13 53 PM" src="https://github.com/apache/spark/assets/9700541/a4e09d94-d216-4629-8b37-9d350365a428"> ### Was this patch authored or co-authored using generative AI tooling? No. Closes #44007 from dongjoon-hyun/SPARK-46095. Authored-by: Dongjoon Hyun <dh...@apple.com> Signed-off-by: Dongjoon Hyun <dh...@apple.com> --- docs/spark-standalone.md | 80 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 80 insertions(+) diff --git a/docs/spark-standalone.md b/docs/spark-standalone.md index ce739cb90b5..2ab68d2a804 100644 --- a/docs/spark-standalone.md +++ b/docs/spark-standalone.md @@ -518,6 +518,8 @@ Spark applications supports the following configuration properties specific to s # Launching Spark Applications +## Spark Protocol + The [`spark-submit` script](submitting-applications.html) provides the most straightforward way to submit a compiled Spark application to the cluster. For standalone clusters, Spark currently supports two deploy modes. In `client` mode, the driver is launched in the same process as the @@ -540,6 +542,84 @@ failing repeatedly, you may do so through: You can find the driver ID through the standalone Master web UI at `http://<master url>:8080`. +## REST API + +If `spark.master.rest.enabled` is enabled, Spark master provides additional REST API +via <code>http://[host:port]/[version]/submissions/[action]</code> where +<code>host</code> is the master host, and +<code>port</code> is the port number specified by `spark.master.rest.port` (default: 6066), and +<code>version</code> is a protocol version, <code>v1</code> as of today, and +<code>action</code> is one of the following supported actions. + +<table class="table table-striped"> + <thead><tr><th style="width:21%">Command</th><th>Description</th><th>HTTP METHOD</th><th>Since Version</th></tr></thead> + <tr> + <td><code>create</code></td> + <td>Create a Spark driver via <code>cluster</code> mode.</td> + <td>POST</td> + <td>1.3.0</td> + </tr> + <tr> + <td><code>kill</code></td> + <td>Kill a single Spark driver.</td> + <td>POST</td> + <td>1.3.0</td> + </tr> + <tr> + <td><code>killall</code></td> + <td>Kill all running Spark drivers.</td> + <td>POST</td> + <td>4.0.0</td> + </tr> + <tr> + <td><code>status</code></td> + <td>Check the status of a Spark job.</td> + <td>GET</td> + <td>1.3.0</td> + </tr> + <tr> + <td><code>clear</code></td> + <td>Clear the completed drivers and applications.</td> + <td>POST</td> + <td>4.0.0</td> + </tr> +</table> + +The following is a <code>curl</code> CLI command example with the `pi.py` and REST API. + +```bash +$ curl -XPOST http://IP:PORT/v1/submissions/create \ +--header "Content-Type:application/json;charset=UTF-8" \ +--data '{ + "appResource": "", + "sparkProperties": { + "spark.master": "spark://master:7077", + "spark.app.name": "Spark Pi", + "spark.driver.memory": "1g", + "spark.driver.cores": "1", + "spark.jars": "" + }, + "clientSparkVersion": "", + "mainClass": "org.apache.spark.deploy.SparkSubmit", + "environmentVariables": { }, + "action": "CreateSubmissionRequest", + "appArgs": [ "/opt/spark/examples/src/main/python/pi.py", "10" ] +}' +``` + +The following is the response from the REST API for the above <code>create</code> request. + +```bash +{ + "action" : "CreateSubmissionResponse", + "message" : "Driver successfully submitted as driver-20231124153531-0000", + "serverSparkVersion" : "4.0.0", + "submissionId" : "driver-20231124153531-0000", + "success" : true +} +``` + + # Resource Scheduling The standalone cluster mode currently only supports a simple FIFO scheduler across applications. --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org