This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new 43ca0b929ab [SPARK-46299][DOCS] Make `spark.deploy.recovery*` docs up-to-date 43ca0b929ab is described below commit 43ca0b929ab3c2f10d1879e5df622195564f8885 Author: Dongjoon Hyun <dh...@apple.com> AuthorDate: Wed Dec 6 19:19:41 2023 -0800 [SPARK-46299][DOCS] Make `spark.deploy.recovery*` docs up-to-date ### What changes were proposed in this pull request? This PR aims to update `Spark Standalone` cluster recovery configurations. ### Why are the changes needed? We need to document - #44173 - #44129 - #44113 ![Screenshot 2023-12-06 at 7 15 24 PM](https://github.com/apache/spark/assets/9700541/04f0be6f-cdfb-4d87-b1b5-c4bf131f460a) ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manual review. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #44227 from dongjoon-hyun/SPARK-46299. Authored-by: Dongjoon Hyun <dh...@apple.com> Signed-off-by: Dongjoon Hyun <dh...@apple.com> --- docs/spark-standalone.md | 26 +++++++++++++++++++++++--- 1 file changed, 23 insertions(+), 3 deletions(-) diff --git a/docs/spark-standalone.md b/docs/spark-standalone.md index 7a89c8124bd..25d2fba47ce 100644 --- a/docs/spark-standalone.md +++ b/docs/spark-standalone.md @@ -735,18 +735,38 @@ In order to enable this recovery mode, you can set SPARK_DAEMON_JAVA_OPTS in spa <tr> <td><code>spark.deploy.recoveryMode</code></td> <td>NONE</td> - <td>The recovery mode setting to recover submitted Spark jobs with cluster mode when it failed and relaunches. - Set to FILESYSTEM to enable single-node recovery mode, ZOOKEEPER to use Zookeeper-based recovery mode, and + <td>The recovery mode setting to recover submitted Spark jobs with cluster mode when it failed and relaunches. Set to + FILESYSTEM to enable file-system-based single-node recovery mode, + ROCKSDB to enable RocksDB-based single-node recovery mode, + ZOOKEEPER to use Zookeeper-based recovery mode, and CUSTOM to provide a customer provider class via additional `spark.deploy.recoveryMode.factory` configuration. + NONE is the default value which disables this recovery mode. </td> <td>0.8.1</td> </tr> <tr> <td><code>spark.deploy.recoveryDirectory</code></td> <td>""</td> - <td>The directory in which Spark will store recovery state, accessible from the Master's perspective.</td> + <td>The directory in which Spark will store recovery state, accessible from the Master's perspective. + Note that the directory should be clearly manualy if <code>spark.deploy.recoveryMode</code>, + <code>spark.deploy.recoverySerializer</code>, or <code>spark.deploy.recoveryCompressionCodec</code> is changed. + </td> <td>0.8.1</td> </tr> + <tr> + <td><code>spark.deploy.recoverySerializer</code></td> + <td>JAVA</td> + <td>A serializer for writing/reading objects to/from persistence engines; JAVA (default) or KRYO. + Java serializer has been the default mode since Spark 0.8.1. + Kryo serializer is a new fast and compact mode from Spark 4.0.0.</td> + <td>4.0.0</td> + </tr> + <tr> + <td><code>spark.deploy.recoveryCompressionCodec</code></td> + <td>(none)</td> + <td>A compression codec for persistence engines. none (default), lz4, lzf, snappy, and zstd. Currently, only FILESYSTEM mode supports this configuration.</td> + <td>4.0.0</td> + </tr> <tr> <td><code>spark.deploy.recoveryMode.factory</code></td> <td>""</td> --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org