This is an automated email from the ASF dual-hosted git repository. srowen pushed a commit to branch branch-3.2 in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.2 by this push: new e78038b [SPARK-38606][DOC] Update document to make a good guide of multiple versions of the Spark Shuffle Service e78038b is described below commit e78038b1b1b9f8de75c6cf5df6613cb5fe8ac9cf Author: zhangxudong1 <zhangxudo...@xiaomi.com> AuthorDate: Mon Mar 21 13:48:20 2022 -0500 [SPARK-38606][DOC] Update document to make a good guide of multiple versions of the Spark Shuffle Service ### What changes were proposed in this pull request? Update document "Running multiple versions of the Spark Shuffle Service" to use colon when writing %s.classpath instead of commas. ### Why are the changes needed? User may be confused when they following the current document to deploy multi-versions Spark Shuffle Service on YARN. We have tried to run multiple versions of the Spark Shuffle Service according https://github.com/apache/spark/blob/master/docs/running-on-yarn.md but, it wont work. Then we solved it by using colon when writing %s.classpath instead of commas. Related discussing is in https://issues.apache.org/jira/browse/YARN-4577?focusedCommentId=17493624&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17493624 ### Does this PR introduce _any_ user-facing change? User document changes. ### How was this patch tested? ![image](https://user-images.githubusercontent.com/7348090/159159057-d85b5235-8979-43fb-a613-aa0edd2067e9.png) Closes #35914 from TonyDoen/SPARK-38606. Authored-by: zhangxudong1 <zhangxudo...@xiaomi.com> Signed-off-by: Sean Owen <sro...@gmail.com> (cherry picked from commit a876f005ecef77d94bb8048fa1ff0841b7f3607a) Signed-off-by: Sean Owen <sro...@gmail.com> --- docs/running-on-yarn.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/docs/running-on-yarn.md b/docs/running-on-yarn.md index 4a1dddf..77d0ed0 100644 --- a/docs/running-on-yarn.md +++ b/docs/running-on-yarn.md @@ -888,8 +888,14 @@ above. For example, you may have configuration like: ```properties yarn.nodemanager.aux-services = spark_shuffle_x,spark_shuffle_y - yarn.nodemanager.aux-services.spark_shuffle_x.classpath = /path/to/spark-x-yarn-shuffle.jar,/path/to/spark-x-config - yarn.nodemanager.aux-services.spark_shuffle_y.classpath = /path/to/spark-y-yarn-shuffle.jar,/path/to/spark-y-config + yarn.nodemanager.aux-services.spark_shuffle_x.classpath = /path/to/spark-x-path/fat.jar:/path/to/spark-x-config + yarn.nodemanager.aux-services.spark_shuffle_y.classpath = /path/to/spark-y-path/fat.jar:/path/to/spark-y-config +``` +Or +```properties + yarn.nodemanager.aux-services = spark_shuffle_x,spark_shuffle_y + yarn.nodemanager.aux-services.spark_shuffle_x.classpath = /path/to/spark-x-path/*:/path/to/spark-x-config + yarn.nodemanager.aux-services.spark_shuffle_y.classpath = /path/to/spark-y-path/*:/path/to/spark-y-config ``` The two `spark-*-config` directories each contain one file, `spark-shuffle-site.xml`. These are XML --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org