This is an automated email from the ASF dual-hosted git repository. ulyssesyou pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git
The following commit(s) were added to refs/heads/main by this push: new 3bc5387c0 [GLUTEN-5142][CELEBORN] Remove Incubating of Celeborn from reference (#5143) 3bc5387c0 is described below commit 3bc5387c0e50f3e012f6ffad55dabbb7c52229c9 Author: Nicholas Jiang <programg...@163.com> AuthorDate: Wed Mar 27 13:44:40 2024 +0800 [GLUTEN-5142][CELEBORN] Remove Incubating of Celeborn from reference (#5143) --- docs/get-started/ClickHouse.md | 8 ++++---- docs/get-started/Velox.md | 8 ++++---- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/get-started/ClickHouse.md b/docs/get-started/ClickHouse.md index ad7183b90..4167af2ee 100644 --- a/docs/get-started/ClickHouse.md +++ b/docs/get-started/ClickHouse.md @@ -629,14 +629,14 @@ public read-only account:gluten/hN2xX3uQ4m ### Celeborn support -Gluten with clickhouse backend has not yet supportted [Celeborn](https://github.com/apache/incubator-celeborn) natively as remote shuffle service using columar shuffle. However, you can still use Celeborn with row shuffle, which means a ColumarBatch will be converted to a row during shuffle. +Gluten with clickhouse backend has not yet supportted [Celeborn](https://github.com/apache/celeborn) natively as remote shuffle service using columar shuffle. However, you can still use Celeborn with row shuffle, which means a ColumarBatch will be converted to a row during shuffle. Below introduction is used to enable this feature: -First refer to this URL(https://github.com/apache/incubator-celeborn) to setup a celeborn cluster. +First refer to this URL(https://github.com/apache/celeborn) to setup a celeborn cluster. Then add the Spark Celeborn Client packages to your Spark application's classpath(usually add them into `$SPARK_HOME/jars`). -- Celeborn: celeborn-client-spark-3-shaded_2.12-0.3.0-incubating.jar +- Celeborn: celeborn-client-spark-3-shaded_2.12-[celebornVersion].jar Currently to use Celeborn following configurations are required in `spark-defaults.conf` @@ -666,7 +666,7 @@ spark.sql.adaptive.localShuffleReader.enabled false spark.celeborn.storage.hdfs.dir hdfs://<namenode>/celeborn # If you want to use dynamic resource allocation, -# please refer to this URL (https://github.com/apache/incubator-celeborn/tree/main/assets/spark-patch) to apply the patch into your own Spark. +# please refer to this URL (https://github.com/apache/celeborn/tree/main/assets/spark-patch) to apply the patch into your own Spark. spark.dynamicAllocation.enabled false ``` diff --git a/docs/get-started/Velox.md b/docs/get-started/Velox.md index 7c3d77abc..1fabfc0fe 100644 --- a/docs/get-started/Velox.md +++ b/docs/get-started/Velox.md @@ -203,11 +203,11 @@ Currently there are several ways to asscess S3 in Spark. Please refer [Velox S3] ## Celeborn support -Gluten with velox backend supports [Celeborn](https://github.com/apache/incubator-celeborn) as remote shuffle service. Currently, the supported Celeborn versions are `0.3.x` and `0.4.0`. +Gluten with velox backend supports [Celeborn](https://github.com/apache/celeborn) as remote shuffle service. Currently, the supported Celeborn versions are `0.3.x` and `0.4.0`. Below introduction is used to enable this feature -First refer to this URL(https://github.com/apache/incubator-celeborn) to setup a celeborn cluster. +First refer to this URL(https://github.com/apache/celeborn) to setup a celeborn cluster. When compiling the Gluten Java module, it's required to enable `rss` profile, as follows: @@ -217,7 +217,7 @@ mvn clean package -Pbackends-velox -Pspark-3.3 -Prss -DskipTests Then add the Gluten and Spark Celeborn Client packages to your Spark application's classpath(usually add them into `$SPARK_HOME/jars`). -- Celeborn: celeborn-client-spark-3-shaded_2.12-0.3.0-incubating.jar +- Celeborn: celeborn-client-spark-3-shaded_2.12-[celebornVersion].jar - Gluten: gluten-velox-bundle-spark3.x_2.12-xx_xx_xx-SNAPSHOT.jar, gluten-thirdparty-lib-xx-xx.jar Currently to use Gluten following configurations are required in `spark-defaults.conf` @@ -248,7 +248,7 @@ spark.sql.adaptive.localShuffleReader.enabled false spark.celeborn.storage.hdfs.dir hdfs://<namenode>/celeborn # If you want to use dynamic resource allocation, -# please refer to this URL (https://github.com/apache/incubator-celeborn/tree/main/assets/spark-patch) to apply the patch into your own Spark. +# please refer to this URL (https://github.com/apache/celeborn/tree/main/assets/spark-patch) to apply the patch into your own Spark. spark.dynamicAllocation.enabled false ``` --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@gluten.apache.org For additional commands, e-mail: commits-h...@gluten.apache.org