This is an automated email from the ASF dual-hosted git repository.
agrove pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/main by this push:
new fe071e08 doc: Add Tuning Guide with shuffle configs (#443)
fe071e08 is described below
commit fe071e0833a76686acee4d6478cf50037bb8008c
Author: Liang-Chi Hsieh <[email protected]>
AuthorDate: Fri May 17 15:17:15 2024 -0700
doc: Add Tuning Guide with shuffle configs (#443)
---
docs/source/index.rst | 1 +
docs/source/user-guide/tuning.md | 60 ++++++++++++++++++++++++++++++++++++++++
2 files changed, 61 insertions(+)
diff --git a/docs/source/index.rst b/docs/source/index.rst
index 9066ce75..eb42950b 100644
--- a/docs/source/index.rst
+++ b/docs/source/index.rst
@@ -47,6 +47,7 @@ as a native runtime to achieve improvement in terms of query
efficiency and quer
Supported Data Types <user-guide/datatypes>
Configuration Settings <user-guide/configs>
Compatibility Guide <user-guide/compatibility>
+ Tuning Guide <user-guide/tuning>
.. _toc.contributor-guide-links:
.. toctree::
diff --git a/docs/source/user-guide/tuning.md b/docs/source/user-guide/tuning.md
new file mode 100644
index 00000000..01fa7bdb
--- /dev/null
+++ b/docs/source/user-guide/tuning.md
@@ -0,0 +1,60 @@
+<!---
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Tuning Guide
+
+Comet provides some tuning options to help you get the best performance from
your queries.
+
+
+## Shuffle
+
+Comet provides Comet shuffle features that can be used to improve the
performance of your queries.
+The following sections describe the different shuffle options available in
Comet.
+
+To enable Comet shuffle, set the following configuration in your Spark
configuration:
+
+```
+spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager
+spark.comet.exec.shuffle.enabled=true
+```
+
+`spark.shuffle.manager` is a Spark static configuration which cannot be
changed at runtime.
+It must be set before the Spark context is created. You can enable or disable
Comet shuffle
+at runtime by setting `spark.comet.exec.shuffle.enabled` to `true` or `false`.
+Once it is disabled, Comet will fallback to the default Spark shuffle manager.
+
+### Columnar Shuffle
+
+By default, once `spark.comet.exec.shuffle.enabled` is enabled, Comet uses
columnar shuffle
+to improve the performance of shuffle operations. Columnar shuffle supports
HashPartitioning,
+RoundRobinPartitioning, RangePartitioning and SinglePartitioning.
+
+Columnar shuffle can be disabled by setting
`spark.comet.columnar.shuffle.enabled` to `false`.
+
+### Native Shuffle
+
+Comet also provides a fully native shuffle implementation that can be used to
improve the performance.
+To enable native shuffle, just disable `spark.comet.columnar.shuffle.enabled`.
+
+Native shuffle only supports HashPartitioning and SinglePartitioning.
+
+
+
+
+
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
