This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 286c231 [SPARK-33679][SQL][DOCS][FOLLOWUP] Enable
spark.sql.adaptive.enabled by default
286c231 is described below
commit 286c231c1eac26405cd0b08965185bc3fab3f578
Author: ulysses-you <ulyssesyo...@gmail.com>
AuthorDate: Mon Jul 12 18:49:46 2021 +0900
[SPARK-33679][SQL][DOCS][FOLLOWUP] Enable spark.sql.adaptive.enabled by
default
### What changes were proposed in this pull request?
Update AQE is `disabled` to `enabled` in sql-performance-tuning docs
### Why are the changes needed?
Make docs correct.
### Does this PR introduce _any_ user-facing change?
yes, docs changed.
### How was this patch tested?
Not need.
Closes #33295 from ulysses-you/enable-AQE.
Lead-authored-by: ulysses-you <ulyssesyo...@gmail.com>
Co-authored-by: Hyukjin Kwon <gurwls...@gmail.com>
Signed-off-by: Hyukjin Kwon <gurwls...@apache.org>
---
docs/sql-performance-tuning.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docs/sql-performance-tuning.md b/docs/sql-performance-tuning.md
index d595ba2..19799d9 100644
--- a/docs/sql-performance-tuning.md
+++ b/docs/sql-performance-tuning.md
@@ -234,7 +234,7 @@ The "REPARTITION_BY_RANGE" hint must have column names and
a partition number is
For more details please refer to the documentation of [Partitioning
Hints](sql-ref-syntax-qry-select-hints.html#partitioning-hints).
## Adaptive Query Execution
-Adaptive Query Execution (AQE) is an optimization technique in Spark SQL that
makes use of the runtime statistics to choose the most efficient query
execution plan. AQE is disabled by default. Spark SQL can use the umbrella
configuration of `spark.sql.adaptive.enabled` to control whether turn it
on/off. As of Spark 3.0, there are three major features in AQE, including
coalescing post-shuffle partitions, converting sort-merge join to broadcast
join, and skew join optimization.
+Adaptive Query Execution (AQE) is an optimization technique in Spark SQL that
makes use of the runtime statistics to choose the most efficient query
execution plan, which is enabled by default since Apache Spark 3.2.0. Spark SQL
can turn on and off AQE by `spark.sql.adaptive.enabled` as an umbrella
configuration. As of Spark 3.0, there are three major features in AQE:
including coalescing post-shuffle partitions, converting sort-merge join to
broadcast join, and skew join optimization.
### Coalescing Post Shuffle Partitions
This feature coalesces the post shuffle partitions based on the map output
statistics when both `spark.sql.adaptive.enabled` and
`spark.sql.adaptive.coalescePartitions.enabled` configurations are true. This
feature simplifies the tuning of shuffle partition number when running queries.
You do not need to set a proper shuffle partition number to fit your dataset.
Spark can pick the proper shuffle partition number at runtime once you set a
large enough initial number of shuffle partitions [...]
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
