This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 6b3a88195e30 [SPARK-48329][SQL] Enable
`spark.sql.sources.v2.bucketing.pushPartValues.enabled` by default
6b3a88195e30 is described below
commit 6b3a88195e30027b74166d7729c232cd7ddba83b
Author: Szehon Ho <szehon.apa...@gmail.com>
AuthorDate: Tue May 21 10:00:14 2024 -0700
[SPARK-48329][SQL] Enable
`spark.sql.sources.v2.bucketing.pushPartValues.enabled` by default
### What changes were proposed in this pull request?
This PR aims to enable
`spark.sql.sources.v2.bucketing.pushPartValues.enabled` by default for Apache
Spark 4.0.0 while keeping `spark.sql.sources.v2.bucketing.enabled` is `false`.
### Why are the changes needed?
`spark.sql.sources.v2.bucketing.pushPartValues.enabled` was added at Apache
Spark 3.4.0 and has been used as one of the datasource v2 bucketing feature.
This PR will help the datasource v2 bucketing users use this feature more
easily.
Note that this change is technically no-op for the default users because
`spark.sql.sources.v2.bucketing.enabled` is `false` still.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Pass the CIs.
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #46673 from szehon-ho/default_pushpart.
Lead-authored-by: Szehon Ho <szehon.apa...@gmail.com>
Co-authored-by: chesterxu <cheste...@tencent.com>
Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
docs/sql-migration-guide.md | 1 +
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala | 2 +-
2 files changed, 2 insertions(+), 1 deletion(-)
diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md
index 98075d019585..6e400ab93711 100644
--- a/docs/sql-migration-guide.md
+++ b/docs/sql-migration-guide.md
@@ -57,6 +57,7 @@ license: |
- Since Spark 4.0, A bug falsely allowing `!` instead of `NOT` when `!` is not
a prefix operator has been fixed. Clauses such as `expr ! IN (...)`, `expr !
BETWEEN ...`, or `col ! NULL` now raise syntax errors. To restore the previous
behavior, set `spark.sql.legacy.bangEqualsNot` to `true`.
- Since Spark 4.0, By default views tolerate column type changes in the query
and compensate with casts. To restore the previous behavior, allowing up-casts
only, set `spark.sql.legacy.viewSchemaCompensation` to `false`.
- Since Spark 4.0, Views allow control over how they react to underlying query
changes. By default views tolerate column type changes in the query and
compensate with casts. To disable thsi feature set
`spark.sql.legacy.viewSchemaBindingMode` to `false`. This also removes the
clause from `DESCRIBE EXTENDED` and `SHOW CREATE TABLE`.
+- Since Spark 4.0, The Storage-Partitioned Join feature flag
`spark.sql.sources.v2.bucketing.pushPartValues.enabled` is set to `true`. To
restore the previous behavior, set
`spark.sql.sources.v2.bucketing.pushPartValues.enabled` to `false`.
## Upgrading from Spark SQL 3.5.1 to 3.5.2
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
index 87b32ca0b9b5..9c4236679f3a 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
@@ -1569,7 +1569,7 @@ object SQLConf {
"side. This could help to eliminate unnecessary shuffles")
.version("3.4.0")
.booleanConf
- .createWithDefault(false)
+ .createWithDefault(true)
val V2_BUCKETING_PARTIALLY_CLUSTERED_DISTRIBUTION_ENABLED =
buildConf("spark.sql.sources.v2.bucketing.partiallyClusteredDistribution.enabled")
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
