This is an automated email from the ASF dual-hosted git repository.
uros-b pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new aac1a59dc08e [SPARK-57775][SQL][FOLLOWUP] Fix the grand-total behavior
change version to Spark 4.2+
aac1a59dc08e is described below
commit aac1a59dc08eb5c6dba68de82ac1ba640236bc13
Author: Uros Bojanic <[email protected]>
AuthorDate: Thu Jul 9 15:54:42 2026 +0200
[SPARK-57775][SQL][FOLLOWUP] Fix the grand-total behavior change version to
Spark 4.2+
### What changes were proposed in this pull request?
Follow-up to #56891 and #56891: aligning the SQL migration guide and SQL
config version to refer to 4.2 (rather than 4.3).
### Why are the changes needed?
The original correctness fix (master/4.x: #56891; 4.2 backport: #57134) is
landing with 4.2.0+.
### Does this PR introduce _any_ user-facing change?
No, docs only change.
### How was this patch tested?
N/A, docs only change.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #57139 from uros-b/SPARK-57775-migration-guide-4.2.
Lead-authored-by: Uros Bojanic <[email protected]>
Co-authored-by: Uros Bojanic <[email protected]>
Signed-off-by: Uros Bojanic <[email protected]>
(cherry picked from commit f10f996776e790d31ce567d708a7efbf820feff0)
Signed-off-by: Uros Bojanic <[email protected]>
---
docs/sql-migration-guide.md | 2 +-
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md
index a8d7682069a4..91fb7a52f670 100644
--- a/docs/sql-migration-guide.md
+++ b/docs/sql-migration-guide.md
@@ -27,7 +27,6 @@ license: |
- Since Spark 4.3, zero-length files are skipped during Parquet schema
inference instead of failing with a `FAILED_READ_FILE.CANNOT_READ_FILE_FOOTER`
error.
- Since Spark 4.3, the configuration key
`spark.sql.sources.v2.bucketing.allowJoinKeysSubsetOfPartitionKeys.enabled` has
been renamed to
`spark.sql.sources.v2.bucketing.allowKeysSubsetOfPartitionKeys.enabled` to
reflect that it now applies to storage-partitioned joins, aggregates, and
windows. The old key continues to work as an alias.
- Since Spark 4.3, the Spark Thrift Server rejects setting JVM system
properties through the `set:system:` session configuration overlay (for
example, in a JDBC connection string). To restore the previous behavior, set
`spark.sql.legacy.hive.thriftServer.allowSettingSystemProperties` to `true`.
-- Since Spark 4.3, `GROUP BY GROUPING SETS (())` (and the equivalent empty
`GROUP BY CUBE()` / `GROUP BY ROLLUP()`) is treated as a grand total and
returns one row over empty input, matching an aggregation with no `GROUP BY`
clause; previously it returned no rows. To restore the previous behavior, set
`spark.sql.analyzer.lowerEmptyGroupingSetToGlobalAggregate.enabled` to `false`.
## Upgrading from Spark SQL 4.1 to 4.2
@@ -42,6 +41,7 @@ license: |
- Since Spark 4.2, when a SQL UDF has a parameter whose name matches a
parameterless built-in function (`current_user`, `current_date`,
`current_time`, `current_timestamp`, `user`, `session_user`, `grouping__id`), a
bare reference to that name in the function body resolves to the built-in
function instead of the parameter, matching the documented name resolution
rules. Rename the parameter to avoid the collision, or set
`spark.sql.legacy.allowUdfParameterToShadowParameterlessFunction` to [...]
- Since Spark 4.2, `SET CATALOG <name>` resolves a bare (unquoted) name as a
session variable first, using the variable's value as the catalog name when
such a variable exists, and otherwise treats the name as a literal catalog
name. Use a string literal (`SET CATALOG 'name'`) to force it to be interpreted
literally.
- Since Spark 4.2, when an error occurs while collecting observed metrics,
`Observation.get` raises the underlying exception (for example,
`SparkRuntimeException` in Scala or `PySparkException` in Python) instead of
silently returning an empty result. Add error handling if your code relied on
receiving an empty result on failure.
+- Since Spark 4.2, `GROUP BY GROUPING SETS (())` (and the equivalent empty
`GROUP BY CUBE()` / `GROUP BY ROLLUP()`) is treated as a grand total and
returns one row over empty input, matching an aggregation with no `GROUP BY`
clause; previously it returned no rows. To restore the previous behavior, set
`spark.sql.analyzer.lowerEmptyGroupingSetToGlobalAggregate.enabled` to `false`.
## Upgrading from Spark SQL 4.0 to 4.1
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
index 6fe7a957a121..ca31519a7069 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
@@ -295,7 +295,7 @@ object SQLConf {
val LOWER_EMPTY_GROUPING_SET_TO_GLOBAL_AGGREGATE =
buildConf("spark.sql.analyzer.lowerEmptyGroupingSetToGlobalAggregate.enabled")
.internal()
- .version("4.3.0")
+ .version("4.2.0")
.doc(
"When true, a grand-total GROUP BY GROUPING SETS (()) (and the
equivalent empty " +
"CUBE() / ROLLUP()) is lowered to a global aggregate during analysis,
so it returns " +
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
