rdtr opened a new pull request, #13566:
URL: https://github.com/apache/gluten/pull/13566
## What changes are proposed in this pull request?
Since Spark 4.2 (SPARK-57298), `collect_set` treats `-0.0` and `0.0` as one
value and returns `0.0`, also when the value is nested in a struct, array or
map. Spark normalizes inside `CollectSet` (`convertToBufferElement`), so the
plan Gluten sees has no `NormalizeNaNAndZero`. On Gluten, `SELECT
collect_set(named_struct('a', v)) FROM VALUES (-0.0D), (0.0D) AS t(v)` returned
`[{-0.0}]` where Spark returns `[{0.0}]`.
On Spark 4.2, `CollectRewriteRule` now wraps the input of `VeloxCollectSet`
with Spark's `NormalizeFloatingNumbers.normalize`, which handles nested types.
`normalize` is `private[sql]`, so `FloatingNumbersNormalizer` in
`org.apache.spark.sql.expression` calls it. The Velox backend runs the
resulting `NormalizeNaNAndZero` with `normalize_nan_and_zero` (#13565). Spark
4.1 and earlier don't change.
This re-enables `SPARK-57298: collect_set normalizes NaN and -0.0 nested in
complex types`. `SPARK-57298: collect_set normalizes NaN and -0.0 for
floating-point types` stays excluded. Velox's Spark `collect_set` keeps
top-level NaNs distinct (facebookincubator/velox#12335, which matched Spark
before 4.2), so `collect_set` over `NaN, NaN` still returns `[NaN, NaN]`. That
needs a Velox change.
This builds on #13565, and only the last commit is new.
Part of #13440
## How was this patch tested?
Ran `GlutenDataFrameAggregateSuite` with `SPARK_TESTING=true`,
`SPARK_ANSI_SQL_MODE=false`, `-Pdelta` and `-Dspark.test.home` set, as in the
CI jobs, against a native build of the pinned Velox (`dft-2026_11_05`):
- Spark 4.2: 169 succeeded, 0 failed, including the nested SPARK-57298 test.
- Spark 3.5: 91 succeeded, 0 failed.
The plan for `collect_set(v)` on Spark 4.2 has `ProjectExecTransformer
[knownfloatingpointnormalized(normalizenanandzero(v)) AS _pre_74]` below the
partial `velox_collect_set`.
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]