rdtr opened a new pull request, #13566:
URL: https://github.com/apache/gluten/pull/13566

   ## What changes are proposed in this pull request?
   
   Since Spark 4.2 (SPARK-57298), `collect_set` treats `-0.0` and `0.0` as one 
value and returns `0.0`, also when the value is nested in a struct, array or 
map. Spark normalizes inside `CollectSet` (`convertToBufferElement`), so the 
plan Gluten sees has no `NormalizeNaNAndZero`. On Gluten, `SELECT 
collect_set(named_struct('a', v)) FROM VALUES (-0.0D), (0.0D) AS t(v)` returned 
`[{-0.0}]` where Spark returns `[{0.0}]`.
   
   On Spark 4.2, `CollectRewriteRule` now wraps the input of `VeloxCollectSet` 
with Spark's `NormalizeFloatingNumbers.normalize`, which handles nested types. 
`normalize` is `private[sql]`, so `FloatingNumbersNormalizer` in 
`org.apache.spark.sql.expression` calls it. The Velox backend runs the 
resulting `NormalizeNaNAndZero` with `normalize_nan_and_zero` (#13565). Spark 
4.1 and earlier don't change.
   
   This re-enables `SPARK-57298: collect_set normalizes NaN and -0.0 nested in 
complex types`. `SPARK-57298: collect_set normalizes NaN and -0.0 for 
floating-point types` stays excluded. Velox's Spark `collect_set` keeps 
top-level NaNs distinct (facebookincubator/velox#12335, which matched Spark 
before 4.2), so `collect_set` over `NaN, NaN` still returns `[NaN, NaN]`. That 
needs a Velox change.
   
   This builds on #13565, and only the last commit is new.
   
   Part of #13440
   
   ## How was this patch tested?
   
   Ran `GlutenDataFrameAggregateSuite` with `SPARK_TESTING=true`, 
`SPARK_ANSI_SQL_MODE=false`, `-Pdelta` and `-Dspark.test.home` set, as in the 
CI jobs, against a native build of the pinned Velox (`dft-2026_11_05`):
   
   - Spark 4.2: 169 succeeded, 0 failed, including the nested SPARK-57298 test.
   - Spark 3.5: 91 succeeded, 0 failed.
   
   The plan for `collect_set(v)` on Spark 4.2 has `ProjectExecTransformer 
[knownfloatingpointnormalized(normalizenanandzero(v)) AS _pre_74]` below the 
partial `velox_collect_set`.
   
   ## Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: Claude Code (Claude Opus 5.5)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

  • [PR] [GLUTEN-13440][VL] Norma... rdtr (via GitHub) via commits

Reply via email to