akshaytayal opened a new issue, #13440: URL: https://github.com/apache/gluten/issues/13440
Part of #13181 (sub-theme #13176). Found while re-enabling the Spark 4.2 UTs disabled in #13163. ### Problem Spark 4.2 (SPARK-57298) normalizes NaN and `-0.0` inside `collect_set`. Gluten returns `[-0.0]` / `[[-0.0]]` where Spark returns `[0.0]` / `[[0.0]]`. Disabled tests in `GlutenDataFrameAggregateSuite`: `SPARK-57298: collect_set normalizes NaN and -0.0 for floating-point types`, `... nested in complex types`. ### Root cause Spark normalizes values in `CollectSet.update` (`convertToBufferElement` with `DOUBLE_/FLOAT_NORMALIZER` and a complex-type normalizer), not via an optimizer rewrite, so Gluten's plan has no normalization. Velox Spark `collect_set` treats `-0.0` and `0.0` as equal but stores the first-seen value. NaN dedup already matches. ### Proposed fix - Option A (recommended, reuses the `normalize_nan_and_zero` function from the SPARK-54918 follow-up): for Spark 4.2+ (shim flag), have `CollectRewriteRule` wrap the input, `VeloxCollectSet(NormalizeFloatingNumbers.normalize(child), ignoreNulls)`; this also covers nested struct/array/map. - Option B (Velox): normalize REAL/DOUBLE values in `CollectSetAggregate` before inserting into the accumulator (nested types are harder). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
