akshaytayal opened a new issue, #13439: URL: https://github.com/apache/gluten/issues/13439
Part of #13181 (sub-theme #13176). Found while re-enabling the Spark 4.2 UTs disabled in #13163. ### Problem Spark 4.2 (SPARK-54918) normalizes `-0.0`/NaN for `array_distinct`, `array_union`, `array_intersect` (and except/overlap). On Gluten the result keeps `-0.0`. 7 tests are disabled in `GlutenDataFrameFunctionsSuite` (`SPARK-54918: array_distinct / array_intersect / array_union normalizes -0.0 to +0.0` [+ `- literals`], `SPARK-54918: array set ops normalize -0.0 and NaN via VALUES inline table`). ### Root cause - Spark 4.2 does this in the optimizer: `NormalizeFloatingNumbers` rewrites e.g. `array_distinct(a)` to `array_distinct(knownfloatingpointnormalized(transform(a, x -> normalizenanandzero(x))))`. - Gluten offloads that plan, but `ExpressionConverter` converts `NormalizeNaNAndZero` as a pass-through (`ChildTransformer`) because Velox has no normalization function, so the normalization is silently dropped. Velox `array_distinct`/`array_intersect`/`array_union` then keep the first-seen `-0.0`. ### Proposed fix - Velox: register a Spark scalar `normalize_nan_and_zero(REAL|DOUBLE)` returning `0.0` for `-0.0` and the canonical NaN for any NaN (Spark `DOUBLE_NORMALIZER`/`FLOAT_NORMALIZER`). - Gluten (after the bump): map `NormalizeNaNAndZero` to it via `ExpressionMappings` instead of the pass-through; keep `KnownFloatingPointNormalized` as a pass-through. Measure join/grouping-key cost (Spark also inserts `NormalizeNaNAndZero` there). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
