jiwen624 opened a new pull request, #56382:
URL: https://github.com/apache/spark/pull/56382
### What changes were proposed in this pull request?
This PR fixes `mode()` (and `pandas_mode()`) returning an incorrect result
when the input contains both -0.0 and 0.0. Mode/PandasMode accumulate value
frequencies in an `OpenHashMap[AnyRef, Long]` keyed by the raw boxed input
value. For FLOAT/DOUBLE, `java.lang.Double.equals/hashCode` are defined via
`doubleToLongBits`, which distinguishes -0.0 from 0.0. As a result the
frequency of a single SQL value is split across two buffer entries, and eval's
`maxBy(_._2)` can then pick a value whose true frequency is strictly smaller.
The fix normalizes the floating-point component of the buffer key at
update time, reusing the existing NormalizeFloatingNumbers helpers:
- top-level DOUBLE/FLOAT → DOUBLE_NORMALIZER / FLOAT_NORMALIZER;
- complex types containing float/double (struct/array/map) → an
UnsafeProjection built from NormalizeFloatingNumbers.normalize;
- all other types → unchanged (InternalRow.copyValue).
### Why are the changes needed?
It is a correctness bug. Spark treats -0.0 = 0.0 under SQL/GROUP BY
semantics everywhere else (e.g. GROUP BY, join keys, array_distinct,
collect_set), so mode() must do the same. Today it does not:
```SQL
SELECT mode(c) FROM VALUES
(0.0D),(0.0D),(-0.0D),(-0.0D),(9.0D),(9.0D),(9.0D) AS t(c);
-- returns 9.0, but the correct mode is 0.0
```
### Does this PR introduce _any_ user-facing change?
Yes, a bug fix.
### How was this patch tested?
New test case added
### Was this patch authored or co-authored using generative AI tooling?
Yes. Claude Code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
