jiwen624 opened a new pull request, #58234:
URL: https://github.com/apache/spark/pull/58234
### What changes were proposed in this pull request?
Route `FloatType/DoubleType` pivot columns through `PivotFirst`'s TreeMap
index instead of the HashMap one, and make the TreeMap ordering null-safe so
null remains a usable pivot value.
### Why are the changes needed?
`groupBy("v").pivot("v").count()` on a float/double column returns an
all-null column for NaN: Scala's HashMap compares keys with ==, under which NaN
!= NaN, so the row is dropped. That contradicts Spark's documented NaN
semantics ("NaN = NaN returns true", "in aggregations, all NaN values are
grouped together"), and pivot's own standard path already returns the right
answer, so the PivotFirst fast path silently changes results. The TreeMap's
interpreted ordering implements those semantics.
The ordering is made null-safe so null stays usable as a pivot value, as it
was on the HashMap path; this also fixes an `INTERNAL_ERROR `for binary and
collated string pivot columns.
### Does this PR introduce _any_ user-facing change?
Yes. Pivoting on a float/double column now matches NaN values instead of
producing an all-null column. Pivoting on a column whose values include null no
longer fails with INTERNAL_ERROR for binary and collated string columns.
### How was this patch tested?
Added UT cases.
### Was this patch authored or co-authored using generative AI tooling?
Yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
