sunchao opened a new issue, #5468:
URL: https://github.com/apache/datafusion-comet/issues/5468
### Describe the bug
Native `WindowGroupLimitExec` can drop rows that Spark retains when a scalar
`FLOAT` or `DOUBLE` ordering key contains different NaN representations or both
zero signs. This is a follow-up to #4870.
Spark considers all NaNs equal and greater than non-NaN values, and treats
`-0.0` and `+0.0` as equal. Arrow row encoding compares their raw total-order
representations instead. Consequently, native `RANK` / `DENSE_RANK` cutoffs can
split one Spark peer group. Native sorting also needs to agree: with a
secondary ordering key, sorting by raw NaN payload or zero sign can make equal
peers noncontiguous.
### Steps to reproduce
On Spark 3.5 or later, enable native execution and
`spark.comet.exec.localTableScan.enabled=true`, keeping
`spark.comet.exec.strictFloatingPoint=false`.
Build a local table with runtime NaNs, for example
`java.lang.Double.longBitsToDouble(0x7ff8000000000001L)` and
`java.lang.Double.longBitsToDouble(0xfff8000000000002L)`, plus finite values
and a partition containing both zero signs. Avoid a Parquet round trip that can
canonicalize NaN payloads. Filter a query of this shape to its top rank:
```sql
SELECT * FROM (
SELECT *, RANK() OVER (PARTITION BY p ORDER BY v DESC NULLS LAST) AS rnk
FROM floating_window_peers
) WHERE rnk <= 1
```
A regression fixture retaining eight rows in Spark returns only four in
unpatched Comet. This reproduces for both `FLOAT` and `DOUBLE` with native
Sort, Window, and WindowGroupLimit operators. Adding a secondary ordering key
also exposes the sorting inconsistency.
### Expected behavior
Scalar floating-point sort keys, window peer comparisons, and native
range-partition keys/boundaries should use consistent Spark-compatible
normalization. Only comparison keys should change; returned values must
preserve their original NaN payloads and zero signs.
### Additional context
The existing strict-floating-point fallback remains an opt-out.
Floating-point values nested in arrays or structs are a separate compatibility
limitation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
