andygrove opened a new issue, #6157:
URL: https://github.com/apache/datafusion-comet/issues/6157
### Describe the bug
`<=>`, `<`, `<=`, `>` and `>=` on arrays and structs with `FLOAT`/`DOUBLE`
leaves run natively and compare the nested values with Arrow's total order, so
`-0.0` and `0.0` are different values. Spark compares nested values with
`TypeUtils.getInterpretedOrdering`, which bottoms out in
`SQLOrderingUtil.compareDoubles`/`compareFloats`, where signed zeros are equal.
`CometExecRule.normalize` only wraps scalar float operands. #6073 fixes `=`,
`<>` and `IN` for nested operands, but leaves `<=>` and the ordering operators
on the raw comparator. #5507 covers `ORDER BY` and rank on nested float keys,
and this is the same gap in the comparison predicates.
### Steps to reproduce
On `main` at cccc08b7c, default Spark 4.1 profile, as a
`CometSqlFileTestSuite` fixture:
```sql
CREATE TABLE t (a ARRAY<DOUBLE>, b ARRAY<DOUBLE>, s STRUCT<v: DOUBLE>, u
STRUCT<v: DOUBLE>) USING parquet;
INSERT INTO t VALUES (
array(CAST('-0.0' AS DOUBLE)), array(CAST('0.0' AS DOUBLE)),
named_struct('v', CAST('-0.0' AS DOUBLE)), named_struct('v', CAST('0.0' AS
DOUBLE)));
SELECT a <=> b FROM t;
SELECT a < b, a >= b, s <=> u, s < u FROM t;
```
Both queries run fully native (`CometProject` over `CometNativeScan`) and
return different answers:
| Query | Spark | Comet |
|---|---|---|
| `a <=> b` | `true` | `false` |
| `a < b, a >= b, s <=> u, s < u` | `false, true, true, false` | `true,
false, false, true` |
`CAST('-0.0' AS DOUBLE)` is a string cast, so the sign survives. `CAST(-0.0
AS DOUBLE)` would produce `+0.0`.
### Expected behavior
The same answers as Spark: signed zeros inside arrays and structs compare
equal for every comparison operator.
### Additional context
Found while reviewing #6073. I only verified signed zero. NaN payloads and
signs probably diverge too, since `total_cmp` orders a negative NaN below every
other value while Spark treats every NaN as the largest value. The equivalence
in #6073's `nested_comparison.rs` could likely be extended into an ordering
comparator for these operators.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
