xgbtxy opened a new issue, #27365:
URL: https://github.com/apache/datafusion/issues/27365
### Describe the bug
In `datafusion/optimizer/src/simplify_expressions/expr_simplifier.rs`
(verified on DataFusion `55.0.0` and `main` `HEAD` `e1c625fdf5`, lines
1151–1166), literal comparison conjunction simplification rewrites:
- `f = 0.0 AND f != -0.0` $\rightarrow$ `f = 0.0`
- `f = 0.0 AND f = -0.0` $\rightarrow$ `false`
- `f IN (0.0) AND f NOT IN (-0.0)` $\rightarrow$ `f = 0.0`
because distinctness between two literal `ScalarValue`s (`l1 != l2`) is
checked using Rust's `ScalarValue::PartialEq` (which uses totalOrder /
bit-aware comparison where `Float64(0.0) != Float64(-0.0)` is `true`), whereas
DataFusion/Arrow's SQL `=` and `!=` operators follow IEEE-754 numeric equality
where `0.0 = -0.0` is `true` and `0.0 != -0.0` is `false`.
As a result, disabling the logical optimizer
(`datafusion.optimizer.max_passes = 0`) vs. running with default optimizations
produces opposite query results.
### To Reproduce
```python
import datafusion
import pyarrow as pa
tbl = pa.table({
"id": pa.array([1, 2, 3], type=pa.int64()),
"f": pa.array([0.0, -0.0, 1.5], type=pa.float64()),
})
ctx_unopt = datafusion.SessionContext(
datafusion.SessionConfig().set("datafusion.optimizer.max_passes", "0")
)
ctx_opt = datafusion.SessionContext()
ctx_unopt.from_arrow(tbl, name="t")
ctx_opt.from_arrow(tbl, name="t")
for q in [
"SELECT id FROM t WHERE f = 0.0 AND f != -0.0 ORDER BY id",
"SELECT id FROM t WHERE f = 0.0 AND f = -0.0 ORDER BY id",
]:
r_unopt = [r["id"] for b in ctx_unopt.sql(q).collect() for r in
b.to_pylist()]
r_opt = [r["id"] for b in ctx_opt.sql(q).collect() for r in
b.to_pylist()]
print(q)
print(" max_passes=0:", r_unopt)
print(" default opt :", r_opt)
```
Output on DataFusion `55.0.0`:
```text
SELECT id FROM t WHERE f = 0.0 AND f != -0.0 ORDER BY id
max_passes=0: []
default opt : [1, 2]
SELECT id FROM t WHERE f = 0.0 AND f = -0.0 ORDER BY id
max_passes=0: [1, 2]
default opt : []
```
### Expected behavior
Under IEEE-754 floating-point equality (`0.0 = -0.0` is `true`):
- `SELECT id FROM t WHERE f = 0.0 AND f != -0.0` should return `[]`
(matching `max_passes=0`, PostgreSQL, DuckDB, and SQLite).
- `SELECT id FROM t WHERE f = 0.0 AND f = -0.0` should return `[1, 2]`.
### Additional context
In `expr_simplifier.rs` (around lines 1151–1166, introduced in #19731, and
similarly in `A = L1 AND A = L2 -> false`), comparing floating-point
`ScalarValue`s (`Float16`, `Float32`, `Float64`) for SQL `=` / `!=`
simplification needs numeric zero-sign-insensitive equality (`+0.0 == -0.0` and
`NaN` handling matching DataFusion's `eq` kernel) rather than
`ScalarValue::PartialEq`.
### Willingness to contribute
I would like to contribute a fix for this bug, but request guidance from the
DataFusion community
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
