schenksj opened a new issue, #27461:
URL: https://github.com/apache/datafusion/issues/27461
### Describe the bug
In DataFusion 55.2.0, grouped `MIN`/`MAX` over a `Float64` column return
wrong values for groups that contain `NaN` or `-inf`. The ungrouped aggregates
over the same rows are correct. This is the defect described in #24432 (and the
older #13415); it is fixed on `main` by #24433 and the fix is present in
`56.0.0-rc1`, but it is not in any 55.x release, including the `55.3.0-rc1` tag.
Could #24433 be backported to the 55.x line (55.3.0)? Downstream projects
pinned to 55.x get wrong grouped float results today.
### To Reproduce
`Cargo.toml`:
```toml
[dependencies]
datafusion = "=55.2.0"
tokio = { version = "1", features = ["rt-multi-thread", "macros"] }
```
`src/main.rs`:
```rust
use std::sync::Arc;
use datafusion::arrow::array::{Float64Array, RecordBatch};
use datafusion::arrow::datatypes::{DataType, Field, Schema};
use datafusion::datasource::MemTable;
use datafusion::prelude::*;
#[tokio::main]
async fn main() {
let nan = f64::NAN; let inf = f64::INFINITY;
let f = vec![0.0, -0.0, nan, nan, nan, nan, nan, inf, -inf, 1.5, -2.5,
0.25, 0.0, -0.0, 0.0, 0.25, 0.25];
let g = vec![nan, -2.5, nan, 0.0, inf, nan, 1.5, nan, 0.25, nan, -0.0,
-inf, nan, -2.5, inf, -inf, -inf];
let schema = Arc::new(Schema::new(vec![
Field::new("f", DataType::Float64, true),
Field::new("g", DataType::Float64, true),
]));
let b = RecordBatch::try_new(schema.clone(), vec![
Arc::new(Float64Array::from(f)), Arc::new(Float64Array::from(g)),
]).unwrap();
let ctx =
SessionContext::new_with_config(SessionConfig::new().with_target_partitions(4));
ctx.register_table("t", Arc::new(MemTable::try_new(schema,
vec![vec![b]]).unwrap())).unwrap();
for q in [
"SELECT f, min(g), max(g), count(*) FROM t GROUP BY f ORDER BY f",
"SELECT min(g), max(g) FROM t WHERE isnan(f)",
"SELECT min(g), max(g) FROM t WHERE f = 0.25",
] {
println!("{q}");
ctx.sql(q).await.unwrap().show().await.unwrap();
}
}
```
Output with `55.2.0` (the same with `target_partitions` = 1):
```
+------+----------+-------------------------+----------+
| f | min(t.g) | max(t.g) | count(*) |
+------+----------+-------------------------+----------+
| 0.0 | -2.5 | inf | 5 | <- group holds
NaN: max should be NaN
| 0.25 | -inf | -1.7976931348623157e308 | 3 | <- group holds
only -inf: max should be -inf
| NaN | 1.5 | 1.5 | 5 | <- NaN-keyed
group: should be min 0.0, max NaN
...
```
while the ungrouped queries over the same rows return `0.0 / NaN` (`WHERE
isnan(f)`) and `-inf / -inf` (`WHERE f = 0.25`).
### Expected behavior
Grouped and ungrouped `MIN`/`MAX` agree. With the same program built against
the `56.0.0-rc1` tag the grouped result is:
```
| 0.0 | -2.5 | NaN | 5 |
| 0.25 | -inf | -inf | 3 |
| NaN | 0.0 | NaN | 5 |
```
### Additional context
Found while building an aggregate pushdown for a DataFusion table provider
whose partial results are combined by DataFusion's final aggregate. Related:
#24432, #24433, #13415, #26138.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]