schenksj opened a new issue, #27461:
URL: https://github.com/apache/datafusion/issues/27461

   ### Describe the bug
   
   In DataFusion 55.2.0, grouped `MIN`/`MAX` over a `Float64` column return 
wrong values for groups that contain `NaN` or `-inf`. The ungrouped aggregates 
over the same rows are correct. This is the defect described in #24432 (and the 
older #13415); it is fixed on `main` by #24433 and the fix is present in 
`56.0.0-rc1`, but it is not in any 55.x release, including the `55.3.0-rc1` tag.
   
   Could #24433 be backported to the 55.x line (55.3.0)? Downstream projects 
pinned to 55.x get wrong grouped float results today.
   
   ### To Reproduce
   
   `Cargo.toml`:
   
   ```toml
   [dependencies]
   datafusion = "=55.2.0"
   tokio = { version = "1", features = ["rt-multi-thread", "macros"] }
   ```
   
   `src/main.rs`:
   
   ```rust
   use std::sync::Arc;
   use datafusion::arrow::array::{Float64Array, RecordBatch};
   use datafusion::arrow::datatypes::{DataType, Field, Schema};
   use datafusion::datasource::MemTable;
   use datafusion::prelude::*;
   
   #[tokio::main]
   async fn main() {
       let nan = f64::NAN; let inf = f64::INFINITY;
       let f = vec![0.0, -0.0, nan, nan, nan, nan, nan, inf, -inf, 1.5, -2.5, 
0.25, 0.0, -0.0, 0.0, 0.25, 0.25];
       let g = vec![nan, -2.5, nan, 0.0, inf, nan, 1.5, nan, 0.25, nan, -0.0, 
-inf, nan, -2.5, inf, -inf, -inf];
       let schema = Arc::new(Schema::new(vec![
           Field::new("f", DataType::Float64, true),
           Field::new("g", DataType::Float64, true),
       ]));
       let b = RecordBatch::try_new(schema.clone(), vec![
           Arc::new(Float64Array::from(f)), Arc::new(Float64Array::from(g)),
       ]).unwrap();
       let ctx = 
SessionContext::new_with_config(SessionConfig::new().with_target_partitions(4));
       ctx.register_table("t", Arc::new(MemTable::try_new(schema, 
vec![vec![b]]).unwrap())).unwrap();
       for q in [
           "SELECT f, min(g), max(g), count(*) FROM t GROUP BY f ORDER BY f",
           "SELECT min(g), max(g) FROM t WHERE isnan(f)",
           "SELECT min(g), max(g) FROM t WHERE f = 0.25",
       ] {
           println!("{q}");
           ctx.sql(q).await.unwrap().show().await.unwrap();
       }
   }
   ```
   
   Output with `55.2.0` (the same with `target_partitions` = 1):
   
   ```
   +------+----------+-------------------------+----------+
   | f    | min(t.g) | max(t.g)                | count(*) |
   +------+----------+-------------------------+----------+
   | 0.0  | -2.5     | inf                     | 5        |   <- group holds 
NaN: max should be NaN
   | 0.25 | -inf     | -1.7976931348623157e308 | 3        |   <- group holds 
only -inf: max should be -inf
   | NaN  | 1.5      | 1.5                     | 5        |   <- NaN-keyed 
group: should be min 0.0, max NaN
   ...
   ```
   
   while the ungrouped queries over the same rows return `0.0 / NaN` (`WHERE 
isnan(f)`) and `-inf / -inf` (`WHERE f = 0.25`).
   
   ### Expected behavior
   
   Grouped and ungrouped `MIN`/`MAX` agree. With the same program built against 
the `56.0.0-rc1` tag the grouped result is:
   
   ```
   | 0.0  | -2.5     | NaN      | 5        |
   | 0.25 | -inf     | -inf     | 3        |
   | NaN  | 0.0      | NaN      | 5        |
   ```
   
   ### Additional context
   
   Found while building an aggregate pushdown for a DataFusion table provider 
whose partial results are combined by DataFusion's final aggregate. Related: 
#24432, #24433, #13415, #26138.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to