sunchao opened a new issue, #26689:
URL: https://github.com/apache/datafusion/issues/26689
### Describe the bug
Two one-element Arrow lists can both represent `[NULL]` when one contains a
null dictionary key and the other contains a key pointing to a null dictionary
value. `GROUPS BETWEEN CURRENT ROW AND CURRENT ROW` counts one row in each
group. `rank()` and `dense_rank()` also split the peers when the planner
chooses `BoundedWindowAggExec`.
### To reproduce
Register this two-row batch in a Rust `SessionContext`.
```rust
use std::sync::Arc;
use datafusion::arrow::{
array::{ArrayRef, DictionaryArray, Int8Array, Int32Array, ListArray,
RecordBatch},
buffer::OffsetBuffer,
datatypes::{Field, Int8Type},
};
use datafusion::prelude::{SessionConfig, SessionContext};
let children: ArrayRef = Arc::new(DictionaryArray::<Int8Type>::try_new(
Int8Array::from(vec![None, Some(0)]),
Arc::new(Int32Array::from(vec![None])),
)?);
let key: ArrayRef = Arc::new(ListArray::new(
Arc::new(Field::new("item", children.data_type().clone(), true)),
OffsetBuffer::from_lengths([1, 1]), children, None,
));
let ctx =
SessionContext::new_with_config(SessionConfig::new().with_target_partitions(1));
ctx.register_batch("t", RecordBatch::try_from_iter([("k", key)])?)?;
```
Run these queries in that context.
```sql
SELECT count(*) OVER (ORDER BY k GROUPS BETWEEN CURRENT ROW AND CURRENT ROW)
FROM t;
SELECT rank() OVER w, dense_rank() OVER w FROM t WINDOW w AS (ORDER BY k);
SELECT rank() OVER w, dense_rank() OVER w, percent_rank() OVER w
FROM t WINDOW w AS (ORDER BY k);
```
| Query | Physical operator observed | Output for the two rows | Expected |
| --- | --- | --- | --- |
| GROUPS count | `BoundedWindowAggExec` (Sorted) | `1, 1` | `2, 2` |
| Rank and dense rank | `BoundedWindowAggExec` (Sorted) | `1, 2` for each
function | `1, 1` for each function |
| Rank and dense rank alongside percent rank | `WindowAggExec` | `1, 1` for
each rank function and `0.0, 0.0` for percent rank | Same |
Reproduced at `99100274dd` during review of #26067. GROUPS frame boundaries
and bounded rank compare extracted `ScalarValue`s. Batch rank uses Arrow
partition boundaries, which group the two rows together. [GROUPS
boundaries](https://github.com/apache/datafusion/blob/99100274dd139574854e40393474caac5c97e16b/datafusion/expr/src/window_state.rs#L710-L739)
· [bounded
rank](https://github.com/apache/datafusion/blob/99100274dd139574854e40393474caac5c97e16b/datafusion/functions-window/src/rank.rs#L276-L308)
· [batch
rank](https://github.com/apache/datafusion/blob/99100274dd139574854e40393474caac5c97e16b/datafusion/physical-expr/src/window/standard.rs#L144-L147)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
