mbutrovich opened a new issue, #11607:
URL: https://github.com/apache/arrow-rs/issues/11607
### Describe the bug
When a `RunEndEncoded` array's values field is not nullable, `filter`,
`interleave`, and `RowConverter::convert_rows` return an array whose values
field is nullable, so the output type differs from the input type. `take` and
`concat` keep the input type. When the run-end encoded array is a struct child,
the changed type makes the parent invalid, and `BatchCoalescer` fails its
`debug_assert_eq!` on the column type.
### To Reproduce
```rust
use std::sync::Arc;
use arrow_array::types::Int32Type;
use arrow_array::{make_array, Array, ArrayRef, BooleanArray, Int32Array,
RunArray, StringArray};
use arrow_schema::{DataType, Field};
fn main() {
// `RunArray::try_new` declares the values field nullable, so the type is
// set through `ArrayData`
let run = RunArray::<Int32Type>::try_new(&Int32Array::from(vec![2, 4]),
&StringArray::from(vec!["a", "b"])).unwrap();
let dt = DataType::RunEndEncoded(
Arc::new(Field::new("run_ends", DataType::Int32, false)),
Arc::new(Field::new("values", DataType::Utf8, false)),
);
let a: ArrayRef =
make_array(run.to_data().into_builder().data_type(dt).build().unwrap());
println!("input: {}", a.data_type()); // RunEndEncoded(non-null
Int32, non-null Utf8)
let mask = BooleanArray::from(vec![true, false, true, true]);
let f = arrow_select::filter::filter(&a, &mask).unwrap();
println!("filter: {}", f.data_type()); // RunEndEncoded(non-null
Int32, Utf8)
let t = arrow_select::take::take(&a, &Int32Array::from(vec![3, 0]),
None).unwrap();
println!("take: {}", t.data_type()); // RunEndEncoded(non-null
Int32, non-null Utf8)
let c = arrow_select::concat::concat(&[a.as_ref(), a.as_ref()]).unwrap();
println!("concat: {}", c.data_type()); // RunEndEncoded(non-null
Int32, non-null Utf8)
let i = arrow_select::interleave::interleave(&[a.as_ref(), a.as_ref()],
&[(0, 1), (1, 3)]).unwrap();
println!("interleave: {}", i.data_type()); // RunEndEncoded(non-null
Int32, Utf8)
let conv =
arrow_row::RowConverter::new(vec![arrow_row::SortField::new(a.data_type().clone())]).unwrap();
let rows = conv.convert_columns(&[a.clone()]).unwrap();
let back = conv.convert_rows(&rows).unwrap();
println!("row format: {}", back[0].data_type()); //
RunEndEncoded(non-null Int32, Utf8)
}
```
### Expected behavior
The input type, `RunEndEncoded(non-null Int32, non-null Utf8)`, from every
kernel, as `take` and `concat` return it.
### Additional context
[`filter_run_end_array`](https://github.com/apache/arrow-rs/blob/dd240116c75a15a2a048a041c808d67f3f32a375/arrow-select/src/filter.rs#L689-L738),
[`interleave_run_end`](https://github.com/apache/arrow-rs/blob/dd240116c75a15a2a048a041c808d67f3f32a375/arrow-select/src/interleave.rs#L648-L710),
and the row format's
[`decode`](https://github.com/apache/arrow-rs/blob/dd240116c75a15a2a048a041c808d67f3f32a375/arrow-row/src/run.rs#L87-L154)
build their output with `RunArray::try_new`, which always declares the values
field
[nullable](https://github.com/apache/arrow-rs/blob/dd240116c75a15a2a048a041c808d67f3f32a375/arrow-array/src/array/run_array.rs#L155-L166).
`take_run` keeps the input's type by building its output with
[`run_array.data_type()`](https://github.com/apache/arrow-rs/blob/dd240116c75a15a2a048a041c808d67f3f32a375/arrow-select/src/take.rs#L1523).
#10994 made `take` respect a values field that is not nullable. Its discussion
checked `filter`, `interleave`, `concat`, `sort` and `z
ip` for introduced nulls, which they do not add. The loss here is in the
output's field, not in its values. Should these kernels keep the input's values
field the same way, or should `RunArray` have a constructor that takes it?
Found with property-based tests that build arrays in every physical layout
the Arrow format allows, on `main` at `72887ed9`. The code is unchanged on
`main` at `dd240116`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
