mbutrovich opened a new issue, #11620: URL: https://github.com/apache/arrow-rs/issues/11620
# `interleave` panics on list views of dictionaries whose combined values overflow the key type **Describe the bug** `interleave` panics on a `ListView` or `LargeListView` of dictionaries when the inputs' dictionaries together hold more values than the key type can address. When `interleave_list_view` picks the per-row copy, `interleave_list_view_copy` builds the child with [`MutableArrayData::new`](https://github.com/apache/arrow-rs/blob/069345b9b252342eff29ca0b1a85748625a5da40/arrow-select/src/interleave.rs#L761). That [concatenates the dictionaries](https://github.com/apache/arrow-rs/blob/069345b9b252342eff29ca0b1a85748625a5da40/arrow-data/src/transform/mod.rs#L655-L680) without merging them and [panics](https://github.com/apache/arrow-rs/blob/069345b9b252342eff29ca0b1a85748625a5da40/arrow-data/src/transform/mod.rs#L447) on the overflow. `List`, `LargeList`, `FixedSizeList` and `Map` interleave their child with [`interleave` itself](https://github.com/apache/arrow-rs/blob/069345b9b252342eff29ca0b1a85748625a5da40/arrow-select/src/interleave.rs#L502-L519), which merges dictionaries, so the same input as a `List` succeeds. The [concat strategy](https://github.com/apache/arrow-rs/blob/069345b9b252342eff29ca0b1a85748625a5da40/arrow-select/src/interleave.rs#L729-L733) for list views also succeeds. Whether a list view input panics therefore depends on which strategy its indices select. **To Reproduce** ```rust use std::sync::Arc; use arrow::array::{ Array, ArrayRef, DictionaryArray, Int8Array, ListArray, ListViewArray, StringArray, }; use arrow::buffer::{OffsetBuffer, ScalarBuffer}; use arrow::compute::interleave; use arrow::datatypes::{DataType, Field, Int8Type}; // A dictionary with 128 values, the most Int8 keys can address, and one key. // Each call allocates its own values, so two of them are not pointer-equal. fn dict() -> ArrayRef { let values = StringArray::from_iter_values((0..128).map(|i| i.to_string())); Arc::new( DictionaryArray::<Int8Type>::try_new(Int8Array::from(vec![0]), Arc::new(values)).unwrap(), ) } fn main() { let field = Arc::new(Field::new_list_field( DataType::Dictionary(Box::new(DataType::Int8), Box::new(DataType::Utf8)), true, )); let list = |v| ListArray::new(field.clone(), OffsetBuffer::from_lengths([1]), v, None); let (a, b) = (list(dict()), list(dict())); let out = interleave(&[&a, &b], &[(0, 0), (1, 0)]).unwrap(); println!( "List: {:?}", out.as_any().downcast_ref::<ListArray>().unwrap() ); // List: [["0"], ["0"]] let view = |v| { ListViewArray::new( field.clone(), ScalarBuffer::from(vec![0]), ScalarBuffer::from(vec![1]), v, None, ) }; let (a, b) = (view(dict()), view(dict())); let out = interleave(&[&a, &b], &[(0, 0), (1, 0)]); // panics: MutableArrayData::new is infallible: DictionaryKeyOverflowError println!("ListView: {out:?}"); } ``` With the indices `[(0, 0), (0, 0), (1, 0), (1, 0)]`, `interleave_list_view` picks the concat strategy and the list view call returns `[["0"], ["0"], ["0"], ["0"]]`. **Expected behavior** The list view call returns `[["0"], ["0"]]`, as the `List` call does. `interleave` returns a `Result` and its rustdoc documents no panic. Should `interleave_list_view_copy` build its child the way `interleave_list_like_child` does, through `interleave` with per-element indices? If the copy has to stay on `MutableArrayData` for speed, `MutableArrayData::try_new` would at least return `DictionaryKeyOverflowError` instead of panicking, as #10675 did for `interleave_fallback`. **Additional context** Property-based tests found this on `main` at 152f0d33c, and the reproduction panics on `main` at 069345b9b. #10674 reported the same panic message for view dictionaries whose distinct values do not fit the key type. #10675 fixed it by switching `interleave_fallback` and `concat_fallback` to `MutableArrayData::try_new`, but `interleave_list_view_copy` still calls `new`. The overflow here is not real: the values are `Utf8`, they deduplicate to one, and the `List` path returns them. The per-row copy came with #9558. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
