mbutrovich opened a new issue, #11834: URL: https://github.com/apache/arrow-rs/issues/11834
### Describe the bug `merge` and `merge_n` build their result with `MutableArrayData::new` ([merge.rs](https://github.com/apache/arrow-rs/blob/01a9917658fecd639991e92a5a5810be710b539c/arrow-select/src/merge.rs#L147), [merge.rs](https://github.com/apache/arrow-rs/blob/01a9917658fecd639991e92a5a5810be710b539c/arrow-select/src/merge.rs#L247)), which concatenates dictionaries without merging them and panics on key overflow ([mod.rs](https://github.com/apache/arrow-rs/blob/01a9917658fecd639991e92a5a5810be710b539c/arrow-data/src/transform/mod.rs#L436-L448)). They panic even when every input has the same values. ### To Reproduce ```rust use std::sync::Arc; use arrow_array::types::Int8Type; use arrow_array::{BooleanArray, DictionaryArray, Int8Array, StringArray}; use arrow_select::merge::{merge, merge_n}; fn main() { // 100 values each, the same in both, so a merged dictionary fits Int8 keys let values: Vec<String> = (0..100).map(|i| format!("v{i}")).collect(); let dict = || { DictionaryArray::<Int8Type>::try_new(Int8Array::from(vec![0]), Arc::new(StringArray::from(values.clone()))) .unwrap() }; let _ = merge(&BooleanArray::from(vec![true, false]), &dict(), &dict()); // panics: MutableArrayData::new is infallible: DictionaryKeyOverflowError let _ = merge_n(&[&dict(), &dict()], &[0usize, 1]); // panics: MutableArrayData::new is infallible: DictionaryKeyOverflowError } ``` ### Expected behavior A merged dictionary, as `concat` and `interleave` produce, or an error if it does not fit. ### Additional context This is the cause of #11594 for `zip` and #11620 for `interleave` of list views, which #11663 and #11629 fix one kernel at a time. Found with property-based tests that build arrays in every physical layout the Arrow format allows, on `main` at `01a991765`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
