mbutrovich opened a new issue, #11594:
URL: https://github.com/apache/arrow-rs/issues/11594
### Describe the bug
`zip` on two dictionary arrays panics with `DictionaryKeyOverflowError` when
the two dictionaries together have more values than the key type can address.
It panics even when the values are the same in both, so a merged dictionary
would fit.
### To Reproduce
```rust
use std::sync::Arc;
use arrow_array::types::Int8Type;
use arrow_array::{BooleanArray, DictionaryArray, Int8Array, StringArray};
use arrow_select::zip::zip;
fn main() {
// 100 values each, the same in both, so a merged dictionary fits Int8
keys
let values: Vec<String> = (0..100).map(|i| format!("v{i}")).collect();
let dict = ||
DictionaryArray::<Int8Type>::try_new(Int8Array::from(vec![0, 1]),
Arc::new(StringArray::from(values.clone()))).unwrap();
let _ = zip(&BooleanArray::from(vec![true, false]), &dict(), &dict());
// panics: MutableArrayData::new is infallible:
DictionaryKeyOverflowError
}
```
### Expected behavior
A result whose dictionary holds the merged values, as `concat` and
`interleave` produce. If the merged values do not fit, an error rather than a
panic.
### Additional context
For two arrays, `zip` builds the result with `MutableArrayData::new`
([zip.rs](https://github.com/apache/arrow-rs/blob/72887ed91c80002bb7ec755ce1e964bab07bebea/arrow-select/src/zip.rs#L195)),
which concatenates the dictionaries without merging and panics on key
overflow, as its docs state
([mod.rs](https://github.com/apache/arrow-rs/blob/72887ed91c80002bb7ec755ce1e964bab07bebea/arrow-data/src/transform/mod.rs#L430-L448)).
`zip`'s docs do not mention a panic. For masks of 1024 rows or more with many
runs, `zip` uses `interleave` instead
([zip.rs](https://github.com/apache/arrow-rs/blob/72887ed91c80002bb7ec755ce1e964bab07bebea/arrow-select/src/zip.rs#L191-L193)),
so the same inputs can succeed or panic depending on the mask.
Found with property-based tests that build arrays in every physical layout
the Arrow format allows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
