ump45nose opened a new pull request, #11663: URL: https://github.com/apache/arrow-rs/pull/11663
# Which issue does this PR close? - Closes #11594. # Rationale for this change `zip` on two dictionary arrays panicked with `DictionaryKeyOverflowError` when the two dictionaries together had more values than the key type can address, even when the values were the same in both and a merged dictionary would fit. Whether it panicked also depended on the mask, because long fragmented masks already went through `interleave`, which merges dictionaries. # What changes are included in this PR? `zip_impl` now builds its `MutableArrayData` with `try_new`. When that returns `DictionaryKeyOverflowError`, it builds the result with `interleave` instead, which merges the dictionaries (and returns an error rather than panicking if the merged values still don't fit). `interleave_arrays` now takes the scalar flags so this fallback also works when one or both sides are scalars. Inputs that didn't overflow take the same path as before. # Are these changes tested? Yes. `test_zip_dictionaries_whose_combined_values_overflow_the_key_type` zips two `Int8` dictionaries of the same 100 values for array/array, array/scalar, scalar/array and scalar/scalar inputs, and compares the decoded values and the data type with the expected ones. It panics without the fix. # Are there any significant user-facing changes? No. AI disclosure: I used an AI coding assistant to write the fix, the test and this description, and I reviewed all of it. The part worth a reviewer's look is the fallback in `zip_impl`, which only triggers on `DictionaryKeyOverflowError`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
