spicysharkco opened a new pull request, #11629: URL: https://github.com/apache/arrow-rs/pull/11629
# Which issue does this PR close? - Closes #11620 # Rationale for this change `interleave` on a `ListView` or `LargeListView` of dictionaries panics with `MutableArrayData::new is infallible: DictionaryKeyOverflowError` when the inputs' dictionaries together hold more values than the key type can address, even if the selected rows reference few distinct values. The same input as a `List` succeeds, because `List` interleaves its child through `interleave`, which merges dictionary values. Which behaviour a list view gets depends on whether its indices select the per-row copy or the concat strategy. # What changes are included in this PR? `interleave_list_view_copy` builds its child with `MutableArrayData::try_new` instead of `new`. If that returns `DictionaryKeyOverflowError`, the child is built with `interleave_list_like_child`, as for the other list types. This also covers a dictionary nested in a struct child. Children without dictionaries still use the per-row copy. Dictionaries whose selected values cannot be merged return `DictionaryKeyOverflowError` instead of panicking. # Are these changes tested? Yes, unit tests in `arrow-select/src/interleave.rs`. On `main`, these four panic with `MutableArrayData::new is infallible: DictionaryKeyOverflowError` (`arrow-data/src/transform/mod.rs:447`) and pass with this change: - `test_list_view_dictionary_overflow_copy` and `test_large_list_view_dictionary_overflow_copy`: the case from the issue, two Int8 dictionaries of 128 values, indices `[(0, 0), (1, 0)]`. - `test_interleave_list_view_struct_dictionary_overflow`: the same with the dictionary inside a struct child. - `test_interleave_list_view_dictionary_overflow_returns_err`: two UInt8 dictionaries of 200 distinct values each now return `DictionaryKeyOverflowError`. `test_list_view_dictionary_overflow_concat` and `test_large_list_view_dictionary_overflow_concat` use indices that select the concat strategy; they pass on `main` and guard that path. # Are there any significant user-facing changes? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
