pearu opened a new pull request, #52547: URL: https://github.com/apache/arrow/pull/52547
### Rationale for this change Converting a `DictionaryArray` with `string`/`binary` values to a NumPy object array (`to_numpy`, `np.asarray`), or to pandas as the child of a nested type, decodes the dictionary into a dense array first. When the decoded data exceeds the 32-bit offset limit this fails with `Take operation overflowed binary array capacity` (a segfault or garbage before #50841), although the result, a NumPy object array, has no such limit. ### What changes are included in this PR? - `DecodingWouldOverflow` predicts from the dictionary offsets and the indices whether decoding a chunk would exceed the limit the `take` kernel enforces. Two cheap bounds settle the common case without reading the rows. - Only when it would overflow, the dictionary is left undecoded and `ObjectWriterVisitor::Visit(const DictionaryType&)` converts each dictionary value to a Python object once and references it per index (the `deduplicate_objects` idea, without the memo table since the dictionary already holds the unique values). - Every other conversion takes the unchanged decode path. Measured on 10⁶–10⁷ rows with dictionaries of 10²–10⁶ entries: time and memory are within noise of the current code. Not covered: map keys and items decode before reaching this check (`ConvertMap`), so a map with an overflowing dictionary child still raises. ### Are these changes tested? Yes: `test_dictionary_to_numpy_without_decoding` (plus null index, null dictionary value and `binary` values) and `test_list_of_dictionary_without_decoding` (the nested path, also with several chunks). The dense form in the tests is 2.1 GB, built from 16385 references to one 128 KiB value, so the tests themselves need under 1 MB. ### Are there any user-facing changes? Conversions that raised now succeed. In that regime, repeated values in the result refer to one Python object, as `to_pandas(deduplicate_objects=True)` already does. ### AI usage disclosure Developed with AI assistance (Claude Code): the approach was worked out and the implementation, tests and benchmarks were generated under my direction and reviewed line by line by me. I own and can debug these changes. * GitHub Issue: #50842 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
