zanmato1984 opened a new pull request, #55170: URL: https://github.com/apache/arrow/pull/55170
### Rationale for this change Basic hash joins store encoded build-side keys and payloads in contiguous RowEncoder buffers. Signed 32-bit offsets and byte counts overflow when the accumulated encoded data exceeds 2 GiB, even when the individual input arrays are valid. This can cause a vector length exception and terminate the calling Python process. For large_string keys, each encoded value includes a one-byte null marker and an eight-byte length prefix, explaining why the failure can occur before the raw string data reaches 2 GiB. ### What changes are included in this PR? - Use 64-bit byte offsets and lengths throughout RowEncoder, including cumulative append sizes and the row length returned by encoded_row(). - Widen shared KeyEncoder length counters and remove narrowing conversions for large scalar and array values. - Update basic Grouper batch offsets, persistent unique-key offsets, and encoded key lengths consistently. - Add opt-in large-memory regressions for encoder length calculation and appends, Grouper buffers, and hash-join keys and payloads. Row IDs, group IDs, and per-column encoded length formats are unchanged. ### Are these changes tested? Added six opt-in large-memory regression tests covering encoder length calculation and cumulative appends, basic Grouper batch and unique-key buffers, and hash-join keys and payloads. ### Are there any user-facing changes? Basic hash joins using large_string or large_binary columns can accumulate more than 2 GiB of encoded build-side keys or payloads without overflowing their cumulative byte offsets. The memory cost is in the offset arrays: each offset grows from 4 to 8 bytes. A RowEncoder storing N rows has N + 1 offsets, so their element storage increases by 4 × (N + 1) bytes—approximately **3.8 MiB per million rows, per encoder**, excluding spare vector capacity and allocator overhead. Separate key and payload encoders each incur this cost. This doubles offset-array storage, not total join memory. Basic Grouper likewise uses four additional bytes per persistent unique-key offset and per temporary batch offset. The encoded key/value bytes and per-column length prefixes are unchanged. There are no public API changes, and existing row-count and group-count limits remain unchanged. **This PR contains a "Critical Fix".** It addresses a crash on valid input when encoded build-side data exceeds the signed 32-bit offset range. ### Was AI used for this PR? The author conducted a thorough review with AI assistance. **PR code and description written by:** - [ ] Human - [x] AI **Reviewed before submission by:** - [x] Human - [x] AI - [ ] Not reviewed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
