zanmato1984 commented on issue #54885: URL: https://github.com/apache/arrow/issues/54885#issuecomment-6349543761
Confirmed: this is a signed 32-bit byte-offset overflow in the **Basic Hash Join** row encoder, rather than a limit on the size of an individual `large_string` value or input chunk. `large_string` / `large_binary` columns select Basic Hash Join because Swiss Join does not support their 64-bit array offsets ([implementation selection](https://github.com/apache/arrow/blob/14132604b3949ba1c25cc79fff2e1782b9efc413/cpp/src/arrow/acero/hash_join_node.cc#L761-L776)). Basic Join accumulates the build-side keys and non-key payloads in separate `RowEncoder` instances. In this encoder, both the cumulative `offsets_` and the `total_length` prefix sum are signed `int32_t`. Once the encoded buffer exceeds the signed 32-bit range, the overflowing total is passed to `bytes_.resize()`, explaining the `std::length_error` / `vector::_M_default_append` in the stack trace ([offset storage](https://github.com/apache/arrow/blob/14132604b3949ba1c25cc79fff2e1782b9efc413/cpp/src/arrow/compute/row/row_encoder_internal.h#L392-L399), [append calculation](https://github.com/apache/arrow/blob/14132604b3949ba1c25cc79fff2e1782b9efc413/cpp/src/arrow/compute/row/row_encoder_internal.cc#L358-L379)). The reported extra 9 bytes per non-null key are also explained by the encoding: one null-marker byte plus an eight-byte length, followed by the string bytes. Thus the example reaches `534866 * 5 * (794 + 9) = 2147486990` encoded bytes, slightly over 2 GiB, even though the raw string data is smaller. This is a cumulative build-side buffer limit, not a restriction specific to left anti joins. The `large_string` case in #33151 and the `large_binary` payload case in #36879 hit the same encoder limitation, but on the **payload** buffer rather than the key buffer. An equivalent C++ Acero regression also reproduces the build-payload overflow on unpatched upstream code. The relevant implementation is unchanged between the tested upstream revision `373d64c02cb2` and current main `14132604b394`. The ordinary `string` variant mentioned in #33151 is a separate Swiss Join / RowTable >4 GiB issue, addressed historically by #43389; that change did not affect this Basic Join encoder. Keeping this issue open as the consolidated tracker, since it has the most detailed boundary reproducer. The unresolved large-offset cases in #33151 and #36879 are being consolidated here as duplicates, **not** marked as fixed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
