zanmato1984 commented on issue #36879: URL: https://github.com/apache/arrow/issues/36879#issuecomment-6349552762
Yes, this shares the root cause of the `large_string` case in #33151 and the more detailed report in #54885. Although the join key here is ordinary `binary`, the **`large_binary` non-key columns** select Basic Hash Join. It encodes all build-side payload columns together in a separate `RowEncoder`, whose cumulative byte offsets and total-length calculation are signed 32-bit. In this example, each blob is `4 * 25000 = 100000` bytes. Each of the two payload columns is individually below 2 GiB, but together their encoded payload is approximately `15000 * 2 * (100000 + 9) = 3000270000` bytes. The 9-byte overhead is the null marker plus 64-bit encoded length. This exceeds the encoder's signed 32-bit range and explains the invalid vector resize / `std::length_error`. The relevant [cumulative append calculation](https://github.com/apache/arrow/blob/14132604b3949ba1c25cc79fff2e1782b9efc413/cpp/src/arrow/compute/row/row_encoder_internal.cc#L358-L379) is still present on current upstream main. The right input is the build side, so reversing the inputs makes the small query table the build side and avoids this accumulation. No matching keys are needed to trigger the failure: the right-side payload is encoded while building the hash table, before probing. An equivalent C++ Acero build-payload regression reproduces this overflow on unpatched upstream code; the original Python script was not rerun. The 2024 Swiss Join >4 GiB row-offset change (#43389) concerns a different implementation and does not cover this Basic Join path. Closing as a duplicate of #54885, where the same unresolved RowEncoder limitation will be tracked for both keys and payloads. This is consolidation, not a claim that the bug is fixed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
