zanmato1984 commented on issue #33151: URL: https://github.com/apache/arrow/issues/33151#issuecomment-6349551958
There are two different implementation paths in this report: 1. **`large_string`**: this selects Basic Hash Join. Its build-side payload `RowEncoder` accumulates the text column using signed 32-bit byte offsets and a signed 32-bit total length. The accumulated encoded payload exceeds that range near 2 GiB, leading to the invalid vector resize discussed in the earlier comments. The same root cause is described in #54885 for build-side **keys**, and in #36879 for build-side **payloads**. The 2024 Swiss Join change does not cover this encoder, and this part remains unresolved on current upstream main. 2. **Ordinary `string`**: this selects Swiss Join on little-endian platforms. That path used a different row table with unsigned 32-bit total byte offsets, imposing a 4 GiB limit. This limitation was removed by #43389, merged on 2024-08-19 as `5e68513d62b0d216e916de6a1ad2db04f5d1a7bf`. I checked the second path with an equivalent C++ Acero LEFT OUTER join using `int32` keys, ordinary `string` payloads and 4,295,032,832 raw build-payload bytes. The execution plan explicitly selects `SwissJoin`. Serial and parallel execution both fail with RowTable offset overflow on the immediate parent `1ae38d0d42c1`, and both pass on the exact `5e68513d62b0` commit and on upstream `373d64c02cb2`, including full output-string checks. A 2 GiB control also passes on the parent. The parent already has an overflow guard, so that check reports an error rather than reproducing the original older segfault. These are equivalent overflow-path checks, not a rerun of the full 16 GiB-per-side Python example. Closing as a duplicate of #54885 to consolidate the **still-unresolved `large_string` / Basic Join** problem in one tracker with a detailed boundary reproducer. This closure does not mean that the `large_string` case is fixed; only the separate ordinary-string Swiss Join limitation was addressed by the historical change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
