mohitgurav20 commented on issue #25712:
URL: https://github.com/apache/datafusion/issues/25712#issuecomment-5818761820
Thanks for the detailed issue report @adriangb!
### Technical Summary & Root Cause
- In `concat_build_batches`, `concat_batches` appends `StringViewArray` data
buffers without pointer deduplication.
- When $N$ build batches share $K$ buffer allocations, the concatenated view
array contains $N \times K$ buffer references. Chaining `CollectLeft` joins
causes geometric reference growth ($1 \to 6 \to 42 \dots \to 1.58\text{M}$
pointers), leading to heavy CPU overhead in
`RecordBatchMemoryCounter::count_buffer_memory_size` and high memory usage (8.5
GB vs 1.2 GB).
### Proposed Fix
Implement a zero-copy pointer deduplication pass in `concat_build_batches`
(`datafusion/physical-plan/src/joins/hash_join/exec.rs`):
1. Identify distinct data buffer pointers via `Buffer::as_ptr()`.
2. Remap the 4-byte `buffer_index` field in the 128-bit view descriptors to
the deduplicated index array.
3. This requires zero string byte copying and immediately caps buffer
reference growth across join chains.
I'd like to work on implementing this fix and unit tests!
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
