JoeryH opened a new pull request, #12307: URL: https://github.com/apache/arrow-rs/pull/12307
# Which issue does this PR close? - Closes #12305 # Rationale for this change `BatchCoalescer` copies the strings of view arrays it gc's into blocks whose size doubles from 8 KiB to 1 MiB and never starts over. After a few output batches, every batch gets at least one 1 MiB block however few bytes it holds. Blocks closed early, when a push that can reuse its buffers arrives, keep their full capacity. `get_array_memory_size` reports that capacity, so the longer a coalescer runs, the more memory each row appears to use. In DataFusion, a hash join building on a hash-repartitioned scan accounted 3–4x process RSS. The issue has a standalone repro and measurements. # What changes are included in this PR? - `BufferSource` starts at 64 KiB and doubles up to 1 MiB (or the size a single request needs, if larger). `finish()` gives each output batch a new source, so block sizes depend only on that batch. - A new `push_completed` closes every block (previously six places did this by hand). It shrinks a block to its contents when less than half of it is used. The 64 KiB start and the per-batch doubling are a trade-off: sizing every block at `max(need, 64 KiB)` accounted a bit less, but with long strings copied in small pushes it made up to 13x more buffers per batch, and `take` on the concatenated output, which clones the buffer list, got up to 1.9x slower. With doubling, that case stays within 5 buffers per batch and matches `main` (numbers in #12305). # Are these changes tested? Yes: - `test_string_view_copied_blocks_track_each_batch` pushes small `take`s of a large array through one coalescer and checks that no output batch's buffers have more than twice the capacity of the strings they hold. On `main` it fails at the third batch (131072 bytes of capacity for 40000). - The `BufferSource` unit tests cover the new sizing. - Four existing layout tests (`test_string_view_batch_large_slice_compact`, `_many_small_compact`, `_many_small_boundary`, `_large_small`) now expect one block shrunk to its contents instead of three doubling ones. `coalesce_kernels` (`utf8view` cases, `main` vs this branch, pinned to 8 cores of a shared host): median change +0.3% across the 103 cases with non-inline strings, spread −22% to +22%. The 21 cases with only inline strings (`max_string_len=8`), which never reach the changed code, spread −23% to +7% (median +0.6%), so that's this host's noise level, and I found no consistent change either way. A run on a quieter machine would be welcome. # Are there any significant user-facing changes? No API changes. Coalesced view arrays may have a different number and size of data buffers than before. # AI usage disclosure I used Claude Code (Anthropic) for this PR: - **Diagnosis:** it found the cause while investigating memory accounting in a DataFusion hash join, and wrote the repro in the issue. - **Code:** it wrote the patch and tests in this PR. - **Benchmarks:** it ran the benchmarks and the comparison of block-sizing strategies. The same patch is carried downstream on arrow-select 59.3.2, where it passes that project's test suite. cc @alamb (author of the StringView coalescer and `BufferSource`, #7650) and @Dandandan (#9132, avoiding overallocation in coalesce views). Would you be able to take a look? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
