Rich-T-kid opened a new issue, #11921: URL: https://github.com/apache/arrow-rs/issues/11921
## Context `BatchCoalescer::push_batch_with_filtered_columns` only uses the fused per-column filter-copy path when the filter selects at most 1/16 of the input and the selected rows fit in the current output buffer. Otherwise it calls `FilterPredicate::filter_record_batch` and passes the materialized result to `push_batch`. #11920 and #11918 avoid temporary Boolean buffers inside the fused path, but Boolean filters above the sparse cutoff still allocate filtered value and validity buffers and then immediately copy those buffers into the coalescer's builders. The `single_boolean` benchmark's 10% and 80% cases therefore do not exercise the new fused implementation. ## Rationale Boolean values and validity are bit-packed, so the filter mask can be applied directly while appending bits to the destination builders. Avoiding materialization would remove per-input allocations and an entire extra write of every selected value and validity bit. The savings may be significant across long streams or schemas containing several Boolean columns, particularly at moderate selectivity where many filtered input batches contribute to each output batch. The difficult case is when the selected rows do not fit in the remaining output capacity. The coalescer promises target-sized output batches, so it cannot simply append the entire filtered input and emit an oversized batch. ## Proposed solution For record batches whose columns can all use the Boolean fused path: 1. Make Boolean filter copying eligible independently of the global sparse-selectivity cutoff. 2. Split the predicate at the source position containing exactly the number of selected rows that fit in the current output batch. 3. Compress that source segment directly into the Boolean value and validity builders, finish the full output batch, and continue with the remainder. The selected-row boundary calculation should live behind a reusable `FilterPredicate` helper rather than in Boolean-specific coalescer code. This keeps null-filter normalization and the "Nth selected row" invariant in one place and may allow other specialized coalescer implementations to use the same mechanism later. The existing materialized path can remain the fallback for mixed schemas containing columns that cannot stream filtered rows directly, and for configured large-batch bypass behavior. ## Validation Benchmark `single_boolean` with null densities 0% and 10% at 10% and 80% selectivity. Compare against the latest #11918 run, while confirming that the sparse cases do not regress. Tests should cover crossing one and multiple output boundaries, sliced inputs, nullable values, and exact target batch sizes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
