neilconway opened a new pull request, #11206: URL: https://github.com/apache/arrow-rs/pull/11206
# Which issue does this PR close? - Closes #11205. # Rationale for this change `filter`, `filter_record_batch`, `BatchCoalescer::push_batch_with_filter`, and `FilterBuilder::new` start by calling `BooleanArray::true_count` on the caller-provided mask. However, some callers may already know the number of set bits in the mask -- for example, because they built the mask one row at a time, because they derived it from another data structure (e.g., a validity bitmap), or because they needed to compute it for some other reason. In those situations, calling `true_count` requires a redundant full pass over the bitmap. Some examples from DataFusion where the count is known before the kernel runs: - `FilterExec` with a `LIMIT` counts the mask to track how many rows remain, then `push_batch_with_filter` counts it again. - `CASE` evaluation filters the remaining rows for each branch; the count of each new mask is the previous predicate's `count()` subtracted from the remaining row count. - `AND`/`OR` short-circuit evaluation counts the left side to decide whether to pre-select rows, then drops the count and filters the batch. - Aggregate `FILTER` clauses count the selection to decide whether filtering is worthwhile, then filter with the same mask. # What changes are included in this PR? - `FilterBuilder::new_with_count` changes from `pub(crate)` to `pub unsafe fn`. `FilterBuilder::new` now calls it. - New unsafe functions `filter_with_count` and `filter_record_batch_with_count` are added. The existing safe functions delegate to them. - `BatchCoalescer::push_batch_with_filter_and_count` is added. The existing safe `push_batch_with_filter` delegates to it. The new APIs are marked unsafe, because they depend on the caller to maintain a correctness invariant. The invariant is checked but only with `debug_assert_eq!`. # Are these changes tested? Yes; existing tests pass, new tests added. # Are there any user-facing changes? No, new APIs only. # AI usage Developed with Claude Code (Fable 5.1). I have reviewed and understand the resulting code. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
