JonSnow1807 opened a new pull request, #12130: URL: https://github.com/apache/arrow-rs/pull/12130
> **Draft: performance regression unresolved.** The sparse-nullable benchmark is consistently about 4% slower. This candidate is not ready for review or merge until that tradeoff is resolved. # Which issue does this PR close? Relates to #11921. # Rationale for this change BatchCoalescer materializes filtered arrays when a Boolean filter exceeds the sparse-copy threshold or its selected rows cross the current output batch boundary. This allocates temporary arrays and buffers before copying their contents into the coalesced output. The proposed change writes selected Boolean bits directly into the final output buffers, including when an input spans multiple output batches. # What changes are included in this PR? Route Boolean-only schemas through direct filtering at dense selectivities. Split reusable predicates at selected-row boundaries so completed batches retain the configured target size. The split masks share the normalized bitmap without building an index vector or recounting the whole remaining predicate. Pre-size the destination values range and initialize the existing bit packer with any destination prefix. Completed packed words write directly into the destination. Sparse nullable inputs retain the original append helper at the existing 1/16 cutoff. Validity remains lazy. Full 64-bit builder appends write bytes directly while preserving partial-byte prefixes; partial-word appends retain their existing behavior. Mixed schemas and the configured large-batch bypass retain their existing materialized paths. There are no new unsafe blocks or public APIs. # Are these changes tested? New tests compare coalesced outputs with materialized filtering across partial buffers, exact and repeated batch boundaries, tiny output sizes, multiple Boolean columns, all/none selections, shorter predicates, configured bypasses, and nullable predicates whose null slots contain physical true bits. Predicate-splitting tests cover optimized and lazy predicates and independently sliced bitmaps around byte and word boundaries. The packed writer is checked against bit-by-bit output for prefix lengths 0–127 and append lengths around 0, 1, 7, 8, 63, 64, 65, 127, 128 and 129. Tests cover reused and truncated storage, raw padding bytes, and subsequent false appends. Full-word Boolean and validity builder tests cover unaligned prefixes and lazy validity materialization. Additional nullable tests select exactly 511, 512 and 513 rows from 8,192-row inputs and compare outputs across 7-, 512- and 1,024-row targets with a partially filled destination. For the exact final candidate, all 382 arrow-buffer library tests and 516 arrow-select library tests pass. The complete workspace suite passes with 7,653 tests passed and 30 ignored. Workspace Clippy with all targets and all features, formatting, and spelling checks also pass. Independent source reviews found no actionable correctness defects. # Allocation measurements Measurements used an Intel Xeon E5-2673 v4 on Linux x86_64, Rust 1.99.1, and default release compiler flags. The allocation harness processes 128 prepared 8,192-row inputs with one Boolean column and an output target of 8,192 rows. Inputs are prepared outside the counted region. All 56 allocation cases preserve the expected output row counts. Selected rows per input | Source nulls | Allocation calls before | Allocation calls after | Requested bytes before | Requested bytes after --- | --- | ---: | ---: | ---: | ---: 819 | 0% | 564 | 64 | 58,208 | 17,472 819 | approximately 10% | 846 | 90 | 92,728 | 31,512 6,553 | 0% | 924 | 514 | 262,688 | 139,152 6,553 | approximately 10% | 1,386 | 720 | 486,568 | 250,392 These are System allocator calls and cumulative requested bytes over the workload, not peak memory or process RSS. The validity and filter masks are deterministic. In a fitting-output control selecting 511 of 8,192 rows, allocation calls remain 4 and requested bytes remain 8,416. The exact final candidate reproduces all 56 allocation results. # CPU performance — unresolved regression The original upstream single_boolean coalesce_kernels benchmark uses 8,192-row input/output batches and collects 50 output batches. Final measurements used two balanced before/after pairs pinned to a sampled quiet physical core, with 30 Criterion samples, a five-second warmup and a three-second measurement period. Execution order was reversed for the second pair. Configuration | Before | After | Paired time change --- | ---: | ---: | ---: 0.1% selection, 10% source nulls | 40.45 ms | 42.05 ms | +4.0% 80% selection, no nulls | 265.2 microseconds | 250.4 microseconds | -5.6% 80% selection, 10% source nulls | 518.9 microseconds | 525.3 microseconds | +1.2% Mixed-schema materialized control | 712.3 microseconds | 709.3 microseconds | -0.4% The sparse case uses Criterion mean estimates because it used flat sampling; the other rows use slope estimates. Times are averages of the two estimates, and percentage changes are geometric centers of paired ratios. The mixed control is a separate local overlay processing 32 Boolean/Int32 input batches with 6,553 selected rows per 8,192-row input and an 8,192-row output target. The sparse-nullable pairs were 3.69% and 4.24% slower. This remains a reason to hold the patch despite its allocation reductions. Nullable dense measurements varied between 0.82% faster and 3.36% slower. An earlier identical-baseline calibration failed and was excluded; the final longer-warmup run passed calibration. No CPU governor or host policy was changed. The allocation harness, threshold controls, and benchmark-selection experiments are separate local overlays and are not included in the implementation diff. Final CPU results above use the original upstream harness, apart from the explicitly identified mixed control. # Are there any significant user-facing changes? No public API or intended result changes. The allocation reductions come with an unresolved sparse-nullable CPU regression, so this candidate is not ready for submission. # AI assistance PR code and description written by: - [x] Human - [x] AI Final review and sign-off: - [x] Human AI assistance was used for implementation, regression tests, independent technical review, and validation. The author personally reviewed the code, tests, and description before submission. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
