haohuaijin opened a new pull request, #24186: URL: https://github.com/apache/datafusion/pull/24186
## Which issue does this PR close? - N/A (follow-up to the bitmap-backed `RowSelection` support added in parquet 59; happy to file a tracking issue if preferred) ## Rationale for this change Parquet 59 added a bitmap (`BooleanBuffer`) backing for `RowSelection` alongside the RLE selector form. However, `ParquetAccessPlan` always materialized selectors, so a caller-provided bitmap selection (via the `ParquetRowSelection` extension or an external `ParquetAccessPlan`) was converted to RLE before reaching the parquet reader. For highly fragmented selections the bitmap form is significantly cheaper to build, slice, and intersect, so it should survive the trip through the access plan. ## What changes are included in this PR? - `ParquetAccessPlan::try_new_from_overall_row_selection` splits a mask-backed selection per row group with `RowSelection::split_off`, which slices the `BooleanBuffer` (O(bitmap) overall) instead of materializing selectors. The selector path is unchanged. - `ParquetAccessPlan::into_overall_row_selection` produces a bitmap-backed overall selection when **any** row group selection is bitmap-backed, promoting selector-backed groups (e.g. groups that page index pruning intersected back to selectors). When no group is bitmap-backed, the existing selector path is preserved. - `ParquetAccessPlan::scan_selection` promotes the incoming selection to a bitmap when the existing selection is bitmap-backed, since `RowSelection::intersection` only stays mask-backed when both sides are masks; the intersection is then a bitwise AND instead of a selector merge. - `reverse_row_selection` (reverse scans for TopK) slices the bitmap per scanned row group and re-concatenates in reverse, keeping the mask backing; a `debug_assert` guards the "selection covers exactly the scanned row groups" contract. - Row-count validation now uses `row_count() + skipped_row_count()`, which avoids materializing selectors from a mask just to count rows. ## Are these changes tested? Yes: - Unit tests for the mask path of `try_new_from_overall_row_selection` (round trip + invalid row count), mixed-backing promotion in `into_overall_row_selection`, mask preservation in `scan_selection`, and bitmap preservation through `PreparedAccessPlan::reverse` (including a fully scanned row group). - End-to-end tests in `external_access_plan.rs`: a bitmap `ParquetRowSelection` spanning row groups, and a bitmap selection combined with a predicate that prunes a row group via statistics. - Existing selector-based tests are unchanged and passing (`datafusion-datasource-parquet` unit tests and the `parquet_integration` suite). ## Are there any user-facing changes? No API changes. Behavior change: when any per-row-group selection is bitmap-backed, the overall `RowSelection` handed to the parquet reader is now bitmap-backed instead of selector-backed (semantically identical selection). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
