haohuaijin opened a new pull request, #25731: URL: https://github.com/apache/datafusion/pull/25731
## Which issue does this PR close? - Closes #23883. - Closes #24488. - Follows up on #24186 using the row-group-local scan path merged in #25608. ## Rationale for this change An external index can supply a bitmap-backed Parquet row selection, but splitting a file-level selection into row groups or intersecting it with selector-backed page pruning still converts it to selectors. Keeping the bitmap avoids materializing fragmented selector runs and lets pruning use a bitwise intersection. ## What changes are included in this PR? - Split bitmap-backed file selections using bitmap slices, preserving partial-group masks and recognizing fully selected or skipped groups. Keep the existing single-pass selector path. - Promote incoming selectors to a bitmap when intersecting with an existing bitmap-backed selection. - Use Parquet's `RowSelection::total_row_count()` directly instead of introducing the temporary `row_selection_len` helper from #24186. - Add regression coverage for bitmap preservation and external selections when a dynamic predicate changes between row groups. Consolidate opener test imports at module scope. The deprecated `into_overall_row_selection` implementation is unchanged. Preparation and reverse scans use the local selections introduced by #25608. Runtime pruning remains disabled for scans with selections. ## What is the testing strategy for this PR? Unit tests cover selector/bitmap intersections, empty intersections, file-length validation, non-byte-aligned bitmap slicing, and preservation through preparation and reversal. Integration tests cover bitmap selections spanning row groups and statistics pruning. The dynamic-pruning regression covers both selector and bitmap external selections, with an unselected control scan that verifies runtime pruning occurs. Temporarily removing the selection guard makes the test fail because rebuilding with `None` returns unselected rows; the guard was restored after this check. Passed targeted tests: - `cargo test -p datafusion-datasource-parquet --lib access_plan::test --offline` (26 tests) - `cargo test -p datafusion-datasource-parquet --lib opener::test --offline` (56 tests) - `cargo test -p datafusion --test parquet_integration external_access_plan --offline` (16 tests) Pre-submission checks passed: `cargo fmt --all`, `cargo clippy --all-targets --all-features -- -D warnings`, and the checks in `uv run ./dev/rust_lint.sh`. The final HTML documentation check was rerun successfully with `uv run ./ci/scripts/check_docs_html.sh` after installing its missing `cargo-depgraph` prerequisite. ## Are there any user-facing changes? No public API or selected-row changes. Bitmap-backed external selections retain their representation through splitting and intersection rather than being converted to selectors. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
