haohuaijin opened a new pull request, #25608: URL: https://github.com/apache/datafusion/pull/25608
## Which issue does this PR close? Related to #24358. This is a preparatory refactor and does not close that issue. ## Rationale for this change Parquet access plans already describe selections per row group, but the scan path flattens them into one selection and reconstructs group boundaries when reversing the scan. Keeping selections attached to their row groups preserves local coordinates and bitmap representations while simplifying ordering and decoder rebuilds. ## What changes are included in this PR? - Pass row-group-local selections to the Parquet push decoder and keep each selection together with its static match status. - Validate selection lengths and remove empty selections during preparation; reverse and reorder whole row-group entries. - Remove flat-selection assembly and reversal helpers, and consolidate the reversal tests. - Reuse the decoder's remaining selections when only the row filter changes, avoiding redundant selector copies. - Preserve the current policy that disables statistics reordering and runtime pruning for scans with selections. ## What is the testing strategy for this PR? The new integration test covers mixed bitmap and selector selections across row groups. Unit tests cover selection validation, empty selections, match-status alignment, and reversal of non-contiguous groups with unequal sizes. Passed targeted tests: - `cargo test -p datafusion-datasource-parquet --lib access_plan::test --offline` - `cargo test -p datafusion-datasource-parquet --lib sort::tests --offline` - `cargo test -p datafusion-datasource-parquet --lib push_decoder::tests --offline` - `cargo test -p datafusion-datasource-parquet --lib statistics_order_tests --offline` - `cargo test -p datafusion --test parquet_integration external_access_plan --offline` - `cargo test -p datafusion --test parquet_integration dynamic_row_group_pruning --offline` Pre-submission checks: `cargo fmt --all`, `cargo clippy --all-targets --all-features -- -D warnings`, and `uv run ./dev/rust_lint.sh`. ## Are there any user-facing changes? Breaking Rust API change: removes `ParquetAccessPlan::into_overall_row_selection` without adding a replacement public conversion API. The 56.0.0 upgrade guide describes migration for direct callers. Users who attach a `ParquetAccessPlan` to a `PartitionedFile` for DataFusion to scan do not need changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
