adriangb opened a new pull request, #25729: URL: https://github.com/apache/datafusion/pull/25729
## Which issue does this PR close? - No issue. Found while working on https://github.com/apache/datafusion/issues/22883. ## Rationale for this change When the page index is enabled (the default), a TopK or join dynamic filter does not prune Parquet row groups at runtime if the page index prunes no rows. Disabling the page index makes the same query faster. TPC-H SF1 `lineitem` (1 file, 53 row groups), default config, debug build: ```sql EXPLAIN ANALYZE SELECT l_orderkey, l_extendedprice FROM lineitem WHERE l_discount < 0.05 ORDER BY l_orderkey LIMIT 10; ``` | | `page_index_pages_pruned` | `row_groups_pruned_dynamic_filter` | scan `output_rows` | `bytes_scanned` | |---|---|---|---|---| | `main`, page index on | 318 total → 318 matched | 0 | 6.00 M | 41.6 MB | | `main`, page index off | 0 total → 0 matched | 41 | 1.36 M | 9.4 MB | | this PR, page index on | 318 total → 318 matched | 41 | 1.36 M | 9.4 MB | Cause: page-index pruning installs a `RowSelection` for each row group that it examines, also when the selection skips no rows. `ParquetAccessPlan::prepare` sets `has_row_selection` when any row group has a selection. That flag disables: - runtime row-group pruning (`RowGroupPruner`, the dynamic filter path), and - `reorder_by_statistics` (row-group reordering for sort pushdown / TopK). Both gates are intentional for real selections (see https://github.com/apache/datafusion/issues/24358). A selection that skips no rows is equal to a full scan, so it must not trigger them. ## What changes are included in this PR? `PagePruningAccessPlanFilter::prune_plan_with_page_index_and_metrics` does not install the selection when it skips 0 rows. The row group stays `RowGroupAccess::Scan`. Selections that skip rows are unchanged. Metrics are unchanged. I did not change the gate in `prepare` to "a selection skips rows". The fix at the source is smaller, and the access plan then does not carry a no-op selection into the decoder. ## What is the testing strategy for this PR? - `page_filter.rs`: `page_pruning_that_skips_no_rows_installs_no_selection`. It fails on `main` (the row group is `Selection([select 6])` instead of `Scan`). - `dynamic_row_group_pruning.slt`: a TopK query where the page index prunes no page. It shows `row_groups_pruned_dynamic_filter=4` with the page index on and off. On `main` the page-index-on case shows `0`. - `cargo test -p datafusion-datasource-parquet`, `cargo test -p datafusion --test parquet_integration` and all sqllogictests pass. No existing expected output changed. ## Are there any user-facing changes? No API change. Some scans with the page index enabled prune more row groups at runtime. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
