alamb opened a new pull request, #24182: URL: https://github.com/apache/datafusion/pull/24182
Note: this PR is stacked on #24150 -- please review only the final commit until that merges. ## Which issue does this PR close? - N/A (small performance follow-on to #24150) ## Rationale for this change During page index pruning, `prune_plan_with_page_index_and_metrics` creates a `StatisticsConverter` for each predicate **inside** the per-row-group loop. Each construction calls `parquet_column`, which does a linear scan of both the Arrow schema fields and the Parquet leaf columns to resolve the predicate column, so the same column resolution is recomputed once per `(predicate, row group)`. For files with many row groups this adds up: the single-file ClickBench `hits.parquet` has 226 row groups and 105 columns, so a query with 2 pushed-down predicate columns resolved the same 2 columns 452 times per file open. ## What changes are included in this PR? - Hoist the `(predicate, StatisticsConverter)` construction out of the row-group loop so each predicate column is resolved once per file, and pass the converter down by reference (`prune_pages_in_one_row_group` and `PagesPruningStatistics` now borrow it). - Minor side effect: predicates skipped because they are multi-column or fail converter creation are now logged once per file rather than once per row group. ## Are these changes tested? Covered by existing tests: `datafusion-datasource-parquet` lib tests and the `page_pruning` integration tests in `datafusion/core/tests/parquet/` all pass. No behavior change intended. ## Are there any user-facing changes? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
