zhuqi-lucas opened a new pull request, #24281: URL: https://github.com/apache/datafusion/pull/24281
## Which issue does this close? Closes #24264. Related to prior `OptimizeProjections` perf work (#21726). ## Rationale for this change `rewrite_projection_given_requirements` (the core of the `OptimizeProjections` rule) prunes a projection's expressions to the subset actually required, then rebuilds the projection with `Projection::try_new`: ```rust Projection::try_new(exprs_used, Arc::new(input)) ``` `try_new` recomputes the output schema from scratch via `projection_schema`, which: - calls `Expr::to_field` for **every** retained expression (type/nullability/qualifier/metadata inference), and - for column expressions resolves the field through `DFSchema::field_from_column` → `index_of_column_by_name`, which is a **linear scan** over the input schema (there is no name→index map). So recomputing one projection's schema is `O(exprs * schema_width)`, and it runs for every projection on every optimizer pass. For wide, `SELECT *`-style projections over wide schemas (tens of columns) this becomes effectively quadratic and shows up prominently in planning profiles. But the retained expressions are a **subset** of the projection's original expressions, and pruning unreferenced *sibling* columns cannot change the retained columns' output fields. The answer is already sitting in `proj.schema` — no need to re-derive it. ## What changes are included in this PR? - `rewrite_projection_given_requirements` now derives the pruned output schema by **selecting the already-computed fields** from the existing projection schema (`project_schema_by_indices`) and builds the projection with `Projection::try_new_with_schema`, instead of recomputing via `try_new`. - When nothing is pruned (the identity / `SELECT *` case), the existing schema `Arc` is reused as-is. - Functional dependencies are projected through the kept indices (`FunctionalDependencies::project_functional_dependencies`). - This mirrors the schema reuse already performed in `merge_consecutive_projections` (which reuses `schema` unchanged when the expression list is unchanged). Complexity for a pruned projection goes from `O(exprs * schema_width)` (schema recompute) to `O(k)` (field slice), and to `O(1)` when nothing is pruned. ## Correctness The sliced schema is identical to the one `projection_schema` would recompute: field `i` of the projection schema corresponds to expression `i`, and `RequiredIndices` yields a sorted, deduplicated index subset, so slicing `proj.schema` at those indices produces exactly the fields of the retained expressions, with qualifiers and metadata preserved. New test `project_schema_by_indices_matches_recompute` asserts, for a mixed expression list (plain column, computed binary expr, alias, nullable literal, qualified column) and every representative index subset, that `project_schema_by_indices(schema, indices)` produces the same fields and qualifiers as `projection_schema(input, exprs_used)`, and that the identity subset reuses the same `Arc`. The full `datafusion-optimizer` test suite (760 unit + 26 integration, including the `EXPLAIN` plan snapshots) passes unchanged, i.e. no optimized plan output changes. ## Are there any user-facing changes? No. This is an internal optimizer performance improvement; planned/optimized plans are unchanged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
