zhuqi-lucas opened a new issue, #24264: URL: https://github.com/apache/datafusion/issues/24264
### Is your feature request related to a problem or challenge? `OptimizeProjections` recomputes projection schemas more than necessary. In `rewrite_projection_given_requirements` (`datafusion/optimizer/src/optimize_projections/mod.rs`), a projection that has been pruned to its required columns is rebuilt with `Projection::try_new(exprs_used, input)`. `try_new` recomputes the schema via `projection_schema` → `Expr::to_field` for every expression, and column resolution (`DFSchema::field_from_column`) is an O(M) linear scan over the input schema (there is no name→index map). So schema construction is **O(exprs × schema_width)** per projection, per optimizer pass. This is most visible on plans with many wide `col AS col` alias projections: an alias is not a bare `Column`, so `is_projection_unnecessary` returns false and the projection is kept, its schema recomputed on every pass — even though the result schema is just a column subset of the input schema that is already available. ### Describe the solution you'd like Build the pruned schema by **slicing the existing input schema** at the (sorted, deduped) required indices and pass it via `Projection::try_new_with_schema`, instead of recomputing it from scratch. This mirrors the schema reuse already done in `merge_consecutive_projections`. It reduces the per-projection cost from O(exprs × width) to O(k) and is behavior-preserving. ### Describe alternatives you've considered Keeping `try_new` but adding a name→index map to `DFSchema` would also help, but is a larger change; slicing the already-available schema is local and sufficient here. ### Additional context A separate, tiny improvement in the same area: in `datafusion/expr/src/expr_schema.rs`, `Expr::to_field` for the `Expr::Alias` branch resolves the inner expression twice — once via `expr.metadata(schema)?` (which internally calls `to_field`) and again via `expr.to_field(schema)`. These can be collapsed into a single `to_field` call, extracting both the field and its metadata from it. I have a local patch for the first item (with a unit test) and am happy to open a PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
