yifei-yang-db opened a new pull request, #58256: URL: https://github.com/apache/spark/pull/58256
### What changes were proposed in this pull request? `QueryPlan.normalizeExpressions(e, input: AttributeSeq)` rewrites an expression's `AttributeReference` exprIds to positional ordinals using `input`'s `exprIdToOrdinal` map, which is an instance-scoped `lazy val`. Several `doCanonicalize` implementations normalize a whole sequence with `seq.map(QueryPlan.normalizeExpressions(_, attrs))`, passing a bare `Seq[Attribute]` as `input`. The implicit `Seq[Attribute] => AttributeSeq` conversion is then re-applied on every element, so a fresh `AttributeSeq` (and a fresh `exprIdToOrdinal` map over all of `attrs`) is built for each element -- `O(seq.size * attrs.size)`. This PR adds an overload `QueryPlan.normalizeExpressions(exprs: Seq[T], input: AttributeSeq): Seq[T]` that binds the `AttributeSeq` a single time (so its lookup map is built once for the whole sequence, `O(n)`) and routes the affected canonicalization call sites through it: `LogicalRelation`, `DataSourceV2ScanRelation`, `FileSourceScanExec`, `BatchScanExec`, `InMemoryRelation`, `InMemoryTableScanExec`, `SubqueryBroadcastExec`, `SubqueryAdaptiveBroadcastExec`. ### Why are the changes needed? When a relation's own output is normalized against itself (`output.map(normalizeExpressions(_, output))`), the current code is quadratic in the number of output columns. Rebuilding the `exprIdToOrdinal` map once per column allocates a large amount of transient garbage on the driver while canonicalizing wide, unpruned relations; binding the `AttributeSeq` once makes it linear. The normalized result is unchanged. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Added a `QueryPlanSuite` test asserting the new `Seq` overload returns the same result as the per-element form. Existing canonicalization/`sameResult` coverage is unchanged. ### Was this patch authored or co-authored using generative AI tooling? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
