kush2439p opened a new pull request, #57356:
URL: https://github.com/apache/spark/pull/57356
### What changes were proposed in this pull request?
Fix a performance regression where ReusedExchangeExec fails to match the
shuffle of the underlying table when a persisted VIEW is referenced in both
branches of a UNION ALL through a CTE, causing the table to be scanned and
shuffled twice.
The fix is confined entirely to the canonicalization layer
(QueryPlan.doCanonicalize()). Inside mapExpressions, a new case is added
**before** the generic case a: Alias branch:
- Detect a *no-op rename alias* — Alias(attr: Attribute, name) where
ame == attr.name AND .metadata == attr.metadata. This is precisely the
condition RemoveRedundantAliases itself uses to decide an alias is truly
redundant.
- When matched, canonicalize the wrapping alias away and normalize the bare
ttr exactly as the existing bare-AttributeReference branches do:
- If ttr.exprId is not in llAttributesSeq (the node's children's
output), assign it the next sequential id (id += 1;
attr.withExprId(ExprId(id)).canonicalized).
- Otherwise normalize via QueryPlan.normalizeExpressions(attr,
allAttributesSeq).
No changes are made to RemoveRedundantAliases, DeduplicateRelations, or
InlineCTE.
### Why are the changes needed?
**Root cause chain (5 steps):**
1. **VIEW construction** (SPARK-34269) wraps each output column in a
schema-compat Alias(UpCast(col, field.dataType), field.name)(explicitMetadata =
Some(field.metadata)) — a genuine rename since #scan != #catalog.
2. **InlineCTE**: both UNION branches share identical ExprIds, making
Union.duplicateResolved = false.
3. **DeduplicateRelations** resolves duplicates by wrapping the non-first
branch in an outer Project(Alias(attr, attr.name)) and rewriting ExprIds via
ewInstance()/ ewriteAttrs, turning the view's inner schema-compat alias into
a no-op Alias(a#X AS a#X).
4. **RemoveRedundantAliases** (SPARK-39887 Union first-child protection):
protects branch 1's attribute set from alias removal, strips the no-op alias
from branch 2. Result: branch 1 keeps the Alias node, branch 2 has a bare
AttributeReference — structurally asymmetric though semantically identical.
This protection is **required for correctness** and is not touched.
5. **Canonicalization asymmetry**: doCanonicalize() assigns a sequential id
to the top-level Alias (branch 1) but uses ordinal-position normalization for
the bare AttributeReference (branch 2). The two branches get different
canonical forms, so ReuseExchangeAndSubquery (and AQE's stage cache) cannot
match the upstream Exchange, and 1 is shuffled twice.
### Does this PR introduce _any_ user-facing change?
Yes — this is a **performance fix**. Queries matching the pattern (a
persisted VIEW referenced in both branches of a UNION ALL via a CTE) will now
correctly reuse the shuffle/exchange across both branches, avoiding redundant
table scans and shuffles.
### How was this patch tested?
Two tests were added:
1. **Unit test** in
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/plans/QueryPlanSuite.scala:
- SPARK-57109: no-op rename Alias canonicalizes the same as a bare
attribute
- Constructs two Project branches over the same LocalRelation: one wraps
the attribute in a no-op rename Alias, the other uses the bare
AttributeReference. Asserts liasedBranch.sameResult(bareBranch) is rue.
2. **End-to-end regression test** in
sql/core/src/test/scala/org/apache/spark/sql/execution/ReuseExchangeAndSubquerySuite.scala:
- SPARK-57109: Exchange reused when a VIEW is referenced via a CTE in
both UNION branches
- Uses the exact repro SQL from the issue (creates 1, 2, view
, runs the CTE + UNION ALL query).
- Asserts that plan.collectWithSubqueries { case re: ReusedExchangeExec
=> re }.nonEmpty.
Closes #57109
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Google Antigravity (Gemini 2.5 Pro)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
