bhollis-dbx opened a new pull request, #58753: URL: https://github.com/apache/spark/pull/58753
### What changes were proposed in this pull request? Process UNION children in one left-to-right pass when DeduplicateRelations adds output projections. This preserves branch order, expression-ID deduplication, projection tags, and streaming checkpoint sharing. Add a direct helper for assigns fresh expression IDs across plans. Use it instead of synthetic self-joins in InlineCTE, ReplaceCTERefWithRepartition, and PushDownJoinThroughUnion. ### Why are the changes needed? The UNION path compares every branch with every later branch and repeatedly rebuilds the remaining sequence. This work grows quadratically with the branch count. Several optimizer paths also construct and analyze synthetic self-joins solely to renew expression IDs for one side, adding unnecessary tree traversal and operator-resolution work. These costs affect generated compatibility views and other plans with hundreds of UNION branches or wide CTE outputs. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Added focused semantic and allocation-scaling tests covering: - UNION branch order and overlapping expression IDs - Projection tags and streaming children - Equivalence with synthetic self-join deduplication - Non-timing allocation bounds Ran: - AnalysisSuite focused DeduplicateRelations tests - PushDownJoinThroughUnionSuite - InlineCTESuite - CTEInlineSuiteAEOff - CTEInlineSuiteAEOn Focused allocation measurements: - 500-branch UNION: 39,284,232 → 9,325,920 bytes - 500-column right-side renewal: 4,391,304 → 2,394,288 bytes A local 500-branch compatibility-view benchmark measured median planning time at 303.95 → 289.28 ms. Whole-query allocation was effectively unchanged at 159.70 → 159.69 MiB because relation renewal dominates this workload. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
