LuciferYang opened a new pull request, #59151: URL: https://github.com/apache/spark/pull/59151
### What changes were proposed in this pull request? This backports SPARK-59122 (#58419, already in master and branch-4.x) to branch-4.3. The change makes `UnionExec` decide once, right after `EnsureRequirements`, whether it is a plain concatenation, and freeze that decision plus the codegen confs in a `TreeNodeTag`, so the fusion gate, `metrics`, `unionRDDs` and the copy of the union inside the codegen shell can no longer give different answers depending on when each is read. See #58419 for the full design and the two failures it fixes. Two conflicts came up against branch-4.3, both because the branch predates code the master change was built on. Neither changes the fix: - branch-4.3 does not have SPARK-57399, so `AQEEnablePipelinedShuffle` is not in the AQE query-stage preparation rule list. The two union barriers are appended after `RemoveRedundantSorts` exactly as on master, without that rule. - branch-4.3 does not have SPARK-58819 or SPARK-59285, so `UnionExec` still derives its candidate partitioning by remapping to the first child's attributes and lifting with a local `toUnionOutput`, and `comparePartitioning` does not compare `keyDataTypes`. The backport keeps that body. The SPARK-59122 edit to it is only the rename of the old `outputPartitioning` body to a private `rawPartitioning`, moving the `spark.sql.unionOutputPartitioning` gate out of an early return and into the stamped decision, and adding the barriers and the preparation-conf snapshot around it. ### Why are the changes needed? SPARK-59122 is a Blocker correctness fix. On default settings a union whose children share a partitioning could concatenate at execution while a parent had already dropped the exchange it needed, giving wrong results; a fused union could also lose the `numOutputRows` metric its generated code asks for and crash. branch-4.3 is exposed to both, since `spark.sql.unionOutputPartitioning` defaults to true there. ### Does this PR introduce _any_ user-facing change? Yes, the same change as #58419: the wrong-result and codegen-crash cases above now behave correctly. The three confs the decision reads no longer reach a plan that has already been prepared, only plans prepared afterwards. ### How was this patch tested? The tests come with the cherry-pick, unchanged from #58419; CI runs them against branch-4.3. The AQE rule-order case asserts positions relative to `EnsureRequirements` and the end of the list, so it holds with `AQEEnablePipelinedShuffle` absent without editing. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
