ulysses-you commented on PR #59630: URL: https://github.com/apache/spark/pull/59630#issuecomment-6595197453
Updated in d3e2d50e020 and rebased onto master. - Removed the union cost tag and the preparation pass that restored it. `SimpleCostEvaluator` now uses `PushDownShuffleExchangeThroughUnion.hasPushedShuffle` to inspect current and proposed plans directly, including partially staged unions. The inserted-exchange tag remains, with its accessor moved into the same companion object. - Kept ordinary exchanges and recognized unions equally weighted. Recognition still relies on direct requirement shuffles/stages or column repartitions whose count differs from the effective default after retargeting; the helper documents this assumption. - Added runtime pruning coverage for two- and three-child unions. Pruning the sole pushed branch can leave a two-child, non-plain union that no longer matches the helper. The test confirms physical shuffles and comparison costs both drop from one to zero. This exposes a limitation of shape-based compensation: the virtual-cost decrease can offset another exchange in the total-cost comparison, though the test does not reproduce such an exchange being adopted. The PR description now reflects the utility-based model and this cost boundary. After rebasing, the push-down, EnsureRequirements and AdaptiveQueryExec suites passed 323 tests. Catalyst/SQL source/test scalastyle, Java lint and the changed-source scan passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
