LuciferYang commented on code in PR #58419:
URL: https://github.com/apache/spark/pull/58419#discussion_r4038321915
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala:
##########
@@ -852,6 +859,10 @@ object QueryExecution {
RemoveRedundantSorts,
ApplyColumnarRulesAndInsertTransitions(
sparkSession.sessionState.columnarRules, outputsColumnar = false),
+ // A barrier for a `UnionExec` an injected columnar rule just created,
which has no decision
+ // yet and would otherwise take one wherever it is first asked. A
decision already stamped on
+ // a node is kept.
+ StampUnionDecisions,
Review Comment:
I would rather keep these unconditional. The condition is a different list
per barrier: the classic suffix guards `sessionState.columnarRules`, the AQE
prep suffix guards `adaptiveRulesHolder.queryStagePrepRules`, and the
post-stage suffix guards the injected optimizer rules and the columnar ones
together. Every hook added at one of those positions later has to be added to
the matching condition too, and a missed one puts a union back on the live conf
silently, for the one user who configured that extension. What it buys is one
`foreach` per phase at planning time, next to
`ApplyColumnarRulesAndInsertTransitions` and `CollapseCodegenStages`, which
transform the same tree. The per-stage walks do not compound either, since
`foreach` stops at `QueryStageExec` and a stage already created is not
revisited.
If the walks are worth removing, I would rather prune the traversal than the
rule: give `UnionExec` a `nodePatternsInternal` of `Seq(UNION)` and have the
barrier ask `plan.containsPattern(UNION)` first, so a plan with no union costs
a bitset test and the invariant stays unconditional. Happy to add that here if
you want it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
