sunchao commented on code in PR #7357:
URL: https://github.com/apache/datafusion-comet/pull/7357#discussion_r4637456786


##########
spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala:
##########
@@ -1108,37 +1131,29 @@ case class CometExecRule(session: SparkSession, 
queryStagePrep: Boolean = false)
 
       // Set up logical links
       newPlan = newPlan.transform {
-        case op: CometExec
-            if op
-              .getTagValue(SparkPlan.LOGICAL_PLAN_TAG)
-              .exists(_.isInstanceOf[LogicalQueryStage]) =>
-          // AQE replanning reuses this physical root and links it to the 
current logical stage.
-          // originalPlan can still point to a subtree hidden inside that 
logical leaf, which
-          // AQE cannot replace in the current logical plan. Only preserve a 
direct stage link,
-          // not a link inherited from an ancestor.
-          // On the ordinary exchange path, the exchange itself is behind a 
QueryStageExec
-          // leaf and is not visited by this transform.
-          // Spark 4.1.3 returns the existing root in 
LogicalQueryStageStrategy and then calls
-          // setLogicalLink from SparkStrategies.plan:
+        case op: CometExec =>
+          // Spark reuses the physical root and links it directly to the 
current AQE stage.
+          // A descendant can inherit that stage if it wraps the descendant's 
original node.
+          // AQE replaces the linked node by identity. A stale link leaves the 
plan unchanged.
           // scalastyle:off line.size.limit
           // 
https://github.com/apache/spark/blob/v4.1.3/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/LogicalQueryStageStrategy.scala#L64-L65
           // 
https://github.com/apache/spark/blob/v4.1.3/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala#L78-L87
+          // 
https://github.com/apache/spark/blob/v4.1.3/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala#L778-L783
           // scalastyle:on line.size.limit
-          op
-        case op: CometExec =>
-          if (op.originalPlan.logicalLink.isEmpty) {
-            op.unsetTagValue(SparkPlan.LOGICAL_PLAN_TAG)
-            op.unsetTagValue(SparkPlan.LOGICAL_PLAN_INHERITED_TAG)
-          } else {
-            op.originalPlan.logicalLink.foreach(op.setLogicalLink)
+          val keepStage = op.logicalLink.exists {
+            case stage: LogicalQueryStage =>
+              // logicalLink prefers the direct tag, so if present it points 
to this stage.
+              op.getTagValue(SparkPlan.LOGICAL_PLAN_TAG).isDefined ||
+              op.originalPlan.logicalLink.exists(stageWraps(stage, _))
+            case _ => false
+          }
+          if (!keepStage) {
+            copyLogicalLinks(op.originalPlan, op, clearMissing = true)
           }
           op
         case op: CometShuffleExchangeExec =>
-          // Original Spark shuffle exchange operator might have empty logical 
link.
-          // But the `setLogicalLink` call above on downstream operator of
-          // `CometShuffleExchangeExec` will set its logical link to the 
downstream
-          // operators which cause AQE behavior to be incorrect. So we need to 
unset
-          // the logical link here.
+          // An exchange without an original logical link can inherit an 
unrelated one from
+          // its parent. Clear it so AQE does not associate the exchange with 
that parent.
           if (op.originalPlan.logicalLink.isEmpty) {
             op.unsetTagValue(SparkPlan.LOGICAL_PLAN_TAG)
             op.unsetTagValue(SparkPlan.LOGICAL_PLAN_INHERITED_TAG)

Review Comment:
   Both exchange branches now use `copyLogicalLinks(op.originalPlan, op, 
clearMissing = true)`. I found an inherited-only source in Spark 4.x. For an 
ordered `LIMIT` inside a subquery, Spark can insert a local sort above a 
single-partition shuffle. Spark tags the sort directly, and the shuffle 
inherits the same logical sort. Calling `setLogicalLink` on the converted 
shuffle made that tag direct, which stopped future updates from its parent. The 
new regression checks that the link stays inherited and points to the exact 
same object. [Spark sort 
insertion](https://github.com/apache/spark/blob/v4.1.3/sql/core/src/main/scala/org/apache/spark/sql/execution/InsertSortForLimitAndOffset.scala#L42-L64)
 · 
[Regression](https://github.com/apache/datafusion-comet/blob/d9f90a6a8eab1ae45584685df58958333b4ccdba/spark/src/test/scala/org/apache/comet/exec/CometExecSuite.scala#L2450-L2466)
   
   AQE reads the exchange’s temporary tag first, then its direct or inherited 
link. It searches the children only when the exchange has neither, so repair 
can copy the two tags on the exchange itself and let each child keep its own 
link. This also covers the empty-link case in #323. The existing DPP test 
checks the temporary tag on a reused broadcast. [AQE 
lookup](https://github.com/apache/spark/blob/v4.1.3/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala#L718-L727)
 · [DPP 
test](https://github.com/apache/datafusion-comet/blob/d9f90a6a8eab1ae45584685df58958333b4ccdba/spark/src/test/scala/org/apache/comet/rules/CometExecRuleSuite.scala#L1700-L1733)
   
   I kept the helper scoped to the planner rules. The transition and cache 
rules normally receive fresh, untagged wrappers. The scan fallbacks rebuild 
leaves, where a residual filter can give the scan an inherited link. Those 
construction and fallback paths belong in the broader audit in #6034. 
[d9f90a6a8](https://github.com/apache/datafusion-comet/pull/7357/commits/d9f90a6a8eab1ae45584685df58958333b4ccdba)
   



##########
spark/src/test/scala/org/apache/comet/rules/CometExecRuleSuite.scala:
##########
@@ -1429,7 +1429,21 @@ class CometExecRuleSuite extends CometTestBase {
     }
   }
 
-  test("CometExecRule repairs ordinary and inherited logical links from the 
original plan") {
+  test("CometExecRule preserves an inherited AQE stage for its original 
logical subtree") {
+    withAdaptiveAggregateConf {
+      val aggregate = createAdaptiveAggregate()
+      val original = aggregate.originalPlan.logicalLink.get
+      val stage = LogicalQueryStage(LogicalQueryStage(original, aggregate), 
aggregate)
+      aggregate.unsetTagValue(SparkPlan.LOGICAL_PLAN_TAG)
+      aggregate.setTagValue(SparkPlan.LOGICAL_PLAN_INHERITED_TAG, stage)
+
+      val repaired = applyCometExecRule(aggregate)
+      assert(repaired.getTagValue(SparkPlan.LOGICAL_PLAN_TAG).isEmpty)
+      assert(repaired.logicalLink.exists(_ eq stage))
+    }
+  }

Review Comment:
   The three tests now share the aggregate setup and run with a direct, 
inherited, or absent tag on the original. The nested-stage test checks that the 
exact outer stage survives as an inherited link in both tagged cases. With an 
untagged original, it checks that repair clears the link. [Shared 
setup](https://github.com/apache/datafusion-comet/blob/d9f90a6a8eab1ae45584685df58958333b4ccdba/spark/src/test/scala/org/apache/comet/rules/CometExecRuleSuite.scala#L1313-L1331)
 · [Nested-stage 
assertions](https://github.com/apache/datafusion-comet/blob/d9f90a6a8eab1ae45584685df58958333b4ccdba/spark/src/test/scala/org/apache/comet/rules/CometExecRuleSuite.scala#L1436-L1450)
 · 
[d9f90a6a8](https://github.com/apache/datafusion-comet/pull/7357/commits/d9f90a6a8eab1ae45584685df58958333b4ccdba)
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to