sunchao opened a new issue, #6034: URL: https://github.com/apache/datafusion-comet/issues/6034
### What is the problem the feature request solves? `CometExecRule` repairs links on `CometExec`, `CometShuffleExchangeExec`, and `CometBroadcastExchangeExec` using `originalPlan.logicalLink`, or clears both logical tags when that link is absent. #5482 / #5483 establish that `originalPlan` is not always authoritative: Spark can reuse an existing physical root and assign a direct link to the current `LogicalQueryStage`. #5483 preserves that direct link on `CometExec`. This issue tracks the separate question raised in [the review](https://github.com/apache/datafusion-comet/pull/5483#pullrequestreview-5154584449): which remaining repairs are necessary, and can link ownership be made explicit or repair narrowed without breaking AQE? ### Describe the potential solution Audit initial conversion, repeated AQE replanning, and exchange/subquery creation to document when the original plan, the current physical node, or Spark's temporary stage link is authoritative. Distinguish fresh nodes from reused physical nodes and direct tags from inherited tags. Decide from that evidence whether to retain the current repair, narrow it, or move necessary assignments to node creation. Any change should preserve: - The current direct `LogicalQueryStage` identity across repeated replanning (#5482). - Necessary restoration/clearing for ordinary and inherited links, including the empty shuffle-link behavior from #323. - Ordinary shuffle/broadcast stage wrapping and DPP subqueries whose adaptive-plan root is a `CometBroadcastExchangeExec`. The [DPP investigation](https://github.com/apache/datafusion-comet/pull/5483#issuecomment-5442492239) found an unchanged, equal-cost replan that Spark rejects, while retaining `TEMP_LOGICAL_PLAN_TAG` for new stage creation. Establish the required contract across supported Spark versions. Back the chosen contract with focused planner tests and end-to-end AQE/DPP coverage that checks results and actual native execution. This audit does not assume all repair is removable or establish another failing exchange case. ### Additional context Keep #5482 focused on its reproduced stale-stage-link defect. The broader design question should remain tracked after #5483 closes that issue. [Current logical-link repair](https://github.com/apache/datafusion-comet/blob/0d8233403514e6d4f3eaf7f488896d18458fb5ad/spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala#L747-L798). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
