marin-ma opened a new pull request, #11252: URL: https://github.com/apache/incubator-gluten/pull/11252
In spark's AQE, the rule `CollapseCodegenStages` is considered as stateful: https://github.com/apache/spark/blob/branch-4.0/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala#L153-L156 However in Gluten, when AQE is on, the columnar rules are applied upon each individual query stages, and there's no stateful context shared across query stages that is visible to the columnar rules. Seems like there's no way to use a stateful counter to generate incremental ids across different query stages. This pr adds a new rule to update the stage id after the physical plan is generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
