andygrove commented on pull request #32195: URL: https://github.com/apache/spark/pull/32195#issuecomment-820895987
> Does `ReuseExchange` rule also possibly suffer from the issue? In the rule, Spark also only checks `canonicalized` plans of exchanges to choose reused exchange. This hasn't been a problem for us. We only ran into issues with the AQE exchange reuse logic because of the adaptive nature where query stages are being created during execution, and with Spark making changes after we provide the new query stage it is too late for our plugin to correct the issue. Without AQE, where the `ReuseExchange` rule would be applied (as far as I can see), our plugin would operate on the physical plan after Spark has applied the exchange reuse logic and we would be able to modify the plan as required after that, and before execution, to insert any necessary transitions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
