ahshahid commented on code in PR #43806: URL: https://github.com/apache/spark/pull/43806#discussion_r1393590377
########## sql/core/src/main/scala/org/apache/spark/sql/execution/SubqueryAdaptiveBroadcastExec.scala: ########## @@ -44,9 +46,21 @@ case class SubqueryAdaptiveBroadcastExec( throw QueryExecutionErrors.executeCodePathUnsupportedError("SubqueryAdaptiveBroadcastExec") } + override def equals(other: Any): Boolean = other match { + case x: SubqueryAdaptiveBroadcastExec => this.name == x.name && this.index == x.index && + this.onlyInBroadcast == x.onlyInBroadcast && this.buildPlan == x.buildPlan && + this.buildKeys == x.buildKeys && this.child == x.child + case y: SubqueryBroadcastExec => this.name == y.name && this.index == y.index && + this.buildKeys == y.buildKeys && this.child == y.child Review Comment: I don't think that would work as such and may actually complicate the things. the issue lies in the following code : In AdaptivePlanExec: where we look for exising stage for reuse: ` if (conf.exchangeReuseEnabled) { // Check the `stageCache` again for reuse. If a match is found, ditch the new stage // and reuse the existing stage found in the `stageCache`, otherwise update the // `stageCache` with the new stage. val queryStage = context.stageCache.getOrElseUpdate( newStage.plan.canonicalized, newStage) if (queryStage.ne(newStage)) { newStage = reuseQueryStage(queryStage, e) } } ` The canonicalized plan contains SubqueryAdaptiveBroadcastExec, while incoming canonicalized plan contains SubqueryBroadcastExec. and hence mistmatch. Any replacement , will potentially complicate things and cause perf issue too, because these nodes though extending SparkPlan, are actually embedded in ExecSubqueryExpression which means checking for each expression in all the spark plan nodes to modify. and then potentially adjusting the logical plan tags too. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org