zhongyu09 opened a new pull request #32562: URL: https://github.com/apache/spark/pull/32562
### What changes were proposed in this pull request? 1. replace executeCollectIterator() by executeCollectIteratorFuture() in SparkPlan.scala to run collect query in async way and return the future of collect result 2. in BroadcastExchangeExec->relationFuture, call executeCollectIteratorFuture() in current thread and get the collectFuture, wait collectFuture in "broadcast-exchange" thread ### Why are the changes needed? #31269 gives a partial fix to SPARK-33933, which is not a perfect solution. This changes can make sure the broadcast collect job is submitted before shuffle map jobs. #31269 ensure the calling of materialize() of BroadcastQueryStage is before ShuffleQueryStage. In BroadcastQueryStage's materialize(), doPrepare() will call relationFuture, which will submit collect job before return the future. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Add UT -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
