abellina commented on PR #43627: URL: https://github.com/apache/spark/pull/43627#issuecomment-1793562824
@mridulm thanks for the comments. I have published a SPIP here https://issues.apache.org/jira/browse/SPARK-45792 that aims to show the bigger picture. Without the change of initialization order in this PR, we couldn't carry out the SPIP linked, because the ShuffleManager is initialized really early in the Executors today. I split this up into a separate PR to not introduce too much change at once, but your point is well taken. I would like to hear your thoughts around the SPIP and how we can proceed. Note there is an alternative I can easily try and that is to instantiate a ShuffleManager wrapper, which would remove the change to the SparkEnv (we would instantiate the wrapper instead of the actual impl). We could then set the impl on this wrapper at a later time, when jars are localized and plugins are loaded. This felt a bit worse than the approach I have in this PR, but I am happy to hear opinions. Thanks again!! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
