Victsm commented on pull request #32007: URL: https://github.com/apache/spark/pull/32007#issuecomment-850024225
In fact, I'm leaning towards taking both handling multiple app attempts and the race conditions out of this PR and fix them in a separate jira. The original scope of this ticket is to add support in DiskBlockManager to handle merge directories. We can still keep the app attempt ID in the merge dir directory path in this PR. However, for handling the registration of executors with a new attempt ID and the cleanup of metadata associated with the previous attempt ID, I think separating these changes out in a dedicated jira makes sense. With SPARK-32922 that @otterc is working on, we will be able to conduct end-to-end test/benchmark for the version of push-based shuffle that's currently being contributed upstream. I prefer to test these changes associated with handling multiple app attempts and the race conditions in such end-to-end test/benchmarks before committing the changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
