xuanyuanking edited a comment on issue #24110: [SPARK-25341][Core] Support rolling back a shuffle map stage and re-generate the shuffle files URL: https://github.com/apache/spark/pull/24110#issuecomment-474424797 ``` To help other people review this patch, can you add the following information in your PR description? ``` Thanks Wenchen, the PR description updated. ``` A side question: can we skip the temp file when writing shuffle files? ``` ~~Let me check this, it's related to https://issues.apache.org/jira/browse/SPARK-8029.~~ As current implement, the answer is no, because we only keep attempt id for the indeterminate stage. For further checking, I think we can achieve the goal of skipping the temp file if we need. We need do two more things below: - All the block id should be extended by adding attempt id, not only the indeterminate ones. - We should keep the mapping of (stage, partition) -> attemptId in MapOutputTracker, because for the determinate stage, we will meet different stage attemptId in single stage while fetch fail happened. Currently implement, just keep the mapping of shuffleId to attemptId for each stage and only use them in indeterminate stage.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
