mayurdb opened a new pull request, #42950:
URL: https://github.com/apache/spark/pull/42950

   ### What changes were proposed in this pull request?
   [SPARK-25342](https://issues.apache.org/jira/browse/SPARK-25342) Added a 
support for rolling back shuffle map stage so that all tasks of the stage can 
be retried when the stage output is indeterminate. This is done by clearing all 
map outputs at the time of stage submission. This approach workouts well except 
for this case:
   
   Assume both Shuffle 1 and 2 are indeterminate
   
   ShuffleMapStage1 ––> Shuffle 1 ---–> ShuffleMapStage2 ----> Shuffle 2 ----> 
ResultStage
   
   - ShuffleMapStage1 is complete
   - A task from ShuffleMapStage2 fails with FetchFailed. Other tasks are still 
running
   - Both ShuffleMapStage1 and ShuffleMapStage2 are retried
   - ShuffleMapStage1 is retried and completes
   - ShuffleMapStage2 reattempt is scheduled for execution
   - Before all tasks of ShuffleMapStage2 reattempt could finish, one/more 
laggard tasks from the original attempt of ShuffleMapStage2 finish and 
ShuffleMapStage2 also gets marked as complete
   - Result Stage gets scheduled and finishes
   
   After this change, such laggard tasks from the old attempt of the 
indeterminate stage will be ignored 
   
   ### Why are the changes needed?
   This can give wrong result when indeterminate stages needs to be retried 
under the circumstances mentioned above
   
   
   ### Does this PR introduce _any_ user-facing change?
   No
   
   
   ### How was this patch tested?
   A new test case
   
   ### Was this patch authored or co-authored using generative AI tooling?
   No
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

Reply via email to