turboFei commented on PR #3144: URL: https://github.com/apache/celeborn/pull/3144#issuecomment-2746539744
> > For celeborn shuffle fetch failure, it will rerun the whole stage, so it is fine to cancel all the running tasks. > > ```shell > shuffleMapStage -> FailedStage > ``` > > In current implementation, when fetch failure happens without indeterministic shuffle map output, Spark will only rerun the whole shuffleMapStage and tasks that are not finished in the FailedStage. And tasks that finished during rerun of shuffleMapStage will still be counted as successful tasks and will not be rerun in retry of FailedStage. > >  > > https://github.com/apache/spark/blob/0670e4f7945ae4935261ed1f45db7ede79aca127/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L1928-L1936 > > Maybe we need to add a condition to only cancel still-running tasks which fetch from indeterministic shuffle output. Please correct me if I have any misunderstanding. This function is for the zombie task case. But for celeborn stage rerun, it would rerun the whole tasks. It means that: 1. if we do not cancel the running tasks in the stage, it would occupy the resources, and the tasks in last stage to re-run might wait these tasks finished before launching 2. The tasks might be keep running with the new stage attempts tasks at the same time, it double the computing resource. So, I think for celeborn stage rerun, cancel all the tasks directly should be fine. @Z1Wu -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
