Github user xuanyuanking commented on a diff in the pull request:
https://github.com/apache/spark/pull/20930#discussion_r182296297
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala ---
@@ -750,6 +752,10 @@ private[spark] class TaskSetManager(
if (tasksSuccessful == numTasks) {
isZombie = true
}
+ } else if (fetchFailedTaskIndexSet.contains(index)) {
+ logInfo("Ignoring task-finished event for " + info.id + " in stage "
+ taskSet.id +
+ " because task " + index + " has already failed by FetchFailed")
+ return
--- End diff --
Yep, as @cloud-fan 's suggestion, handle this in `DAGScheduler` is a better
choice.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
