Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21758#discussion_r203617306
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -1349,6 +1339,48 @@ class DAGScheduler(
s"longer running")
}
+ if (mapStage.rdd.isBarrier()) {
+ // Mark all the map as broken in the map stage, to ensure
retry all the tasks on
+ // resubmitted stage attempt.
+ mapOutputTracker.unregisterAllMapOutput(shuffleId)
+ } else if (mapId != -1) {
+ // Mark the map whose fetch failed as broken in the map stage
+ mapOutputTracker.unregisterMapOutput(shuffleId, mapId,
bmAddress)
+ }
+
+ if (failedStage.rdd.isBarrier()) {
+ failedStage match {
+ case mapStage: ShuffleMapStage =>
--- End diff --
please pick a different name. `mapStage` is already used before..
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
