Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/4055#discussion_r33965943 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -1193,8 +1193,10 @@ class DAGScheduler( // TODO: This will be really slow if we keep accumulating shuffle map stages for ((shuffleId, stage) <- shuffleToMapStage) { stage.removeOutputsOnExecutor(execId) - val locs = stage.outputLocs.map(list => if (list.isEmpty) null else list.head).toArray - mapOutputTracker.registerMapOutputs(shuffleId, locs, changeEpoch = true) + if (!runningStages.contains(stage)) { --- End diff -- @suyanNone I'm not entirely sure I understand -- let me try to explain it back to you, tell me if this is correct: this change is not for **correctness**, its more for efficiency. There is no point in registering map output for running stages, since we'll only register some subset of the stages full output. Eventually when the stage completes, the full set of map outputs will be registered, and that is all that matters anyway. So there is no point registering the map output here. Does that sound right?
--- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org