Github user mateiz commented on a diff in the pull request:
https://github.com/apache/incubator-spark/pull/641#discussion_r10003953
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -272,8 +272,10 @@ class DAGScheduler(
if (mapOutputTracker.has(shuffleDep.shuffleId)) {
val serLocs =
mapOutputTracker.getSerializedMapOutputStatuses(shuffleDep.shuffleId)
val locs = MapOutputTracker.deserializeMapStatuses(serLocs)
- for (i <- 0 until locs.size) stage.outputLocs(i) = List(locs(i))
- stage.numAvailableOutputs = locs.size
+ for (i <- 0 until locs.size) {
+ stage.outputLocs(i) = Option(locs(i)).toList // locs(i) will be
null if missing
--- End diff --
Yeah, the problem is that the MapOutputTracker returns an array of
MapStatus, where position i of the array is supposed to be the location of the
output of task i. This array contains nulls initially to denote a missing
location, because a MapStatus can only indicate a real location. So I think
that either way we'll have to check for some "missing output" condition.
With this fix I believe we won't be launching reduce stages when some maps
are missing anymore, so we should be okay.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
