gczsjdy commented on issue #24462: [SPARK-26268][CORE] Do not resubmit tasks when executors are lost URL: https://github.com/apache/spark/pull/24462#issuecomment-514078853 @squito I met with a condition that cannot be satisfied without this PR: - On map side, all shuffle files are written to remote Hadoop filesystem, there isn't any shuffle files managed by `BlockManager`s. So I simply should make `ShuffleWriters` return `MapStatus.location == null`? No, because it cannot fulfil the need during shuffle write. - On reduce side, I want to read the shuffle index files from the cache on the executors who wrote them, so I need the `BlockManagerId` in the `MapStatus` to tell each reducer which executor to find. This is what I mentioned: > what Driver decides to do when invalidating an executor(what this PR works on) and how the Executors tell Driver the MapStatus(with or without a location) are two different things.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
