gczsjdy commented on issue #24462: [SPARK-26268][CORE] Do not resubmit tasks when executors are lost URL: https://github.com/apache/spark/pull/24462#issuecomment-510766013 @yifeih Thank you, I understand now. But can your way (making `MapStatus` able to contain an empty location in order to not resubmit map stage tasks) deals with this condition: the `MapStatus` returned contains a valid location, at the same time, we don't want the Driver to unregister this shuffle output when executors lost(Maybe due to the map output is also backed up in DFS)? In other words, what Driver decides to do when invalidating an executor(what this PR works on) and how the Executors tell Driver the `MapStatus`(with or without a location) are two different things.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
