itskals commented on issue #26975: [SPARK-30325][CORE] Stage retry and executor crash cause app hung up forever URL: https://github.com/apache/spark/pull/26975#issuecomment-568664326 I was of the opinion that the when a task is started by a previous stage and still in progress, no subsequent retries from other stage attempt must be made, unless it is fate is known. To know if the partition is already assigned to some task, the MapStatus entry for the partition could denote the intermediate step.(As of now MapStatusEntry is either null or filled, kind of boolean. I think we can have the third stage). I am not sure if there is any issue in making such assumption. I can try to elaborate the design if needed. By this proposed model, we can have the compute resources also saved(no need to start a redundant computation if one stage attempt is already working on it). However, we allow speculation as its within same stage attempt.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
