jiangxb1987 commented on issue #26614: [SPARK-29976][CORE] Trigger speculation for stages with too few tasks URL: https://github.com/apache/spark/pull/26614#issuecomment-561410074 I think I get your concern now, we might have two stages running concurrently, the expected task duration for the first stage could be 15mins and for the second stage it could be 1hr. Thus if we set the `speculationTaskDurationThresOpt` to 30mins then tasks from the second stage would all get speculated which is not desired. However I don't see why this is related to comparing the `speculationTaskDurationThresOpt` with `unfinished tasks` versus `total tasks`. Even if we choose `total tasks` instead of `unfinished tasks`, it can still happen that one stage contains only one task, but the task duration is actually expected to be longer than `speculationTaskDurationThresOpt`, then a speculative task shall get launched anyway.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
