Andrew Or created SPARK-13054: --------------------------------- Summary: Always post TaskEnd event for tasks in cancelled stages Key: SPARK-13054 URL: https://issues.apache.org/jira/browse/SPARK-13054 Project: Spark Issue Type: Bug Components: Scheduler Affects Versions: 1.0.0 Reporter: Andrew Or Assignee: Andrew Or
{code} // The success case is dealt with separately below. // TODO: Why post it only for failed tasks in cancelled stages? Clarify semantics here. if (event.reason != Success) { val attemptId = task.stageAttemptId listenerBus.post(SparkListenerTaskEnd( stageId, attemptId, taskType, event.reason, event.taskInfo, taskMetrics)) } {code} Today we only post task end events for canceled stages if the task failed. There is no reason why we shouldn't just post it for all the tasks, including the ones that succeeded. If we do that we will be able to simplify another branch in the DAGScheduler, which needs a lot of simplification. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org