[
https://issues.apache.org/jira/browse/SPARK-20087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16451732#comment-16451732
]
Xianjin YE commented on SPARK-20087:
------------------------------------
cc [~jiangxb1987] [~irashid], I am going to send a new pr if you still think
this is the desired behaviour.
> Include accumulators / taskMetrics when sending TaskKilled to onTaskEnd
> listeners
> ---------------------------------------------------------------------------------
>
> Key: SPARK-20087
> URL: https://issues.apache.org/jira/browse/SPARK-20087
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 2.1.0
> Reporter: Charles Lewis
> Priority: Major
>
> When tasks end due to an ExceptionFailure, subscribers to onTaskEnd receive
> accumulators / task metrics for that task, if they were still available.
> These metrics are not currently sent when tasks are killed intentionally,
> such as when a speculative retry finishes, and the original is killed (or
> vice versa). Since we're killing these tasks ourselves, these metrics should
> almost always exist, and we should treat them the same way as we treat
> ExceptionFailures.
> Sending these metrics with the TaskKilled end reason makes aggregation across
> all tasks in an app more accurate. This data can inform decisions about how
> to tune the speculation parameters in order to minimize duplicated work, and
> in general, the total cost of an app should include both successful and
> failed tasks, if that information exists.
> PR: https://github.com/apache/spark/pull/17422
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
