[
https://issues.apache.org/jira/browse/SPARK-3577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15563038#comment-15563038
]
Gaoxiang Liu edited comment on SPARK-3577 at 10/11/16 4:49 AM:
---------------------------------------------------------------
[~rxin] I find that the spill size metrics is already added in
https://github.com/apache/spark/commit/bb8098f203e61111faddf2e1a04b03d62037e6c7#diff-1bd3dc38f6306e0a822f93d62c32b1d0,
and I have confirm in the UI. (please refer to the attachment of this JIRA -
https://issues.apache.org/jira/secure/attachment/12832515/spill_size.jpg)
Also, we noticed that it's wield that the spill size is somehow not reported in
the reducer , but reported in the mapper.
Back to the previous question, for the spill time, if it's still relevant to
add, then I plan to work on it if there is no objections.
was (Author: dreamworks007):
I find that the spill size metrics is already added in
https://github.com/apache/spark/commit/bb8098f203e61111faddf2e1a04b03d62037e6c7#diff-1bd3dc38f6306e0a822f93d62c32b1d0,
and I have confirm in the UI. (please refer to the attachment of this JIRA -
https://issues.apache.org/jira/secure/attachment/12832515/spill_size.jpg)
Also, we noticed that it's wield that the spill size is somehow not reported in
the reducer , but reported in the mapper.
Back to the previous question, for the spill time, if it's still relevant to
add, then I plan to work on it if there is no objections.
> Add task metric to report spill time
> ------------------------------------
>
> Key: SPARK-3577
> URL: https://issues.apache.org/jira/browse/SPARK-3577
> Project: Spark
> Issue Type: Bug
> Components: Shuffle, Spark Core
> Affects Versions: 1.1.0
> Reporter: Kay Ousterhout
> Priority: Minor
> Attachments: spill_size.jpg
>
>
> The {{ExternalSorter}} passes its own {{ShuffleWriteMetrics}} into
> {{ExternalSorter}}. The write time recorded in those metrics is never used.
> We should probably add task metrics to report this spill time, since for
> shuffles, this would have previously been reported as part of shuffle write
> time (with the original hash-based sorter).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
