[
https://issues.apache.org/jira/browse/SPARK-21334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16095056#comment-16095056
]
Raajay Viswanathan commented on SPARK-21334:
--------------------------------------------
[~jerryshao] I am using external shuffle service in a standalone manner.
Also, I am mistaken in the claim that the "blockTransferRate" is incorrect. It
is done in a proper manner in the latest version of Spark; it was fixed as part
of [SPARK-20994].
> Fix metrics for external shuffle service
> ----------------------------------------
>
> Key: SPARK-21334
> URL: https://issues.apache.org/jira/browse/SPARK-21334
> Project: Spark
> Issue Type: Bug
> Components: Shuffle
> Affects Versions: 2.1.1
> Reporter: Raajay Viswanathan
> Labels: external-shuffle-service
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> SPARK-16405 introduced metrics for external shuffle service. However, as it
> is currently there are two issues.
> 1. The shuffle service metrics system does not report values ever.
> 2. -The current method for determining "blockTransferRate" is incorrect. The
> entire block is assumed to be transferred once the OpenBlocks message if
> processed. The actual data fetch from the disk and the succeeding transfer
> over the wire happens much later when MessageEncoder invokes encode on
> ChunkFetchSuccess message.-
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
