[ https://issues.apache.org/jira/browse/SPARK-21334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16095056#comment-16095056 ]
Raajay Viswanathan commented on SPARK-21334: -------------------------------------------- [~jerryshao] I am using external shuffle service in a standalone manner. Also, I am mistaken in the claim that the "blockTransferRate" is incorrect. It is done in a proper manner in the latest version of Spark; it was fixed as part of [SPARK-20994]. > Fix metrics for external shuffle service > ---------------------------------------- > > Key: SPARK-21334 > URL: https://issues.apache.org/jira/browse/SPARK-21334 > Project: Spark > Issue Type: Bug > Components: Shuffle > Affects Versions: 2.1.1 > Reporter: Raajay Viswanathan > Labels: external-shuffle-service > Original Estimate: 168h > Remaining Estimate: 168h > > SPARK-16405 introduced metrics for external shuffle service. However, as it > is currently there are two issues. > 1. The shuffle service metrics system does not report values ever. > 2. -The current method for determining "blockTransferRate" is incorrect. The > entire block is assumed to be transferred once the OpenBlocks message if > processed. The actual data fetch from the disk and the succeeding transfer > over the wire happens much later when MessageEncoder invokes encode on > ChunkFetchSuccess message.- -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org