[ https://issues.apache.org/jira/browse/SPARK-21334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16079389#comment-16079389 ]
Saisai Shao edited comment on SPARK-21334 at 7/9/17 12:43 AM: -------------------------------------------------------------- Are you using external shuffle service with YARN? If so I think currently it is not supported, we could improve the code to make it work (either through Hadoop metrics system or Spark's own one). was (Author: jerryshao): Are you using external shuffle service with YARN? If so I think currently it cannot be supported, we could improve the code to make it work (either through Hadoop metrics system or Spark's own one). > Fix metrics for external shuffle service > ---------------------------------------- > > Key: SPARK-21334 > URL: https://issues.apache.org/jira/browse/SPARK-21334 > Project: Spark > Issue Type: Bug > Components: Shuffle > Affects Versions: 2.1.1 > Reporter: Raajay Viswanathan > Labels: external-shuffle-service > Original Estimate: 168h > Remaining Estimate: 168h > > SPARK-16405 introduced metrics for external shuffle service. However, as it > is currently there are two issues. > 1. The shuffle service metrics system does not report values ever. > 2. The current method for determining "blockTransferRate" is incorrect. The > entire block is assumed to be transferred once the OpenBlocks message if > processed. The actual data fetch from the disk and the succeeding transfer > over the wire happens much later when MessageEncoder invokes encode on > ChunkFetchSuccess message. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org