[jira] [Commented] (SPARK-21334) Fix metrics for external shuffle service
[ https://issues.apache.org/jira/browse/SPARK-21334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095204#comment-16095204 ] Raajay Viswanathan commented on SPARK-21334: I think SPARK-18364 aims to implement metrics in YarnShuffleService. The current issue builds on existing work (SPARK-16405) for ExternalShuffleService and implements a reporting service that was earlier missing. > Fix metrics for external shuffle service > > > Key: SPARK-21334 > URL: https://issues.apache.org/jira/browse/SPARK-21334 > Project: Spark > Issue Type: Bug > Components: Shuffle >Affects Versions: 2.1.1 >Reporter: Raajay Viswanathan > Labels: external-shuffle-service > Original Estimate: 168h > Remaining Estimate: 168h > > SPARK-16405 introduced metrics for external shuffle service. However, as it > is currently there are two issues. > 1. The shuffle service metrics system does not report values ever. > 2. -The current method for determining "blockTransferRate" is incorrect. The > entire block is assumed to be transferred once the OpenBlocks message if > processed. The actual data fetch from the disk and the succeeding transfer > over the wire happens much later when MessageEncoder invokes encode on > ChunkFetchSuccess message.- -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21334) Fix metrics for external shuffle service
[ https://issues.apache.org/jira/browse/SPARK-21334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095095#comment-16095095 ] Robert Kruszewski commented on SPARK-21334: --- I think this is a dupe of https://issues.apache.org/jira/browse/SPARK-18364 > Fix metrics for external shuffle service > > > Key: SPARK-21334 > URL: https://issues.apache.org/jira/browse/SPARK-21334 > Project: Spark > Issue Type: Bug > Components: Shuffle >Affects Versions: 2.1.1 >Reporter: Raajay Viswanathan > Labels: external-shuffle-service > Original Estimate: 168h > Remaining Estimate: 168h > > SPARK-16405 introduced metrics for external shuffle service. However, as it > is currently there are two issues. > 1. The shuffle service metrics system does not report values ever. > 2. -The current method for determining "blockTransferRate" is incorrect. The > entire block is assumed to be transferred once the OpenBlocks message if > processed. The actual data fetch from the disk and the succeeding transfer > over the wire happens much later when MessageEncoder invokes encode on > ChunkFetchSuccess message.- -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21334) Fix metrics for external shuffle service
[ https://issues.apache.org/jira/browse/SPARK-21334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095063#comment-16095063 ] Apache Spark commented on SPARK-21334: -- User 'raajay' has created a pull request for this issue: https://github.com/apache/spark/pull/18690 > Fix metrics for external shuffle service > > > Key: SPARK-21334 > URL: https://issues.apache.org/jira/browse/SPARK-21334 > Project: Spark > Issue Type: Bug > Components: Shuffle >Affects Versions: 2.1.1 >Reporter: Raajay Viswanathan > Labels: external-shuffle-service > Original Estimate: 168h > Remaining Estimate: 168h > > SPARK-16405 introduced metrics for external shuffle service. However, as it > is currently there are two issues. > 1. The shuffle service metrics system does not report values ever. > 2. -The current method for determining "blockTransferRate" is incorrect. The > entire block is assumed to be transferred once the OpenBlocks message if > processed. The actual data fetch from the disk and the succeeding transfer > over the wire happens much later when MessageEncoder invokes encode on > ChunkFetchSuccess message.- -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21334) Fix metrics for external shuffle service
[ https://issues.apache.org/jira/browse/SPARK-21334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095056#comment-16095056 ] Raajay Viswanathan commented on SPARK-21334: [~jerryshao] I am using external shuffle service in a standalone manner. Also, I am mistaken in the claim that the "blockTransferRate" is incorrect. It is done in a proper manner in the latest version of Spark; it was fixed as part of [SPARK-20994]. > Fix metrics for external shuffle service > > > Key: SPARK-21334 > URL: https://issues.apache.org/jira/browse/SPARK-21334 > Project: Spark > Issue Type: Bug > Components: Shuffle >Affects Versions: 2.1.1 >Reporter: Raajay Viswanathan > Labels: external-shuffle-service > Original Estimate: 168h > Remaining Estimate: 168h > > SPARK-16405 introduced metrics for external shuffle service. However, as it > is currently there are two issues. > 1. The shuffle service metrics system does not report values ever. > 2. -The current method for determining "blockTransferRate" is incorrect. The > entire block is assumed to be transferred once the OpenBlocks message if > processed. The actual data fetch from the disk and the succeeding transfer > over the wire happens much later when MessageEncoder invokes encode on > ChunkFetchSuccess message.- -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21334) Fix metrics for external shuffle service
[ https://issues.apache.org/jira/browse/SPARK-21334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16079389#comment-16079389 ] Saisai Shao commented on SPARK-21334: - Are you using external shuffle service with YARN? If so I think currently it cannot be supported, we could improve the code to make it work (either through Hadoop metrics system or Spark's own one). > Fix metrics for external shuffle service > > > Key: SPARK-21334 > URL: https://issues.apache.org/jira/browse/SPARK-21334 > Project: Spark > Issue Type: Bug > Components: Shuffle >Affects Versions: 2.1.1 >Reporter: Raajay Viswanathan > Labels: external-shuffle-service > Original Estimate: 168h > Remaining Estimate: 168h > > SPARK-16405 introduced metrics for external shuffle service. However, as it > is currently there are two issues. > 1. The shuffle service metrics system does not report values ever. > 2. The current method for determining "blockTransferRate" is incorrect. The > entire block is assumed to be transferred once the OpenBlocks message if > processed. The actual data fetch from the disk and the succeeding transfer > over the wire happens much later when MessageEncoder invokes encode on > ChunkFetchSuccess message. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
