[
https://issues.apache.org/jira/browse/SPARK-44209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17744500#comment-17744500
]
Deependra Patel commented on SPARK-44209:
-----------------------------------------
Created PR - [https://github.com/apache/spark/pull/42071]
> Expose amount of shuffle data available on the node
> ---------------------------------------------------
>
> Key: SPARK-44209
> URL: https://issues.apache.org/jira/browse/SPARK-44209
> Project: Spark
> Issue Type: New Feature
> Components: Shuffle
> Affects Versions: 3.4.1
> Reporter: Deependra Patel
> Priority: Trivial
>
> [ShuffleMetrics|https://github.com/apache/spark/blob/43f7a86a05ad8c7ec7060607e43d9ca4d0fe4166/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalBlockHandler.java#L318]
> doesn't have metrics like
> "totalShuffleDataBytes" and "numAppsWithShuffleData", these metrics are per
> node published by External Shuffle Service.
>
> Adding these metrics would help in -
> 1. Deciding if we can decommission the node if no shuffle data present
> 2. Better live monitoring of customer's workload to see if there is skewed
> shuffle data present on the node
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
