Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/19160#discussion_r139795417
--- Diff: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
---
@@ -248,6 +251,16 @@ private[spark] class BlockManager(
logInfo(s"Initialized BlockManager: $blockManagerId")
}
+ def shuffleMetricsSource: Source = {
+ import BlockManager._
+
+ if (externalShuffleServiceEnabled) {
+ new ShuffleMetricsSource("ExternalShuffle",
shuffleClient.shuffleMetrics())
+ } else {
+ new ShuffleMetricsSource("NettyBlockTransfer",
shuffleClient.shuffleMetrics())
--- End diff --
do you think we really need to distinguish these two cases? whether or not
you have the external shuffle service, this memory is still owned by the
executor JVM (its really only external on the remote end).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
