[ https://issues.apache.org/jira/browse/SPARK-31646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17418366#comment-17418366 ]
Yongjun Zhang commented on SPARK-31646: --------------------------------------- HI [~mauzhang], Thanks a lot for your answers and sorry for late reply. I think I understand it better now why you are doing this change: the registeredConnections metrics added in ExternalShuffleBlockHandler was not used. However, the one added to TransportContext is used, see in YarnShuffleService.java: {code:java} // register metrics on the block handler into the Node Manager's metrics system. blockHandler.getAllMetrics().getMetrics().put("numRegisteredConnections", shuffleServer.getRegisteredConnections()); YarnShuffleServiceMetrics serviceMetrics = new YarnShuffleServiceMetrics(blockHandler.getAllMetrics()); MetricsSystemImpl metricsSystem = (MetricsSystemImpl) DefaultMetricsSystem.instance(); metricsSystem.register( "sparkShuffleService", "Metrics on the Spark Shuffle Service", serviceMetrics); logger.info("Registered metrics with Hadoop's DefaultMetricsSystem"); {code} The TransportContext version of registeredConnections is retrieved by "shuffleServer.getRegisteredConnections())" in the above code. That means both the activeConnections and registeredConnections are still available with your change. Is that your expectation? If my understanding is correct, we can either derive "registeredConnections - activeConnections" as the backlogged connections, or we can add a new metrics as backloggedConnection to have the value of "registeredConnections - activeConnections" . What do you think? Thanks! > Remove unused registeredConnections counter from ShuffleMetrics > --------------------------------------------------------------- > > Key: SPARK-31646 > URL: https://issues.apache.org/jira/browse/SPARK-31646 > Project: Spark > Issue Type: Improvement > Components: Deploy, Shuffle, Spark Core > Affects Versions: 3.0.0 > Reporter: Manu Zhang > Assignee: Manu Zhang > Priority: Minor > Fix For: 3.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org