[ https://issues.apache.org/jira/browse/SPARK-11373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14978322#comment-14978322 ]
Steve Loughran commented on SPARK-11373: ---------------------------------------- # This has tangible benefit for the SPARK-1537 YARN ATS binding, because connectivity failures, GET performance and similar do surface. There are some {{AtomicLong}} counters in its {{YarnHistoryProvider}}, but I'm not planning to add counters and metrics until after that is checked in. # All providers will benefit from the standard JVM performance counters, GC &c. # the FS history provider could also track time to list and load histories; time of last refresh, time to load most recent history, etc —information needed to identify where an unresponsive UI is getting its problems from. > Add metrics to the History Server and providers > ----------------------------------------------- > > Key: SPARK-11373 > URL: https://issues.apache.org/jira/browse/SPARK-11373 > Project: Spark > Issue Type: New Feature > Components: Spark Core > Affects Versions: 1.5.1 > Reporter: Steve Loughran > > The History server doesn't publish metrics about JVM load or anything from > the history provider plugins. This means that performance problems from > massive job histories aren't visible to management tools, and nor are any > provider-generated metrics such as time to load histories, failed history > loads, the number of connectivity failures talking to remote services, etc. > If the history server set up a metrics registry and offered the option to > publish its metrics, then management tools could view this data. > # the metrics registry would need to be passed down to the instantiated > {{ApplicationHistoryProvider}}, in order for it to register its metrics. > # if the codahale metrics servlet were registered under a path such as > {{/metrics}}, the values would be visible as HTML and JSON, without the need > for management tools. > # Integration tests could also retrieve the JSON-formatted data and use it as > part of the test suites. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org