[ https://issues.apache.org/jira/browse/SPARK-7171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14521131#comment-14521131 ]
Jacek Lewandowski commented on SPARK-7171: ------------------------------------------ https://github.com/apache/spark/pull/5805 > Allow for more flexible use of metric sources > --------------------------------------------- > > Key: SPARK-7171 > URL: https://issues.apache.org/jira/browse/SPARK-7171 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 1.3.1 > Reporter: Jacek Lewandowski > Priority: Minor > > With the current API, the user is allowed to add a custom metric source by > providing its class in metrics configuration. Metrics themselves are provided > by Codahale and therefore they allow to register multiple metrics in a single > source. Basically we can break the available types of metrics into two types: > "push" and "pull" - by push metrics I mean that some execution code updates > the metric by itself either periodically or every n events. On the other > hand, the pull metrics include some function which pulls the data from the > execution environment, when triggered. > h5.Problem > The metric source is instantiated and registered during initialisation. Then, > the user has no way to access the instantiated object. It is also almost > impossible to access the execution environment of the current task. > Therefore, the user who wanted to provide his own {{RDD}} implementation > along with a dedicated metrics source, would find it very difficult to do > this in a safe, concise and elegant way. > h5.Proposed solution > At least, for the "push" metrics, it would be nice to be able to retrieve the > metrics source of particular type or with particular id from {{TaskContext}}. > It would allow custom tasks to update various metrics and would greatly > improve the usability of metrics. > This could be achieved quite easily since {{TaskContext}} is created by > {{Executor}}, which has access to the metrics system, it would inject some > method to retrieve the particular metrics source. > This solution wouldn't change the current API, but just introduce one more > method in {{TaskContext}}. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org