[jira] [Commented] (SPARK-14151) Propose to refactor and expose Metrics Sink and Source interface
[ https://issues.apache.org/jira/browse/SPARK-14151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16082508#comment-16082508 ] Saisai Shao commented on SPARK-14151: - Thanks [~jiangxb] for your response. I'm not sure if it requires a standard SPIP for this JIRA, because this is a simple improvement and we just only expose APIs, not create new APIs. I would think only big new changes require SPIP, there's not so many things I could address in the SPIP. Would like to hear other's comment. > Propose to refactor and expose Metrics Sink and Source interface > > > Key: SPARK-14151 > URL: https://issues.apache.org/jira/browse/SPARK-14151 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Reporter: Saisai Shao >Priority: Minor > > MetricsSystem is designed for plug-in different sources and sinks, user could > write their own sources and sinks and configured through metrics.properties, > MetricsSystem will register it through reflection. But current Source and > Sink interface is private, which means user cannot create their own sources > and sinks unless using the same package. > So here propose to expose source and sink interface, this will let user build > and maintain their own source and sink, alleviate the maintenance overhead of > spark codebase. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-14151) Propose to refactor and expose Metrics Sink and Source interface
[ https://issues.apache.org/jira/browse/SPARK-14151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16082283#comment-16082283 ] Jiang Xingbo commented on SPARK-14151: -- Since this is purposed to add a set of public API, it would make more sense to start a SPIP for the topic. [~jerryshao] Would you like to do that? > Propose to refactor and expose Metrics Sink and Source interface > > > Key: SPARK-14151 > URL: https://issues.apache.org/jira/browse/SPARK-14151 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Reporter: Saisai Shao >Priority: Minor > > MetricsSystem is designed for plug-in different sources and sinks, user could > write their own sources and sinks and configured through metrics.properties, > MetricsSystem will register it through reflection. But current Source and > Sink interface is private, which means user cannot create their own sources > and sinks unless using the same package. > So here propose to expose source and sink interface, this will let user build > and maintain their own source and sink, alleviate the maintenance overhead of > spark codebase. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-14151) Propose to refactor and expose Metrics Sink and Source interface
[ https://issues.apache.org/jira/browse/SPARK-14151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15839567#comment-15839567 ] Marco Buccini commented on SPARK-14151: --- Hello everyone, Can anybody give a short status about this task? I think it would be really great for Spark to provide a public API for metrics. > Propose to refactor and expose Metrics Sink and Source interface > > > Key: SPARK-14151 > URL: https://issues.apache.org/jira/browse/SPARK-14151 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Reporter: Saisai Shao >Priority: Minor > > MetricsSystem is designed for plug-in different sources and sinks, user could > write their own sources and sinks and configured through metrics.properties, > MetricsSystem will register it through reflection. But current Source and > Sink interface is private, which means user cannot create their own sources > and sinks unless using the same package. > So here propose to expose source and sink interface, this will let user build > and maintain their own source and sink, alleviate the maintenance overhead of > spark codebase. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-14151) Propose to refactor and expose Metrics Sink and Source interface
[ https://issues.apache.org/jira/browse/SPARK-14151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15409535#comment-15409535 ] G Lingle commented on SPARK-14151: -- Hi, This is quite an important Jira for us, as we have custom metrics to expose. What's the status on this? What release will this be in? Has another metrics solution been proposed/implemented? Thanks! > Propose to refactor and expose Metrics Sink and Source interface > > > Key: SPARK-14151 > URL: https://issues.apache.org/jira/browse/SPARK-14151 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Reporter: Saisai Shao >Priority: Minor > > MetricsSystem is designed for plug-in different sources and sinks, user could > write their own sources and sinks and configured through metrics.properties, > MetricsSystem will register it through reflection. But current Source and > Sink interface is private, which means user cannot create their own sources > and sinks unless using the same package. > So here propose to expose source and sink interface, this will let user build > and maintain their own source and sink, alleviate the maintenance overhead of > spark codebase. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-14151) Propose to refactor and expose Metrics Sink and Source interface
[ https://issues.apache.org/jira/browse/SPARK-14151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15213954#comment-15213954 ] Apache Spark commented on SPARK-14151: -- User 'jerryshao' has created a pull request for this issue: https://github.com/apache/spark/pull/11994 > Propose to refactor and expose Metrics Sink and Source interface > > > Key: SPARK-14151 > URL: https://issues.apache.org/jira/browse/SPARK-14151 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Reporter: Saisai Shao >Priority: Minor > > MetricsSystem is designed for plug-in different sources and sinks, user could > write their own sources and sinks and configured through metrics.properties, > MetricsSystem will register it through reflection. But current Source and > Sink interface is private, which means user cannot create their own sources > and sinks unless using the same package. > So here propose to expose source and sink interface, this will let user build > and maintain their own source and sink, alleviate the maintenance overhead of > spark codebase. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-14151) Propose to refactor and expose Metrics Sink and Source interface
[ https://issues.apache.org/jira/browse/SPARK-14151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15211640#comment-15211640 ] Saisai Shao commented on SPARK-14151: - Yes, sure I will add the doc. > Propose to refactor and expose Metrics Sink and Source interface > > > Key: SPARK-14151 > URL: https://issues.apache.org/jira/browse/SPARK-14151 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Reporter: Saisai Shao >Priority: Minor > > MetricsSystem is designed for plug-in different sources and sinks, user could > write their own sources and sinks and configured through metrics.properties, > MetricsSystem will register it through reflection. But current Source and > Sink interface is private, which means user cannot create their own sources > and sinks unless using the same package. > So here propose to expose source and sink interface, this will let user build > and maintain their own source and sink, alleviate the maintenance overhead of > spark codebase. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-14151) Propose to refactor and expose Metrics Sink and Source interface
[ https://issues.apache.org/jira/browse/SPARK-14151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15211638#comment-15211638 ] Pete Robbins commented on SPARK-14151: -- Agreed that is the way to go. I was also working on it but will leave it to you. In addition we will need to document the interfaces and use. > Propose to refactor and expose Metrics Sink and Source interface > > > Key: SPARK-14151 > URL: https://issues.apache.org/jira/browse/SPARK-14151 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Reporter: Saisai Shao >Priority: Minor > > MetricsSystem is designed for plug-in different sources and sinks, user could > write their own sources and sinks and configured through metrics.properties, > MetricsSystem will register it through reflection. But current Source and > Sink interface is private, which means user cannot create their own sources > and sinks unless using the same package. > So here propose to expose source and sink interface, this will let user build > and maintain their own source and sink, alleviate the maintenance overhead of > spark codebase. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-14151) Propose to refactor and expose Metrics Sink and Source interface
[
https://issues.apache.org/jira/browse/SPARK-14151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15211634#comment-15211634
]
Saisai Shao commented on SPARK-14151:
-
I'm currently working on the second choice, to offer external developers
another choice of constructor without {{SecurityManager}}, that will be more
consistent with the current code.
> Propose to refactor and expose Metrics Sink and Source interface
>
>
> Key: SPARK-14151
> URL: https://issues.apache.org/jira/browse/SPARK-14151
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Reporter: Saisai Shao
>Priority: Minor
>
> MetricsSystem is designed for plug-in different sources and sinks, user could
> write their own sources and sinks and configured through metrics.properties,
> MetricsSystem will register it through reflection. But current Source and
> Sink interface is private, which means user cannot create their own sources
> and sinks unless using the same package.
> So here propose to expose source and sink interface, this will let user build
> and maintain their own source and sink, alleviate the maintenance overhead of
> spark codebase.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
