[jira] [Commented] (SPARK-31683) Make Prometheus output consistent with DropWizard 4.1 result
[ https://issues.apache.org/jira/browse/SPARK-31683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128625#comment-17128625 ] Jorge Machado commented on SPARK-31683: --- It would be great if we could use the RPC backend from spark to be able to aggregate this into the driver only This way we only need to scrape the driver. I have made an implementation based on yours that registers it to consul. This way we can discover the yarn applications via consul for example... > Make Prometheus output consistent with DropWizard 4.1 result > > > Key: SPARK-31683 > URL: https://issues.apache.org/jira/browse/SPARK-31683 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 3.0.0 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Fix For: 3.0.0 > > > SPARK-29032 adds Prometheus support. > After that, SPARK-29674 upgraded DropWizard for JDK9+ support and causes > difference in output labels and number of keys. > > This issue aims to fix this inconsistency in Spark. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31683) Make Prometheus output consistent with DropWizard 4.1 result
[ https://issues.apache.org/jira/browse/SPARK-31683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128606#comment-17128606 ] Dongjoon Hyun commented on SPARK-31683: --- [~jomach]. No, SPARK-29032 exposed the prometheus metrics at Master/Worker/Driver. - https://github.com/apache/spark/pull/25769 I will have a Spark+AI Summit presentation for this whole picture. > Make Prometheus output consistent with DropWizard 4.1 result > > > Key: SPARK-31683 > URL: https://issues.apache.org/jira/browse/SPARK-31683 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 3.0.0 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Fix For: 3.0.0 > > > SPARK-29032 adds Prometheus support. > After that, SPARK-29674 upgraded DropWizard for JDK9+ support and causes > difference in output labels and number of keys. > > This issue aims to fix this inconsistency in Spark. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31683) Make Prometheus output consistent with DropWizard 4.1 result
[ https://issues.apache.org/jira/browse/SPARK-31683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17126671#comment-17126671 ] Jorge Machado commented on SPARK-31683: --- [~dongjoon] this only exporters the metrics from the driver right? > Make Prometheus output consistent with DropWizard 4.1 result > > > Key: SPARK-31683 > URL: https://issues.apache.org/jira/browse/SPARK-31683 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 3.0.0 >Reporter: Dongjoon Hyun >Assignee: Dongjoon Hyun >Priority: Major > Fix For: 3.0.0 > > > SPARK-29032 adds Prometheus support. > After that, SPARK-29674 upgraded DropWizard for JDK9+ support and causes > difference in output labels and number of keys. > > This issue aims to fix this inconsistency in Spark. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31683) Make Prometheus output consistent with DropWizard 4.1 result
[ https://issues.apache.org/jira/browse/SPARK-31683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105058#comment-17105058 ] Apache Spark commented on SPARK-31683: -- User 'dongjoon-hyun' has created a pull request for this issue: https://github.com/apache/spark/pull/28510 > Make Prometheus output consistent with DropWizard 4.1 result > > > Key: SPARK-31683 > URL: https://issues.apache.org/jira/browse/SPARK-31683 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 3.0.0 >Reporter: Dongjoon Hyun >Priority: Major > > SPARK-29032 adds Prometheus support. > After that, SPARK-29674 upgraded DropWizard for JDK9+ support and causes > difference in output labels and number of keys. > > This issue aims to fix this inconsistency in Spark. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-31683) Make Prometheus output consistent with DropWizard 4.1 result
[ https://issues.apache.org/jira/browse/SPARK-31683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105056#comment-17105056 ] Apache Spark commented on SPARK-31683: -- User 'dongjoon-hyun' has created a pull request for this issue: https://github.com/apache/spark/pull/28510 > Make Prometheus output consistent with DropWizard 4.1 result > > > Key: SPARK-31683 > URL: https://issues.apache.org/jira/browse/SPARK-31683 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 3.0.0 >Reporter: Dongjoon Hyun >Priority: Major > > SPARK-29032 adds Prometheus support. > After that, SPARK-29674 upgraded DropWizard for JDK9+ support and causes > difference in output labels and number of keys. > > This issue aims to fix this inconsistency in Spark. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
