[jira] [Commented] (SOLR-10654) Expose Metrics in Prometheus format DIRECTLY from Solr
[ https://issues.apache.org/jira/browse/SOLR-10654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17871844#comment-17871844 ] ASF subversion and git services commented on SOLR-10654: Commit 6b88470ba86d7a13ba811505681f4d5061c18829 in solr's branch refs/heads/branch_9x from Matthew Biscocho [ https://gitbox.apache.org/repos/asf?p=solr.git;h=6b88470ba86 ] SOLR-10654: Rename SolrPrometheusExporter to SolrPrometheusFormatter (#2620) Co-authored-by: Matthew Biscocho (cherry picked from commit cda87a14dad93ffbef0052f63008377797d930ab) > Expose Metrics in Prometheus format DIRECTLY from Solr > -- > > Key: SOLR-10654 > URL: https://issues.apache.org/jira/browse/SOLR-10654 > Project: Solr > Issue Type: Improvement > Components: metrics >Reporter: Keith Laban >Priority: Major > Labels: pull-request-available > Fix For: 9.7 > > Attachments: prometheus_metrics.txt > > Time Spent: 9h > Remaining Estimate: 0h > > Expose metrics via a `wt=prometheus` response type. > Example scape_config in prometheus.yml: > {code:java} > scrape_configs: > - job_name: 'solr' > metrics_path: '/solr/admin/metrics' > params: > wt: ["prometheus"] > static_configs: > - targets: ['localhost:8983'] > {code} > [Rationale|https://issues.apache.org/jira/browse/SOLR-11795?focusedCommentId=17261423&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17261423] > for having this despite the "Prometheus Exporter". They have different > strengths and weaknesses. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-10654) Expose Metrics in Prometheus format DIRECTLY from Solr
[ https://issues.apache.org/jira/browse/SOLR-10654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17871843#comment-17871843 ] ASF subversion and git services commented on SOLR-10654: Commit 0a36defd9a9077424a2f68714f7fff6658990980 in solr's branch refs/heads/branch_9_7 from Matthew Biscocho [ https://gitbox.apache.org/repos/asf?p=solr.git;h=0a36defd9a9 ] SOLR-10654: Rename SolrPrometheusExporter to SolrPrometheusFormatter (#2620) Co-authored-by: Matthew Biscocho (cherry picked from commit cda87a14dad93ffbef0052f63008377797d930ab) > Expose Metrics in Prometheus format DIRECTLY from Solr > -- > > Key: SOLR-10654 > URL: https://issues.apache.org/jira/browse/SOLR-10654 > Project: Solr > Issue Type: Improvement > Components: metrics >Reporter: Keith Laban >Priority: Major > Labels: pull-request-available > Fix For: 9.7 > > Attachments: prometheus_metrics.txt > > Time Spent: 9h > Remaining Estimate: 0h > > Expose metrics via a `wt=prometheus` response type. > Example scape_config in prometheus.yml: > {code:java} > scrape_configs: > - job_name: 'solr' > metrics_path: '/solr/admin/metrics' > params: > wt: ["prometheus"] > static_configs: > - targets: ['localhost:8983'] > {code} > [Rationale|https://issues.apache.org/jira/browse/SOLR-11795?focusedCommentId=17261423&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17261423] > for having this despite the "Prometheus Exporter". They have different > strengths and weaknesses. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-10654) Expose Metrics in Prometheus format DIRECTLY from Solr
[ https://issues.apache.org/jira/browse/SOLR-10654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17871841#comment-17871841 ] ASF subversion and git services commented on SOLR-10654: Commit cda87a14dad93ffbef0052f63008377797d930ab in solr's branch refs/heads/main from Matthew Biscocho [ https://gitbox.apache.org/repos/asf?p=solr.git;h=cda87a14dad ] SOLR-10654: Rename SolrPrometheusExporter to SolrPrometheusFormatter (#2620) Co-authored-by: Matthew Biscocho > Expose Metrics in Prometheus format DIRECTLY from Solr > -- > > Key: SOLR-10654 > URL: https://issues.apache.org/jira/browse/SOLR-10654 > Project: Solr > Issue Type: Improvement > Components: metrics >Reporter: Keith Laban >Priority: Major > Labels: pull-request-available > Fix For: 9.7 > > Attachments: prometheus_metrics.txt > > Time Spent: 9h > Remaining Estimate: 0h > > Expose metrics via a `wt=prometheus` response type. > Example scape_config in prometheus.yml: > {code:java} > scrape_configs: > - job_name: 'solr' > metrics_path: '/solr/admin/metrics' > params: > wt: ["prometheus"] > static_configs: > - targets: ['localhost:8983'] > {code} > [Rationale|https://issues.apache.org/jira/browse/SOLR-11795?focusedCommentId=17261423&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17261423] > for having this despite the "Prometheus Exporter". They have different > strengths and weaknesses. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-10654) Expose Metrics in Prometheus format DIRECTLY from Solr
[ https://issues.apache.org/jira/browse/SOLR-10654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17871836#comment-17871836 ] ASF subversion and git services commented on SOLR-10654: Commit c1b6b7195ba9ddb7e4e350e638a3c83f39857c86 in solr's branch refs/heads/branch_9_7 from Matthew Biscocho [ https://gitbox.apache.org/repos/asf?p=solr.git;h=c1b6b7195ba ] SOLR-10654: PrometheusResponseWriter Content Type was wrong (#2616) Co-authored-by: Matthew Biscocho Co-authored-by: David Smiley (cherry picked from commit e664779f5da06bdd4a3e16a86cb70b012a094285) > Expose Metrics in Prometheus format DIRECTLY from Solr > -- > > Key: SOLR-10654 > URL: https://issues.apache.org/jira/browse/SOLR-10654 > Project: Solr > Issue Type: Improvement > Components: metrics >Reporter: Keith Laban >Priority: Major > Labels: pull-request-available > Fix For: 9.7 > > Attachments: prometheus_metrics.txt > > Time Spent: 8h 40m > Remaining Estimate: 0h > > Expose metrics via a `wt=prometheus` response type. > Example scape_config in prometheus.yml: > {code:java} > scrape_configs: > - job_name: 'solr' > metrics_path: '/solr/admin/metrics' > params: > wt: ["prometheus"] > static_configs: > - targets: ['localhost:8983'] > {code} > [Rationale|https://issues.apache.org/jira/browse/SOLR-11795?focusedCommentId=17261423&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17261423] > for having this despite the "Prometheus Exporter". They have different > strengths and weaknesses. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-10654) Expose Metrics in Prometheus format DIRECTLY from Solr
[ https://issues.apache.org/jira/browse/SOLR-10654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17871834#comment-17871834 ] ASF subversion and git services commented on SOLR-10654: Commit a5903b15dcb2e96b251025421b48838303483858 in solr's branch refs/heads/branch_9x from Matthew Biscocho [ https://gitbox.apache.org/repos/asf?p=solr.git;h=a5903b15dcb ] SOLR-10654: PrometheusResponseWriter Content Type was wrong (#2616) Co-authored-by: Matthew Biscocho Co-authored-by: David Smiley (cherry picked from commit e664779f5da06bdd4a3e16a86cb70b012a094285) > Expose Metrics in Prometheus format DIRECTLY from Solr > -- > > Key: SOLR-10654 > URL: https://issues.apache.org/jira/browse/SOLR-10654 > Project: Solr > Issue Type: Improvement > Components: metrics >Reporter: Keith Laban >Priority: Major > Labels: pull-request-available > Fix For: 9.7 > > Attachments: prometheus_metrics.txt > > Time Spent: 8h 40m > Remaining Estimate: 0h > > Expose metrics via a `wt=prometheus` response type. > Example scape_config in prometheus.yml: > {code:java} > scrape_configs: > - job_name: 'solr' > metrics_path: '/solr/admin/metrics' > params: > wt: ["prometheus"] > static_configs: > - targets: ['localhost:8983'] > {code} > [Rationale|https://issues.apache.org/jira/browse/SOLR-11795?focusedCommentId=17261423&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17261423] > for having this despite the "Prometheus Exporter". They have different > strengths and weaknesses. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-10654) Expose Metrics in Prometheus format DIRECTLY from Solr
[ https://issues.apache.org/jira/browse/SOLR-10654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17871833#comment-17871833 ] ASF subversion and git services commented on SOLR-10654: Commit e664779f5da06bdd4a3e16a86cb70b012a094285 in solr's branch refs/heads/main from Matthew Biscocho [ https://gitbox.apache.org/repos/asf?p=solr.git;h=e664779f5da ] SOLR-10654: PrometheusResponseWriter Content Type was wrong (#2616) Co-authored-by: Matthew Biscocho Co-authored-by: David Smiley > Expose Metrics in Prometheus format DIRECTLY from Solr > -- > > Key: SOLR-10654 > URL: https://issues.apache.org/jira/browse/SOLR-10654 > Project: Solr > Issue Type: Improvement > Components: metrics >Reporter: Keith Laban >Priority: Major > Labels: pull-request-available > Fix For: 9.7 > > Attachments: prometheus_metrics.txt > > Time Spent: 8h 40m > Remaining Estimate: 0h > > Expose metrics via a `wt=prometheus` response type. > Example scape_config in prometheus.yml: > {code:java} > scrape_configs: > - job_name: 'solr' > metrics_path: '/solr/admin/metrics' > params: > wt: ["prometheus"] > static_configs: > - targets: ['localhost:8983'] > {code} > [Rationale|https://issues.apache.org/jira/browse/SOLR-11795?focusedCommentId=17261423&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17261423] > for having this despite the "Prometheus Exporter". They have different > strengths and weaknesses. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-10654) Expose Metrics in Prometheus format DIRECTLY from Solr
[ https://issues.apache.org/jira/browse/SOLR-10654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17871341#comment-17871341 ] Matthew Biscocho commented on SOLR-10654: - [~dsmiley] this is a good point. I'll make the appropriate change for the naming to avoid confusion between the two. > Expose Metrics in Prometheus format DIRECTLY from Solr > -- > > Key: SOLR-10654 > URL: https://issues.apache.org/jira/browse/SOLR-10654 > Project: Solr > Issue Type: Improvement > Components: metrics >Reporter: Keith Laban >Priority: Major > Labels: pull-request-available > Fix For: 9.7 > > Attachments: prometheus_metrics.txt > > Time Spent: 7h 50m > Remaining Estimate: 0h > > Expose metrics via a `wt=prometheus` response type. > Example scape_config in prometheus.yml: > {code:java} > scrape_configs: > - job_name: 'solr' > metrics_path: '/solr/admin/metrics' > params: > wt: ["prometheus"] > static_configs: > - targets: ['localhost:8983'] > {code} > [Rationale|https://issues.apache.org/jira/browse/SOLR-11795?focusedCommentId=17261423&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17261423] > for having this despite the "Prometheus Exporter". They have different > strengths and weaknesses. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-10654) Expose Metrics in Prometheus format DIRECTLY from Solr
[ https://issues.apache.org/jira/browse/SOLR-10654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17870766#comment-17870766 ] David Smiley commented on SOLR-10654: - [~mbiscocho], the internal class names for this new functionality has names that sound like the existing so-called "prometheus exporter" (e.g. "SolrPrometheusExporter" !). Without looking carefully for the location of where the source lives or the package, the class names will confuse us developers, I think. Might you suggest alternative names; perhaps ones without "Exporter" -- like maybe swap "Format" for it? (Not sure if we discussed this previously but it seems glaring to me now) > Expose Metrics in Prometheus format DIRECTLY from Solr > -- > > Key: SOLR-10654 > URL: https://issues.apache.org/jira/browse/SOLR-10654 > Project: Solr > Issue Type: Improvement > Components: metrics >Reporter: Keith Laban >Priority: Major > Fix For: 9.7 > > Attachments: prometheus_metrics.txt > > Time Spent: 7h 20m > Remaining Estimate: 0h > > Expose metrics via a `wt=prometheus` response type. > Example scape_config in prometheus.yml: > {code:java} > scrape_configs: > - job_name: 'solr' > metrics_path: '/solr/admin/metrics' > params: > wt: ["prometheus"] > static_configs: > - targets: ['localhost:8983'] > {code} > [Rationale|https://issues.apache.org/jira/browse/SOLR-11795?focusedCommentId=17261423&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17261423] > for having this despite the "Prometheus Exporter". They have different > strengths and weaknesses. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-10654) Expose Metrics in Prometheus format DIRECTLY from Solr
[ https://issues.apache.org/jira/browse/SOLR-10654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17866878#comment-17866878 ] ASF subversion and git services commented on SOLR-10654: Commit 728f8b57deb6c828725e187d8d354224a9d14ab7 in solr's branch refs/heads/branch_9x from Chris M. Hostetter [ https://gitbox.apache.org/repos/asf?p=solr.git;h=728f8b57deb ] SOLR-10654: PrometheusResponseWriter test tweaks - Fix MetricsHandlerTest - AwaitsFix TestPrometheusResponseWriter pending SOLR-17368 (cherry picked from commit 6967c7b695c39b045e1ac49eb346ab49ceab2a13) > Expose Metrics in Prometheus format DIRECTLY from Solr > -- > > Key: SOLR-10654 > URL: https://issues.apache.org/jira/browse/SOLR-10654 > Project: Solr > Issue Type: Improvement > Components: metrics >Reporter: Keith Laban >Priority: Major > Attachments: prometheus_metrics.txt > > Time Spent: 7h 20m > Remaining Estimate: 0h > > Expose metrics via a `wt=prometheus` response type. > Example scape_config in prometheus.yml: > {code:java} > scrape_configs: > - job_name: 'solr' > metrics_path: '/solr/admin/metrics' > params: > wt: ["prometheus"] > static_configs: > - targets: ['localhost:8983'] > {code} > [Rationale|https://issues.apache.org/jira/browse/SOLR-11795?focusedCommentId=17261423&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17261423] > for having this despite the "Prometheus Exporter". They have different > strengths and weaknesses. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-10654) Expose Metrics in Prometheus format DIRECTLY from Solr
[ https://issues.apache.org/jira/browse/SOLR-10654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17866877#comment-17866877 ] ASF subversion and git services commented on SOLR-10654: Commit a8c4a90e9fdb72fab6b1bfbc5bb76b8c72f020d0 in solr's branch refs/heads/branch_9x from Matthew Biscocho [ https://gitbox.apache.org/repos/asf?p=solr.git;h=a8c4a90e9fd ] SOLR-10654: Metrics: New wt=prometheus option (#2405) An alternative to the "Prometheus Exporter" in which each Solr node can return the Prometheus format natively from the MetricsHandler using `wt=prometheus` param. It's much faster and architecturally simpler, albeit less flexible. Co-authored-by: mbiscocho Co-authored-by: Christine Poerschke Co-authored-by: David Smiley (cherry picked from commit fd7d44771e0c35edb5373d22a499929b3e828469) > Expose Metrics in Prometheus format DIRECTLY from Solr > -- > > Key: SOLR-10654 > URL: https://issues.apache.org/jira/browse/SOLR-10654 > Project: Solr > Issue Type: Improvement > Components: metrics >Reporter: Keith Laban >Priority: Major > Attachments: prometheus_metrics.txt > > Time Spent: 7h 20m > Remaining Estimate: 0h > > Expose metrics via a `wt=prometheus` response type. > Example scape_config in prometheus.yml: > {code:java} > scrape_configs: > - job_name: 'solr' > metrics_path: '/solr/admin/metrics' > params: > wt: ["prometheus"] > static_configs: > - targets: ['localhost:8983'] > {code} > [Rationale|https://issues.apache.org/jira/browse/SOLR-11795?focusedCommentId=17261423&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17261423] > for having this despite the "Prometheus Exporter". They have different > strengths and weaknesses. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-10654) Expose Metrics in Prometheus format DIRECTLY from Solr
[ https://issues.apache.org/jira/browse/SOLR-10654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17866143#comment-17866143 ] Chris M. Hostetter commented on SOLR-10654: --- {quote}Is this really specifically about Gauges (vs say Timers)? Any way, my understanding is that we can call SolrMetricManager.removeRegistry on the various registries. Perhaps tests generally or maybe just this test isn't doing that. As we have test infrastructure that detects leaks of things, maybe we need to detect a metric registry leak. {quote} I am not an expert on the dropwizard stuff, and it's been a while since i looked at it, but yes: Gauges are particularly tricky to deal with from a lifecycle standpoint, which is why Solr has it's own GaugeWrapper. But that doesn't help us when other libraries register gauges. SOLR-16918 (which i linked to previously) is an example of problematic gauge lifecycle situations that (i believe) could leak between tests. Any gauges created directly by jetty are also likeley to leak between tests – which is i believe exactly what is happening here. > Expose Metrics in Prometheus format DIRECTLY from Solr > -- > > Key: SOLR-10654 > URL: https://issues.apache.org/jira/browse/SOLR-10654 > Project: Solr > Issue Type: Improvement > Components: metrics >Reporter: Keith Laban >Priority: Major > Attachments: prometheus_metrics.txt > > Time Spent: 7h 20m > Remaining Estimate: 0h > > Expose metrics via a `wt=prometheus` response type. > Example scape_config in prometheus.yml: > {code:java} > scrape_configs: > - job_name: 'solr' > metrics_path: '/solr/admin/metrics' > params: > wt: ["prometheus"] > static_configs: > - targets: ['localhost:8983'] > {code} > [Rationale|https://issues.apache.org/jira/browse/SOLR-11795?focusedCommentId=17261423&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17261423] > for having this despite the "Prometheus Exporter". They have different > strengths and weaknesses. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-10654) Expose Metrics in Prometheus format DIRECTLY from Solr
[ https://issues.apache.org/jira/browse/SOLR-10654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17866008#comment-17866008 ] ASF subversion and git services commented on SOLR-10654: Commit 6967c7b695c39b045e1ac49eb346ab49ceab2a13 in solr's branch refs/heads/pr/2550 from Chris M. Hostetter [ https://gitbox.apache.org/repos/asf?p=solr.git;h=6967c7b695c ] SOLR-10654: PrometheusResponseWriter test tweaks - Fix MetricsHandlerTest - AwaitsFix TestPrometheusResponseWriter pending SOLR-17368 > Expose Metrics in Prometheus format DIRECTLY from Solr > -- > > Key: SOLR-10654 > URL: https://issues.apache.org/jira/browse/SOLR-10654 > Project: Solr > Issue Type: Improvement > Components: metrics >Reporter: Keith Laban >Priority: Major > Attachments: prometheus_metrics.txt > > Time Spent: 7h 20m > Remaining Estimate: 0h > > Expose metrics via a `wt=prometheus` response type. > Example scape_config in prometheus.yml: > {code:java} > scrape_configs: > - job_name: 'solr' > metrics_path: '/solr/admin/metrics' > params: > wt: ["prometheus"] > static_configs: > - targets: ['localhost:8983'] > {code} > [Rationale|https://issues.apache.org/jira/browse/SOLR-11795?focusedCommentId=17261423&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17261423] > for having this despite the "Prometheus Exporter". They have different > strengths and weaknesses. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-10654) Expose Metrics in Prometheus format DIRECTLY from Solr
[ https://issues.apache.org/jira/browse/SOLR-10654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17866001#comment-17866001 ] ASF subversion and git services commented on SOLR-10654: Commit fd7d44771e0c35edb5373d22a499929b3e828469 in solr's branch refs/heads/pr/2550 from Matthew Biscocho [ https://gitbox.apache.org/repos/asf?p=solr.git;h=fd7d44771e0 ] SOLR-10654: Metrics: New wt=prometheus option (#2405) An alternative to the "Prometheus Exporter" in which each Solr node can return the Prometheus format natively from the MetricsHandler using `wt=prometheus` param. It's much faster and architecturally simpler, albeit less flexible. Co-authored-by: mbiscocho Co-authored-by: Christine Poerschke Co-authored-by: David Smiley > Expose Metrics in Prometheus format DIRECTLY from Solr > -- > > Key: SOLR-10654 > URL: https://issues.apache.org/jira/browse/SOLR-10654 > Project: Solr > Issue Type: Improvement > Components: metrics >Reporter: Keith Laban >Priority: Major > Attachments: prometheus_metrics.txt > > Time Spent: 7h 20m > Remaining Estimate: 0h > > Expose metrics via a `wt=prometheus` response type. > Example scape_config in prometheus.yml: > {code:java} > scrape_configs: > - job_name: 'solr' > metrics_path: '/solr/admin/metrics' > params: > wt: ["prometheus"] > static_configs: > - targets: ['localhost:8983'] > {code} > [Rationale|https://issues.apache.org/jira/browse/SOLR-11795?focusedCommentId=17261423&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17261423] > for having this despite the "Prometheus Exporter". They have different > strengths and weaknesses. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-10654) Expose Metrics in Prometheus format DIRECTLY from Solr
[ https://issues.apache.org/jira/browse/SOLR-10654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17865642#comment-17865642 ] David Smiley commented on SOLR-10654: - Thanks for jumping in Hoss. I agree it'd be a good idea to test the values match the format (e.g. is an integer vs string vs float). If the content needs to be scrubbed, we should *replace* values with a nominal value (i.e. 42 becomes 0) instead of with nothing. We agree that the regex in there is too complex! I complained in the PR but should have insisted on further improvements. In the JVM category, we should also blank out certain label values that are JVM specific like {{{}space="CodeCache"{}}}. {quote}then WTF is the point of having the test {quote} New/removed metrics *might* require code updates. This is one of the bigger concerns I have with the underlying approach; it's a rather hard-coded translation because we decided to not switch out DropWizard with a Prometheus native metrics backend. Whoever updates the TXT file should do it as a thinking person who considers that the change makes sense – has a name, label, label-value, and metric format that makes sense – matches conventions of others as well too, for example. A metric could inexplicably disappear; should be investigated if not intended. {quote}Unless/until the lifecycle of Gauges is drastically overhauled in a future version of dropwizard {quote} Is this really specifically about Gauges (vs say Timers)? Any way, my understanding is that we can call SolrMetricManager.removeRegistry on the various registries. Perhaps tests generally or maybe just this test isn't doing that. As we have test infrastructure that detects leaks of things, maybe we need to detect a metric registry leak. > Expose Metrics in Prometheus format DIRECTLY from Solr > -- > > Key: SOLR-10654 > URL: https://issues.apache.org/jira/browse/SOLR-10654 > Project: Solr > Issue Type: Improvement > Components: metrics >Reporter: Keith Laban >Priority: Major > Attachments: prometheus_metrics.txt > > Time Spent: 7h 20m > Remaining Estimate: 0h > > Expose metrics via a `wt=prometheus` response type. > Example scape_config in prometheus.yml: > {code:java} > scrape_configs: > - job_name: 'solr' > metrics_path: '/solr/admin/metrics' > params: > wt: ["prometheus"] > static_configs: > - targets: ['localhost:8983'] > {code} > [Rationale|https://issues.apache.org/jira/browse/SOLR-11795?focusedCommentId=17261423&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17261423] > for having this despite the "Prometheus Exporter". They have different > strengths and weaknesses. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-10654) Expose Metrics in Prometheus format DIRECTLY from Solr
[ https://issues.apache.org/jira/browse/SOLR-10654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17865589#comment-17865589 ] Chris M. Hostetter commented on SOLR-10654: --- Since we've established that both problems seem to be test specific, "rolling back" doesn't seem necessary. I went ahead and did the quick fix to MetricsHandlerTest and opened SOLR-17368 to track improving TestPrometheusResponseWriter so that we can re-resolve this jira. > Expose Metrics in Prometheus format DIRECTLY from Solr > -- > > Key: SOLR-10654 > URL: https://issues.apache.org/jira/browse/SOLR-10654 > Project: Solr > Issue Type: Improvement > Components: metrics >Reporter: Keith Laban >Priority: Major > Attachments: prometheus_metrics.txt > > Time Spent: 7h 20m > Remaining Estimate: 0h > > Expose metrics via a `wt=prometheus` response type. > Example scape_config in prometheus.yml: > {code:java} > scrape_configs: > - job_name: 'solr' > metrics_path: '/solr/admin/metrics' > params: > wt: ["prometheus"] > static_configs: > - targets: ['localhost:8983'] > {code} > [Rationale|https://issues.apache.org/jira/browse/SOLR-11795?focusedCommentId=17261423&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17261423] > for having this despite the "Prometheus Exporter". They have different > strengths and weaknesses. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-10654) Expose Metrics in Prometheus format DIRECTLY from Solr
[ https://issues.apache.org/jira/browse/SOLR-10654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17865587#comment-17865587 ] ASF subversion and git services commented on SOLR-10654: Commit 6967c7b695c39b045e1ac49eb346ab49ceab2a13 in solr's branch refs/heads/main from Chris M. Hostetter [ https://gitbox.apache.org/repos/asf?p=solr.git;h=6967c7b695c ] SOLR-10654: PrometheusResponseWriter test tweaks - Fix MetricsHandlerTest - AwaitsFix TestPrometheusResponseWriter pending SOLR-17368 > Expose Metrics in Prometheus format DIRECTLY from Solr > -- > > Key: SOLR-10654 > URL: https://issues.apache.org/jira/browse/SOLR-10654 > Project: Solr > Issue Type: Improvement > Components: metrics >Reporter: Keith Laban >Priority: Major > Attachments: prometheus_metrics.txt > > Time Spent: 7h 20m > Remaining Estimate: 0h > > Expose metrics via a `wt=prometheus` response type. > Example scape_config in prometheus.yml: > {code:java} > scrape_configs: > - job_name: 'solr' > metrics_path: '/solr/admin/metrics' > params: > wt: ["prometheus"] > static_configs: > - targets: ['localhost:8983'] > {code} > [Rationale|https://issues.apache.org/jira/browse/SOLR-11795?focusedCommentId=17261423&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17261423] > for having this despite the "Prometheus Exporter". They have different > strengths and weaknesses. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-10654) Expose Metrics in Prometheus format DIRECTLY from Solr
[ https://issues.apache.org/jira/browse/SOLR-10654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17865546#comment-17865546 ] Chris M. Hostetter commented on SOLR-10654: --- Ok, but your stated goal... {quote}The test using solr-prometheus-output.txt, which I asked for, is intended to ensure we don't accidentally remove an exported metric due to some internal renaming choice or a change in the type of metric. That's it. It also serves as an integration test. {quote} Is 100% compatible with my suggestion... {quote}The test needs to parse the "actual" response and make some basic assertions about the _structure_ it finds, and if there are certain metrics that should *always* be present, assert they exist with values that are _valid_ (ie: can be parsed as an int, or can be parsed as a float) w/o asserting exactly what those values are. {quote} ...if you want to keep that list of "mandatory" expected metrics that should *always* be present in a txt file because you think that's easier to maintain then a static {{Arrays.asList(...)}} so be it. But the way it works now: asserting that the entire actual response, with a confusing regex applied, will exactly equals the contents of text file is seriously problematic... # It's Brittle AF, and is going to be tedius to update anytime we add metrics – or anytime a library we use (like jetty or dropwizard's executor service) adds metrics. And if the dev process is "update this exepcted txt fle to match the actual response anytime the test fails" ... then WTF is the point of having the test? # As seen in the 100% jenkins failure rate: Unless/until the lifecycle of Gauges is drastically overhauled in a future version of dropwizard, you can never prededict the full list of (jetty, maybe other) metrics that will be in the actual response depending on what other tests ran before this one in the same JVM. #2 is really my biggest concern right now – this isn't a question of what to do *_*IF*_* the test fails in the future – this test has literally *_never passed!_* > Expose Metrics in Prometheus format DIRECTLY from Solr > -- > > Key: SOLR-10654 > URL: https://issues.apache.org/jira/browse/SOLR-10654 > Project: Solr > Issue Type: Improvement > Components: metrics >Reporter: Keith Laban >Priority: Major > Attachments: prometheus_metrics.txt > > Time Spent: 7h 20m > Remaining Estimate: 0h > > Expose metrics via a `wt=prometheus` response type. > Example scape_config in prometheus.yml: > {code:java} > scrape_configs: > - job_name: 'solr' > metrics_path: '/solr/admin/metrics' > params: > wt: ["prometheus"] > static_configs: > - targets: ['localhost:8983'] > {code} > [Rationale|https://issues.apache.org/jira/browse/SOLR-11795?focusedCommentId=17261423&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17261423] > for having this despite the "Prometheus Exporter". They have different > strengths and weaknesses. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-10654) Expose Metrics in Prometheus format DIRECTLY from Solr
[ https://issues.apache.org/jira/browse/SOLR-10654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17865296#comment-17865296 ] David Smiley commented on SOLR-10654: - The test using solr-prometheus-output.txt, which I asked for, is intended to ensure we don't accidentally remove an exported metric due to some internal renaming choice or a change in the type of metric. That's it. It also serves as an integration test. The test may fail when metrics are added, yes, but we need only re-save the file. Ideally this test would have docs to state such things, and maybe cater to the situation of needing to re-save the file (such as by dumping the output to a file that can then be saved-over. > Expose Metrics in Prometheus format DIRECTLY from Solr > -- > > Key: SOLR-10654 > URL: https://issues.apache.org/jira/browse/SOLR-10654 > Project: Solr > Issue Type: Improvement > Components: metrics >Reporter: Keith Laban >Priority: Major > Attachments: prometheus_metrics.txt > > Time Spent: 7h 20m > Remaining Estimate: 0h > > Expose metrics via a `wt=prometheus` response type. > Example scape_config in prometheus.yml: > {code:java} > scrape_configs: > - job_name: 'solr' > metrics_path: '/solr/admin/metrics' > params: > wt: ["prometheus"] > static_configs: > - targets: ['localhost:8983'] > {code} > [Rationale|https://issues.apache.org/jira/browse/SOLR-11795?focusedCommentId=17261423&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17261423] > for having this despite the "Prometheus Exporter". They have different > strengths and weaknesses. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-10654) Expose Metrics in Prometheus format DIRECTLY from Solr
[ https://issues.apache.org/jira/browse/SOLR-10654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17865249#comment-17865249 ] Chris M. Hostetter commented on SOLR-10654: --- Unrelated to the test issues – I'm no prometheus expert, but this can't possibly be correct... {noformat} $ curl -sS -I 'http://localhost:8983/solr/admin/metrics?wt=prometheus' | grep ^Content-Type Content-Type: application/json;charset=utf-8 {noformat} I suspect PrometheusResponseWriter needs to override the the {{getContentType}} method? > Expose Metrics in Prometheus format DIRECTLY from Solr > -- > > Key: SOLR-10654 > URL: https://issues.apache.org/jira/browse/SOLR-10654 > Project: Solr > Issue Type: Improvement > Components: metrics >Reporter: Keith Laban >Priority: Major > Attachments: prometheus_metrics.txt > > Time Spent: 7h 20m > Remaining Estimate: 0h > > Expose metrics via a `wt=prometheus` response type. > Example scape_config in prometheus.yml: > {code:java} > scrape_configs: > - job_name: 'solr' > metrics_path: '/solr/admin/metrics' > params: > wt: ["prometheus"] > static_configs: > - targets: ['localhost:8983'] > {code} > [Rationale|https://issues.apache.org/jira/browse/SOLR-11795?focusedCommentId=17261423&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17261423] > for having this despite the "Prometheus Exporter". They have different > strengths and weaknesses. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-10654) Expose Metrics in Prometheus format DIRECTLY from Solr
[ https://issues.apache.org/jira/browse/SOLR-10654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17865247#comment-17865247 ] Chris M. Hostetter commented on SOLR-10654: --- {quote}In the PR, I have a regex to try and remove some of the JVM implementations but seems as it's not enough. {quote} For {{{}TestPrometheusResponseWriter{}}}... You need to just plain kill the "equals" assertion against {{solr-prometheus-output.txt}} ... no matter how many regex tricks you try, that's way to brittle to survive future additions/tweaks to the metrics solr supports. The test needs to parse the "actual" response and make some basic assertions about the _structure_ it finds, and if there are certain metrics that should *always* be present, assert they exist with values that are _valid_ (ie: can be parsed as an int, or can be parsed as a float) w/o asserting exactly what those values are. But under no circumstances should you assert that you know the full list of every metrics that _might_ someday exist in solr/jetty. (this would also be a good place register dummy metrics with fake names that you can easily assert have _specific_ expect values) For {{MetricsHandlerTest.testPrometheusMetricsJvm}}... {code} s/assertEquals(0, /assertNotNull(/ {code} > Expose Metrics in Prometheus format DIRECTLY from Solr > -- > > Key: SOLR-10654 > URL: https://issues.apache.org/jira/browse/SOLR-10654 > Project: Solr > Issue Type: Improvement > Components: metrics >Reporter: Keith Laban >Priority: Major > Attachments: prometheus_metrics.txt > > Time Spent: 7h 20m > Remaining Estimate: 0h > > Expose metrics via a `wt=prometheus` response type. > Example scape_config in prometheus.yml: > {code:java} > scrape_configs: > - job_name: 'solr' > metrics_path: '/solr/admin/metrics' > params: > wt: ["prometheus"] > static_configs: > - targets: ['localhost:8983'] > {code} > [Rationale|https://issues.apache.org/jira/browse/SOLR-11795?focusedCommentId=17261423&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17261423] > for having this despite the "Prometheus Exporter". They have different > strengths and weaknesses. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-10654) Expose Metrics in Prometheus format DIRECTLY from Solr
[ https://issues.apache.org/jira/browse/SOLR-10654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17865246#comment-17865246 ] Chris M. Hostetter commented on SOLR-10654: --- This is the assert that (sometimes) fails in {{{}MetricsHandlerTest.testPrometheusMetricsJvm{}}}... {code:java} actualSnapshot = getMetricSnapshot(actualSnapshots, "solr_metrics_jvm_memory_pools_bytes"); actualGaugeDataPoint = getGaugeDatapointSnapshot( actualSnapshot, Labels.of("item", "committed", "space", "CodeHeap-'non-nmethods'")); assertEquals(0, actualGaugeDataPoint.getValue(), 0); {code} ...I couldn't fathom how that assert could ever be successful, on any JVM, and went down a rabbit hole of trying to figure out what it is about committed CodeHeap sizes that might prevent them from being used in test JVMs ... until i realized that you are (attempting) to stub out those gauges in your test setup with dummies that always return 0! *EXCEPT:* * As mentioned, the lifecycle of gauges is prblematic – if any other test (or code path) in this JVM creates those same gauges (and doesn't explicitly unregister them) your stubs ill be ignored. * this is the *only* one of those dummy gauges that you create where you also {{{}assertEquals(0, ...){}}}... ** For all the other dummy gauges this test method looks at, the only assertion made is {{assertNotNull(...)}} ** Almost as if [you already ran into this problem with the gauges, but overlooked fixing this assert when you fixed all the others|https://github.com/mlbiscoc/solr/commit/835c946ecac04900f840ccd25cf278a6a6a2aef4] and added this comment... {noformat} // Some JVM metrics are non-deterministic due to testing environment such as // availableProcessors. We confirm snapshot exists and is nonNull instead. {noformat} > Expose Metrics in Prometheus format DIRECTLY from Solr > -- > > Key: SOLR-10654 > URL: https://issues.apache.org/jira/browse/SOLR-10654 > Project: Solr > Issue Type: Improvement > Components: metrics >Reporter: Keith Laban >Priority: Major > Attachments: prometheus_metrics.txt > > Time Spent: 7h 20m > Remaining Estimate: 0h > > Expose metrics via a `wt=prometheus` response type. > Example scape_config in prometheus.yml: > {code:java} > scrape_configs: > - job_name: 'solr' > metrics_path: '/solr/admin/metrics' > params: > wt: ["prometheus"] > static_configs: > - targets: ['localhost:8983'] > {code} > [Rationale|https://issues.apache.org/jira/browse/SOLR-11795?focusedCommentId=17261423&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17261423] > for having this despite the "Prometheus Exporter". They have different > strengths and weaknesses. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-10654) Expose Metrics in Prometheus format DIRECTLY from Solr
[ https://issues.apache.org/jira/browse/SOLR-10654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17865239#comment-17865239 ] Matthew Biscocho commented on SOLR-10654: - Thanks Chris, I agree. My intention wasn't to just fix the test failures but to try and get a failure on my end to try and remove as much environmental variables as possible. In the PR, I have a regex to try and remove some of the JVM implementations but seems as it's not enough. The gauge issue is interesting but plan on taking a deeper look to fix these up. > Expose Metrics in Prometheus format DIRECTLY from Solr > -- > > Key: SOLR-10654 > URL: https://issues.apache.org/jira/browse/SOLR-10654 > Project: Solr > Issue Type: Improvement > Components: metrics >Reporter: Keith Laban >Priority: Major > Attachments: prometheus_metrics.txt > > Time Spent: 7h 20m > Remaining Estimate: 0h > > Expose metrics via a `wt=prometheus` response type. > Example scape_config in prometheus.yml: > {code:java} > scrape_configs: > - job_name: 'solr' > metrics_path: '/solr/admin/metrics' > params: > wt: ["prometheus"] > static_configs: > - targets: ['localhost:8983'] > {code} > [Rationale|https://issues.apache.org/jira/browse/SOLR-11795?focusedCommentId=17261423&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17261423] > for having this despite the "Prometheus Exporter". They have different > strengths and weaknesses. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-10654) Expose Metrics in Prometheus format DIRECTLY from Solr
[ https://issues.apache.org/jira/browse/SOLR-10654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17865236#comment-17865236 ] Chris M. Hostetter commented on SOLR-10654: --- {quote}These tests must be failing depending on environmen or type of JVM being used as I am unable to reproduce this on my end. Do you have more information on how the Jenkins builds are being run? {quote} I don't, but even if i/you did "fixing" the test to pass on jenkins shouldn't be the goal – you'd still be leaving a time bomb for some future dev whose JVM doesn't exactly match yours The bottom line is that every tests we have should be able pass on *ANY* valid JVM – tests that are known to be problematic on _specific_ JVM implementations (like IBM's OpenJ9) should have specific {{assume()}} calls to prevent them from running. If test assertions are based on what metrics are available, they should _only_ look at metrics that are guaranteed to be available on every JVM, and not make assumptions about what "extra" metrics may or may not exist on _some_ JVMs (and may or may not have values that differ from what you see on *your* JVM) In the case of {{TestPrometheusResponseWriter}} – it's pretty easy to reproduce the problem on _any_ JVM by running multiple "jetty" based test in the same Test JVM – the jetty dispatches/request metrics that aren't accounted for in your hardcoded "expected" value seem to get created on the fly once there are some requests (and they seem to stick around beyond the lifecycle of the tests – (i suspect they are gauges which are known to have problematic lifecycles, but i haven't checked) ... {noformat} $ ./gradlew test -Ptests.jvms=1 -Ptests.verbose=true --tests TestPrometheusResponseWriter --tests \*Jetty\* ... org.apache.solr.response.TestPrometheusResponseWriter > testPrometheusOutput FAILED org.junit.ComparisonFailure: expected:<... TYPE solr_metrics_j[vm_buffers gauge solr_metrics_jvm_buffers{item="Count",pool="direct"} solr_metrics_jvm_buffers{item="Count",pool="mapped"} solr_metrics_jvm_buffers{item="Count",pool="mapped - 'non-volatile memory'"} # TYPE solr_metrics_jvm_buffers_bytes gauge solr_metrics_jvm_buffers_bytes{item="MemoryUsed",pool="direct"} solr_metrics_jvm_buffers_bytes{item="MemoryUsed",pool="mapped"} solr_metrics_jvm_buffers_bytes{item="MemoryUsed",pool="mapped - 'non-volatile memory'"} solr_metrics_jvm_buffers_bytes{item="TotalCapacity",pool="direct"} solr_metrics_jvm_buffers_bytes{item="TotalCapacity",pool="mapped"} solr_metrics_jvm_buffers_bytes{item="TotalCapacity",pool="mapped - 'non-volatile memory'"} # TYPE solr_metrics_jvm_gc gauge solr_metrics_jvm_gc{} solr_metrics_jvm_gc{} # TYPE solr_metrics_jvm_gc_seconds gauge solr_metrics_jvm_gc_seconds{} solr_metrics_jvm_gc_seconds{} # TYPE solr_metrics_jvm_heap gauge solr_metrics_jvm_heap{item="committed",memory="heap"} solr_metrics_jvm_heap{item="committed",memory="non-heap"} solr_metrics_jvm_heap{item="committed",memory="total"} solr_metrics_jvm_heap{item="init",memory="heap"} solr_metrics_jvm_heap{item="init",memory="non-heap"} solr_metrics_jvm_heap{item="init",memory="total"} solr_metrics_jvm_heap{item="max",memory="heap"} solr_metrics_jvm_heap{item="max",memory="non-heap"} solr_metrics_jvm_heap{item="max",memory="total"} solr_metrics_jvm_heap{item="usage",memory="heap"} solr_metrics_jvm_heap{item="usage",memory="non-heap"} solr_metrics_jvm_heap{item="used",memory="heap"} solr_metrics_jvm_heap{item="used",memory="non-heap"} solr_metrics_jvm_heap{item="used",memory="total"} # TYPE solr_metrics_jvm_memory_pools_bytes gauge solr_metrics_jvm_memory_pools_bytes{item="committed",space="CodeCache]"} solr_metrics_jvm_...> but was:<... TYPE solr_metrics_j[etty_dispatches_total counter solr_metrics_jetty_dispatches_total 0.0 # TYPE solr_metrics_jetty_requests_total counter solr_metrics_jetty_requests_total{method="active"} # TYPE solr_metrics_jetty_response_total counter solr_metrics_jetty_response_total{status="2xx"} # TYPE solr_metrics_jvm_buffers gauge solr_metrics_jvm_buffers{item="Count",pool="direct"} solr_metrics_jvm_buffers{item="Count",pool="mapped"} solr_metrics_jvm_buffers{item="Count",pool="mapped - 'non-volatile memory'"} # TYPE solr_metrics_jvm_buffers_bytes gauge solr_metrics_jvm_buffers_bytes{item="MemoryUsed",pool="direct"} solr_metrics_jvm_buffers_bytes{item="MemoryUsed",pool="mapped"} solr_metrics_jvm_buffers_bytes{item="MemoryUsed",pool="mapped - 'non-volatile memory'"} solr_metrics_jvm_buffers_bytes{item="TotalCapacity",pool="direct"} solr_metrics_jvm_buffers_bytes{item="TotalCapacity",pool="mapped"} solr_metrics_jvm_buffers_bytes{item="TotalCapacity",pool="mapped - 'non-volatile memory
[jira] [Commented] (SOLR-10654) Expose Metrics in Prometheus format DIRECTLY from Solr
[ https://issues.apache.org/jira/browse/SOLR-10654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17864665#comment-17864665 ] Matthew Biscocho commented on SOLR-10654: - These tests must be failing depending on environmen or type of JVM being used as I am unable to reproduce this on my end. Do you have more information on how the Jenkins builds are being run? > Expose Metrics in Prometheus format DIRECTLY from Solr > -- > > Key: SOLR-10654 > URL: https://issues.apache.org/jira/browse/SOLR-10654 > Project: Solr > Issue Type: Improvement > Components: metrics >Reporter: Keith Laban >Priority: Major > Attachments: prometheus_metrics.txt > > Time Spent: 7h 20m > Remaining Estimate: 0h > > Expose metrics via a `wt=prometheus` response type. > Example scape_config in prometheus.yml: > {code:java} > scrape_configs: > - job_name: 'solr' > metrics_path: '/solr/admin/metrics' > params: > wt: ["prometheus"] > static_configs: > - targets: ['localhost:8983'] > {code} > [Rationale|https://issues.apache.org/jira/browse/SOLR-11795?focusedCommentId=17261423&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17261423] > for having this despite the "Prometheus Exporter". They have different > strengths and weaknesses. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-10654) Expose Metrics in Prometheus format DIRECTLY from Solr
[ https://issues.apache.org/jira/browse/SOLR-10654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17864416#comment-17864416 ] Chris M. Hostetter commented on SOLR-10654: --- {{TestPrometheusResponseWriter.testPrometheusOutput}} has failed in 100% of the jenkins builds that have run it since being added yesterday ... the problem appears to be some jardcoded assumptions about metric ordering? {noformat} org.apache.solr.response.TestPrometheusResponseWriter > testPrometheusOutput FAILED org.junit.ComparisonFailure: expected:<... TYPE solr_metrics_j[]vm_buffers gauge sol...> but was:<... TYPE solr_metrics_j[etty_dispatches_total counter solr_metrics_jetty_dispatches_total 0.0 # TYPE solr_metrics_jetty_requests_total counter solr_metrics_jetty_requests_total{method="active"} # TYPE solr_metrics_jetty_response_total counter solr_metrics_jetty_response_total{status="2xx"} # TYPE solr_metrics_j]vm_buffers gauge sol...> at __randomizedtesting.SeedInfo.seed([76D2036CF55247FD:F7C26E458CF7A80B]:0) at org.junit.Assert.assertEquals(Assert.java:117) at org.junit.Assert.assertEquals(Assert.java:146) at org.apache.solr.response.TestPrometheusResponseWriter.testPrometheusOutput(TestPrometheusResponseWriter.java:86) {noformat} Likewise {{MetricsHandlerTest.testPrometheusMetricsJvm}} has failed 50% of the jenkins builds that jave run it since being added yesterday ... notably always on Uwe's jenkins machine, suggesting some hardcoded assumptions about metrics that may not be true in long running JVMs... {noformat} org.apache.solr.handler.admin.MetricsHandlerTest > testPrometheusMetricsJvm FAILED java.lang.AssertionError: expected:<0.0> but was:<3604480.0> at __randomizedtesting.SeedInfo.seed([5F3548D00F329FF4:577F9BC55C62D284]:0) at org.junit.Assert.fail(Assert.java:89) at org.junit.Assert.failNotEquals(Assert.java:835) at org.junit.Assert.assertEquals(Assert.java:555) at org.junit.Assert.assertEquals(Assert.java:685) at org.apache.solr.handler.admin.MetricsHandlerTest.testPrometheusMetricsJvm(MetricsHandlerTest.java:925) {noformat} > Expose Metrics in Prometheus format DIRECTLY from Solr > -- > > Key: SOLR-10654 > URL: https://issues.apache.org/jira/browse/SOLR-10654 > Project: Solr > Issue Type: Improvement > Components: metrics >Reporter: Keith Laban >Priority: Major > Attachments: prometheus_metrics.txt > > Time Spent: 7h 20m > Remaining Estimate: 0h > > Expose metrics via a `wt=prometheus` response type. > Example scape_config in prometheus.yml: > {code:java} > scrape_configs: > - job_name: 'solr' > metrics_path: '/solr/admin/metrics' > params: > wt: ["prometheus"] > static_configs: > - targets: ['localhost:8983'] > {code} > [Rationale|https://issues.apache.org/jira/browse/SOLR-11795?focusedCommentId=17261423&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17261423] > for having this despite the "Prometheus Exporter". They have different > strengths and weaknesses. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-10654) Expose Metrics in Prometheus format DIRECTLY from Solr
[ https://issues.apache.org/jira/browse/SOLR-10654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17863994#comment-17863994 ] ASF subversion and git services commented on SOLR-10654: Commit fd7d44771e0c35edb5373d22a499929b3e828469 in solr's branch refs/heads/main from Matthew Biscocho [ https://gitbox.apache.org/repos/asf?p=solr.git;h=fd7d44771e0 ] SOLR-10654: Metrics: New wt=prometheus option (#2405) An alternative to the "Prometheus Exporter" in which each Solr node can return the Prometheus format natively from the MetricsHandler using `wt=prometheus` param. It's much faster and architecturally simpler, albeit less flexible. Co-authored-by: mbiscocho Co-authored-by: Christine Poerschke Co-authored-by: David Smiley > Expose Metrics in Prometheus format DIRECTLY from Solr > -- > > Key: SOLR-10654 > URL: https://issues.apache.org/jira/browse/SOLR-10654 > Project: Solr > Issue Type: Improvement > Components: metrics >Reporter: Keith Laban >Priority: Major > Attachments: prometheus_metrics.txt > > Time Spent: 7h 20m > Remaining Estimate: 0h > > Expose metrics via a `wt=prometheus` response type. > Example scape_config in prometheus.yml: > {code:java} > scrape_configs: > - job_name: 'solr' > metrics_path: '/solr/admin/metrics' > params: > wt: ["prometheus"] > static_configs: > - targets: ['localhost:8983'] > {code} > [Rationale|https://issues.apache.org/jira/browse/SOLR-11795?focusedCommentId=17261423&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17261423] > for having this despite the "Prometheus Exporter". They have different > strengths and weaknesses. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-10654) Expose Metrics in Prometheus format DIRECTLY from Solr
[ https://issues.apache.org/jira/browse/SOLR-10654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17848396#comment-17848396 ] Matthew Biscocho commented on SOLR-10654: - The overhead post processing due to JQ is not the main problem but certainly is one. I would say, running the Prometheus exporter can be costly, especially at scale and running multiple instances. It offers the flexibility for configurability but I don't think that solves everyones use-case as it is not free to just run the Prometheus Exporter. I think the overhead of aggregating metrics should happen on the Grafana or Prometheus level while the exposed metrics themselves should just be raw values. With this PR, prometheus can just scrape and then the aggregation can be done on Grafana directly and skips the extra http call hops from the prometheus exporter and JQ processing. I took a bit of time to measure some performance between my PR and the prometheus exporter. I created a cloud with 2 nodes and 50 collections to get a bunch of metrics. For the cloud, I curl'd each node individually and captured the response time of each node. Not sure if prometheus scrapes sequentially or in parallel but looks like both just take around ~0.6s locally. I modified the Prometheus exporter config to only scrape the same metrics my PR currently exports (Core registry) and added a few lines of code to capture the timing it takes for scraping and JQ processing. Looking at the timing it was taking around 4-5 seconds per collection interval which is significantly longer. `My PR:` `curl -o /dev/null -s -w 'Total: %\{time_total}s\n' 'localhost:8983/solr/admin/metrics?wt=prometheus'` `Total: 0.614125s` `curl -o /dev/null -s -w 'Total: %\{time_total}s\n' 'localhost:7574/solr/admin/metrics?wt=prometheus'` `Total: 0.597078s` `Prometheus Exporter:` INFO - 2024-05-21 18:10:28.930; org.apache.solr.prometheus.collector.SchedulerMetricsCollector; Completed metrics collection INFO - 2024-05-21 18:11:28.923; org.apache.solr.prometheus.collector.SchedulerMetricsCollector; Beginning metrics collection PT4.355627S I want to say this is due to the Http calls and JQ processing the Prometheus Exporter needs to do while my PR is doing a straight internal conversion. Although it is doing the conversion per call, it doesn't seem to be as costly as the prometheus exporter is. > Expose Metrics in Prometheus format DIRECTLY from Solr > -- > > Key: SOLR-10654 > URL: https://issues.apache.org/jira/browse/SOLR-10654 > Project: Solr > Issue Type: Improvement > Components: metrics >Reporter: Keith Laban >Priority: Major > Attachments: prometheus_metrics.txt > > Time Spent: 3h > Remaining Estimate: 0h > > Expose metrics via a `wt=prometheus` response type. > Example scape_config in prometheus.yml: > {code} > scrape_configs: > - job_name: 'solr' > metrics_path: '/solr/admin/metrics' > params: > wt: ["prometheus"] > static_configs: > - targets: ['localhost:8983'] > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-10654) Expose Metrics in Prometheus format DIRECTLY from Solr
[ https://issues.apache.org/jira/browse/SOLR-10654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17848325#comment-17848325 ] David Smiley commented on SOLR-10654: - A possible way to have it both ways would be to +embed+ the Prometheus Exporter _partially_, without the caching aspect -- it'd be a request handler that fetches metrics locally (would talk to MetricsHandler in a direct way) and post-processes via JQ. I don't love JQ but... hey, some do. XSLT/XQuery is more my thing. No new dependencies to add directly to Solr; people would just add the Exporter's as if it's a module. Regardless of some details, there would still be *some* overhead in this post processing due to JQ. I'm not sure that's the pain point we're solving for here? I haven't measured lately. It could be interesting to compare the performance of the Prometheus Exporter and this patch. > Expose Metrics in Prometheus format DIRECTLY from Solr > -- > > Key: SOLR-10654 > URL: https://issues.apache.org/jira/browse/SOLR-10654 > Project: Solr > Issue Type: Improvement > Components: metrics >Reporter: Keith Laban >Priority: Major > Attachments: prometheus_metrics.txt > > Time Spent: 3h > Remaining Estimate: 0h > > Expose metrics via a `wt=prometheus` response type. > Example scape_config in prometheus.yml: > {code} > scrape_configs: > - job_name: 'solr' > metrics_path: '/solr/admin/metrics' > params: > wt: ["prometheus"] > static_configs: > - targets: ['localhost:8983'] > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-10654) Expose Metrics in Prometheus format DIRECTLY from Solr
[ https://issues.apache.org/jira/browse/SOLR-10654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17847998#comment-17847998 ] David Smiley commented on SOLR-10654: - Overall I wonder what others think about maintaining two parallel yet consistent metrics mappings -- one in the "Prometheus Exporter" configured using lots of "jq" and that which is very flexible (intended for users to configure/hack as needed), the second what this PR does, basically as hard-coded as can be. For example if we add a new metric, we then probably need to update the exporter's config, and also edit source code being added here. This could be helped by having the Prometheus Exporter fetch certain metrics pass-through on-demand. But based on the design of the Prometheus Exporter, I think that could be tricky/awkward. > Expose Metrics in Prometheus format DIRECTLY from Solr > -- > > Key: SOLR-10654 > URL: https://issues.apache.org/jira/browse/SOLR-10654 > Project: Solr > Issue Type: Improvement > Components: metrics >Reporter: Keith Laban >Priority: Major > Attachments: prometheus_metrics.txt > > Time Spent: 3h > Remaining Estimate: 0h > > Expose metrics via a `wt=prometheus` response type. > Example scape_config in prometheus.yml: > {code} > scrape_configs: > - job_name: 'solr' > metrics_path: '/solr/admin/metrics' > params: > wt: ["prometheus"] > static_configs: > - targets: ['localhost:8983'] > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-10654) Expose Metrics in Prometheus format DIRECTLY from Solr
[ https://issues.apache.org/jira/browse/SOLR-10654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17838265#comment-17838265 ] Matthew Biscocho commented on SOLR-10654: - Created a draft [PR|https://github.com/apache/solr/pull/2405] for exposing prometheus metrics. Going to work on writing tests, but put up the draft to get some initial feedback on implementation. Output looks similar to the prometheus exporter. > Expose Metrics in Prometheus format DIRECTLY from Solr > -- > > Key: SOLR-10654 > URL: https://issues.apache.org/jira/browse/SOLR-10654 > Project: Solr > Issue Type: Improvement > Components: metrics >Reporter: Keith Laban >Priority: Major > Attachments: prometheus_metrics.txt > > Time Spent: 20m > Remaining Estimate: 0h > > Expose metrics via a `wt=prometheus` response type. > Example scape_config in prometheus.yml: > {code} > scrape_configs: > - job_name: 'solr' > metrics_path: '/solr/admin/metrics' > params: > wt: ["prometheus"] > static_configs: > - targets: ['localhost:8983'] > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-10654) Expose Metrics in Prometheus format DIRECTLY from Solr
[ https://issues.apache.org/jira/browse/SOLR-10654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17836390#comment-17836390 ] Matthew Biscocho commented on SOLR-10654: - Thanks for the info, [~dsmiley]! Actually came back to this recently since I wrote that last comment and I agree, parallel registry is way too much overhead and would probably make things much more difficult. Currently I have a lot of work written to transform the dropwizard registries to Prometheus with relevant tags for Solr! Looks pretty similar to the Prometheus exporter. Eager to post the first PR to get some feedback on my implementation. Hoping to get it out soon. > Expose Metrics in Prometheus format DIRECTLY from Solr > -- > > Key: SOLR-10654 > URL: https://issues.apache.org/jira/browse/SOLR-10654 > Project: Solr > Issue Type: Improvement > Components: metrics >Reporter: Keith Laban >Priority: Major > Attachments: prometheus_metrics.txt > > Time Spent: 10m > Remaining Estimate: 0h > > Expose metrics via a `wt=prometheus` response type. > Example scape_config in prometheus.yml: > {code} > scrape_configs: > - job_name: 'solr' > metrics_path: '/solr/admin/metrics' > params: > wt: ["prometheus"] > static_configs: > - targets: ['localhost:8983'] > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-10654) Expose Metrics in Prometheus format DIRECTLY from Solr
[ https://issues.apache.org/jira/browse/SOLR-10654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17836368#comment-17836368 ] David Smiley commented on SOLR-10654: - Cool to see [your PR|https://github.com/apache/solr/pull/2375] Mathew :-) You didn't title that PR starting with this JIRA issue, so it didn't auto-link. Also it's good to comment in the JIRA issue about a PR any way since there is no notification whatsoever to those watching this JIRA issue even when the link is auto-added. Disclaimer: I haven't been looking closely at metrics lately. A parallel registry seems painful -- overhead and the synchronization risks. Moving to Micrometer -- do you think it would affect most metrics publishers in Solr (thus touch tons of source files) or only the metrics internals/plumbing? Either way, probably for Solr 10 if we go this way. Maybe there could be a hard-coded algorithmic approach that can convert the raw name to a tagged/labelled one metric? > Expose Metrics in Prometheus format DIRECTLY from Solr > -- > > Key: SOLR-10654 > URL: https://issues.apache.org/jira/browse/SOLR-10654 > Project: Solr > Issue Type: Improvement > Components: metrics >Reporter: Keith Laban >Priority: Major > Attachments: prometheus_metrics.txt > > Time Spent: 10m > Remaining Estimate: 0h > > Expose metrics via a `wt=prometheus` response type. > Example scape_config in prometheus.yml: > {code} > scrape_configs: > - job_name: 'solr' > metrics_path: '/solr/admin/metrics' > params: > wt: ["prometheus"] > static_configs: > - targets: ['localhost:8983'] > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[jira] [Commented] (SOLR-10654) Expose Metrics in Prometheus format DIRECTLY from Solr
[ https://issues.apache.org/jira/browse/SOLR-10654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17799191#comment-17799191 ] Matthew Biscocho commented on SOLR-10654: - So I found some time to look more into a solution to implement these metrics and ran into a few problems mostly around Solr metric collection with Dropwizard. I took a first approach by taking Solr Dropwizard Registry and coverting it with a Util function from the prometheus client library to a Prometheus Registry and use that output in new response writer I made. Doing so gave an output that looks like so [^prometheus_metrics.txt]. I ran into the first issue in that 1 to 1 naming from Dropwizard isn't very useful and doesn't follow [prometheus recommended naming convention. |https://prometheus.io/docs/practices/naming/] The bigger issue is that Dropwizard doesn't support tagging metrics and without tags/label essentially makes these metrics useless for aggregation across Solr nodes. So based on this, the only other way I can find to expose these metrics from Solr was to either: 1. Create a separate [Prometheus Registry|https://prometheus.github.io/client_java/getting-started/registry/] next to the existing Dropwizard Registry and collect these Solr metrics in parallel but with Prometheus having its own naming convention and applying tags/labels 2. Possibly migrate off of Dropwizard and migrate to Micrometer as the new default registry as it supports metric tagging. I did some research on Micrometer and it looks to support [prometheus and a lot of other monitoring systems|https://micrometer.io/docs/concepts#_supported_monitoring_systems] making it possible to output micrometer in prometheus format. I'm still newish to the Solr code base but was hoping to get someone who has experience with metrics in Solr ( maybe [~dsmiley] ) to discuss possible other approaches before going forward and help choose the best? > Expose Metrics in Prometheus format DIRECTLY from Solr > -- > > Key: SOLR-10654 > URL: https://issues.apache.org/jira/browse/SOLR-10654 > Project: Solr > Issue Type: Improvement > Components: metrics >Reporter: Keith Laban >Priority: Major > Attachments: prometheus_metrics.txt > > Time Spent: 10m > Remaining Estimate: 0h > > Expose metrics via a `wt=prometheus` response type. > Example scape_config in prometheus.yml: > {code} > scrape_configs: > - job_name: 'solr' > metrics_path: '/solr/admin/metrics' > params: > wt: ["prometheus"] > static_configs: > - targets: ['localhost:8983'] > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org