Thanks Brian. This is helpful. Sent from Outlook for Android<https://aka.ms/AAb9ysg> ________________________________ From: prometheus-users@googlegroups.com <prometheus-users@googlegroups.com> on behalf of Brian Candler <b.cand...@pobox.com> Sent: Sunday, January 29, 2023 6:46:12 PM To: Prometheus Users <prometheus-users@googlegroups.com> Subject: Re: [prometheus-users] Re: Prometheus same host multiple endpoints
Probably the ideal would be to fix the problem at source: have an alerting rule which tells you if the same JVM is exposing both sets of metrics, and push it back to the JVM owner. Otherwise your dashboard metrics will need to take into account the duplicates. You could simplify this a bit by having recording rules which take one or the other, taking care that some labels will match and some won't. e.g. expr: foo{job="jmx"} on ignoring(job) foo{job="springboot") Then you can remote write this timeseries. On Saturday, 28 January 2023 at 15:07:55 UTC navee...@gmail.com wrote: Hi Brian/Stuart: The prometheus implementation has around 2000+JVMs and uses two approaches (JMX exporter or SpringBoot Actuator/Micrometer). The implementation is up to the JVM owner. JMX exporter exposes metrics at http://host:port/metrics whereas Micrometer exposes them at http://host:port/actuator/metrics. Given we don't know where metrics might be exposed, we have both of them mentioned in prometheus.yaml. This works fine, given we expect metrics to only come from one source. However, we have found cases where they can come through both http://host:port/metrics and http://host:port/actuator/metrics. For these cases, there are duplicate metrics stored (it is redundant data) and also gets double counted in dashboards. It might be quite possible there are incorrect implementations where JVM is configured to use both JMX exporter and Actuator/Micrometer. Thinking of a way of how we can stop storing the duplicates as we find and correct the implementations. The targets to be scrapped are stored in json files which are referred to by prometheus.yaml. We will be able to help with examples of json files and prometheus.yaml if required. Let us know. Thanks, Naveen On Mon, Jan 23, 2023 at 9:08 PM Brian Candler <b.ca...@pobox.com> wrote: Can you give an specific example of "same metrics are published on two different endpoints" ? You might mean: - two different metric names - the same metric name, but different labels And it might be that you're scraping the same target twice, or you're scraping one target but that target is (for some reason) returning duplicates in the scrape results. Or you might have a more complex scenario, e.g. multiple prometheus servers scraping for redundancy, and then you're combining the results together somehow. > Is it possible to pick one endpoint and discard the other while writing a > PromQL query ? Sure. Just filter in the PromQL query. For example, if you have foo{aaa="bbb",ccc="ddd"} 123.0 foo{aaa="bbb",ccc="fff"} 123.0 and you consider the one with ccc="fff" to be a "duplicate" metric, then foo{ccc!="fff"} might be what you want. Otherwise, you can avoid ingesting the duplicate metrics: - by not scraping the second set in the first place - if they all come from the same scrape, then using metric_relabel_configs to drop the metrics that you don't want to keep On Monday, 23 January 2023 at 14:40:40 UTC kishore...@gmail.com wrote: Hi, We have a situation where same metrics are published on two different endpoints. Is it possible to pick one endpoint and discard the other while writing a PromQL query ? Is it possible to configure Prometheus to collect metrics from only one endpoint? / Kishore -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/c777e2ab-0089-4fa8-8fcd-efe49b95e2een%40googlegroups.com<https://groups.google.com/d/msgid/prometheus-users/c777e2ab-0089-4fa8-8fcd-efe49b95e2een%40googlegroups.com?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com<mailto:prometheus-users+unsubscr...@googlegroups.com>. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/56b73228-b117-4040-9500-80c6ed4c65a4n%40googlegroups.com<https://groups.google.com/d/msgid/prometheus-users/56b73228-b117-4040-9500-80c6ed4c65a4n%40googlegroups.com?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/PU4P216MB1088E9BF3F4D64C819547D20A6D39%40PU4P216MB1088.KORP216.PROD.OUTLOOK.COM.