Cheers. BTW I mistyped that, it should have been "or" not "on".
expr: foo{job="jmx"} *or* ignoring(job) foo{job="springboot"}
On Monday, 30 January 2023 at 02:07:09 UTC navee...@gmail.com wrote:
> Thanks Brian. This is helpful.
>
> Sent from Outlook for Android <https://aka.ms/AAb9ysg>
> ------------------------------
> *From:* promethe...@googlegroups.com <promethe...@googlegroups.com> on
> behalf of Brian Candler <b.ca...@pobox.com>
> *Sent:* Sunday, January 29, 2023 6:46:12 PM
> *To:* Prometheus Users <promethe...@googlegroups.com>
> *Subject:* Re: [prometheus-users] Re: Prometheus same host multiple
> endpoints
>
> Probably the ideal would be to fix the problem at source: have an alerting
> rule which tells you if the same JVM is exposing both sets of metrics, and
> push it back to the JVM owner.
>
> Otherwise your dashboard metrics will need to take into account the
> duplicates.
>
> You could simplify this a bit by having recording rules which take one or
> the other, taking care that some labels will match and some won't. e.g.
>
> expr: foo{job="jmx"} on ignoring(job) foo{job="springboot")
>
> Then you can remote write this timeseries.
>
> On Saturday, 28 January 2023 at 15:07:55 UTC navee...@gmail.com wrote:
>
> Hi Brian/Stuart:
>
> The prometheus implementation has around 2000+JVMs and uses two approaches
> (JMX exporter or SpringBoot Actuator/Micrometer). The implementation is up
> to the JVM owner.
>
> JMX exporter exposes metrics at http://host:port/metrics whereas
> Micrometer exposes them at http://host:port/actuator/metrics. Given we
> don't know where metrics might be exposed, we have both of them mentioned
> in prometheus.yaml.
>
> This works fine, given we expect metrics to only come from one source.
> However, we have found cases where they can come through both
> http://host:port/metrics and http://host:port/actuator/metrics. For these
> cases, there are duplicate metrics stored (it is redundant data) and also
> gets double counted in dashboards.
>
> It might be quite possible there are incorrect implementations where JVM
> is configured to use both JMX exporter and Actuator/Micrometer. Thinking of
> a way of how we can stop storing the duplicates as we find and correct the
> implementations.
>
> The targets to be scrapped are stored in json files which are referred
> to by prometheus.yaml.
>
> We will be able to help with examples of json files and prometheus.yaml if
> required. Let us know.
>
> Thanks,
> Naveen
>
>
>
> On Mon, Jan 23, 2023 at 9:08 PM Brian Candler <b.ca...@pobox.com> wrote:
>
> Can you give an specific example of "same metrics are published on two
> different endpoints" ?
>
> You might mean:
> - two different metric names
> - the same metric name, but different labels
>
> And it might be that you're scraping the same target twice, or you're
> scraping one target but that target is (for some reason) returning
> duplicates in the scrape results. Or you might have a more complex
> scenario, e.g. multiple prometheus servers scraping for redundancy, and
> then you're combining the results together somehow.
>
> > Is it possible to pick one endpoint and discard the other while writing
> a PromQL query ?
>
> Sure. Just filter in the PromQL query. For example, if you have
>
> foo{aaa="bbb",ccc="ddd"} 123.0
> foo{aaa="bbb",ccc="fff"} 123.0
>
> and you consider the one with ccc="fff" to be a "duplicate" metric, then
>
> foo{ccc!="fff"}
>
> might be what you want.
>
> Otherwise, you can avoid ingesting the duplicate metrics:
> - by not scraping the second set in the first place
> - if they all come from the same scrape, then using metric_relabel_configs
> to drop the metrics that you don't want to keep
>
> On Monday, 23 January 2023 at 14:40:40 UTC kishore...@gmail.com wrote:
>
> Hi,
> We have a situation where same metrics are published on two different
> endpoints. Is it possible to pick one endpoint and discard the other while
> writing a PromQL query ?
> Is it possible to configure Prometheus to collect metrics from only one
> endpoint?
>
> / Kishore
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/c777e2ab-0089-4fa8-8fcd-efe49b95e2een%40googlegroups.com
>
> <https://groups.google.com/d/msgid/prometheus-users/c777e2ab-0089-4fa8-8fcd-efe49b95e2een%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/56b73228-b117-4040-9500-80c6ed4c65a4n%40googlegroups.com
>
> <https://groups.google.com/d/msgid/prometheus-users/56b73228-b117-4040-9500-80c6ed4c65a4n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/861c7610-45ee-4fe3-981e-ce9de36fccddn%40googlegroups.com.
