[prometheus-users] Extracting long queries from multiple histograms

Dear Colleages, There is a web app which exports its metrics as multiple histograms, one histogram per Web endpoint. So each set of histogram data is also labelled by the {endpoint} label. There are about 50 endpoints so about 50 histograms. I would like to detect and graph slow endpoints, that i

Re: [prometheus-users] Facing 5m staleness issue even with 2.x

It depends on: 1. How often Gatling sends its graphite metrics 2. How often Prometheus scrapes graphite-exporter If Prometheus is scraping graphite-exporter every 15 seconds, then you'll need to keep --graphite.sample-expiry to at least 15 seconds; otherwise you may lose the last metric value wr

Re: [prometheus-users] Facing 5m staleness issue even with 2.x

Thanks a lot Brian.. Setting --graphite.sample-expiry flag solved the issue. For now, I have kept it to 15 seconds... any guidance on how to decide this correct value would be appreciated. On Tue, Apr 19, 2022, 4:56 PM Aniket Kulkarni wrote: > Thanks for the response Stuart.. > > To explain you

Re: [prometheus-users] Facing 5m staleness issue even with 2.x

This is an issue with graphite-exporter, not prometheus or staleness. The problem is this: if your application simply stops sending data to graphite-exporter, then graphite-exporter has no idea whether the time series has finished or not, so it keeps exporting it for a while. See https://github.

Re: [prometheus-users] Facing 5m staleness issue even with 2.x

Thanks for the response Stuart.. To explain you more.. I am load testing an application through Gatling scripts (similar to jmeter). Now I want to have a real time monitoring of this load test. For this, Gatling supports graphite writer protocol(it can't directly talk with prometheus hence I hav

Re: [prometheus-users] Facing 5m staleness issue even with 2.x

On 2022-04-19 08:58, Aniket Kulkarni wrote: Hi, I have referred below links: I understand this was a problem with 1.x https://github.com/prometheus/prometheus/issues/398 I also got this link as a solution https://promcon.io/2017-munich/talks/staleness-in-prometheus-2-0/ No doubt it's a great

[prometheus-users] Facing 5m staleness issue even with 2.x

Hi, I have referred below links: I understand this was a problem with 1.x https://github.com/prometheus/prometheus/issues/398 I also got this link as a solution https://promcon.io/2017-munich/talks/staleness-in-prometheus-2-0/ No doubt it's a great session. But I am still not clear as to what c

Re: [prometheus-users] Re: Prometheus inner metrics can't be remote write

Thanks a lot. my mistake. On Mon, Apr 18, 2022 at 8:19 PM Brian Candler wrote: > By "inner metrics" I guess you mean "internal metrics". These *are* > written to the remote storage. For example: I use VictoriaMetrics for > remote storage, and the "up" metrics *do* propagate there. > > So I thi

[prometheus-users] Re: HTTP status 503 service unavailable: json_exporter with basic_auth

Also you should realise that if you set "basic_auth" in prometheus.yml, this only sets basic auth for the HTTP request from prometheus to json_exporter, not from json_exporter to target. Does the target endpoint http://localhost:9015/services/v2/mpoints/E_TIP5/statistics require authentication

[prometheus-users] Re: HTTP status 503 service unavailable: json_exporter with basic_auth

The configuration of prometheus isn't really of interest, because it's json_exporter that's returning the error. Scrape the exporter by hand: curl -vg 'http://localhost:7979/probe?target=http:%2f%2flocalhost%3a9015%2fservices%2fv2%2fmpoints%2fE_TIP5%2fstatistics' I suspect you'll see the 503 e