Re: [prometheus-users] Guaranteed ingestion of metrics with historical timestamps

2022-06-18 Thread Ben Kochie
For this use case, it's likely what they want is Prometheus in agent mode, which uses remote write, which can buffer and catch up. On Sat, Jun 18, 2022, 1:13 PM Stuart Clark wrote: > On 14/06/2022 18:32, Jeremy Collette wrote: > > Hello, > > We have written a custom exporter that exposes metrics

Re: [prometheus-users] Guaranteed ingestion of metrics with historical timestamps

2022-06-18 Thread Stuart Clark
On 14/06/2022 18:32, Jeremy Collette wrote: Hello, We have written a custom exporter that exposes metrics with explicit timestamps, which Prometheus periodically scrapes. In the case where Prometheus becomes temporarily unavailable, these metric samples will be cached in the exporter until th

[prometheus-users] Central pushgateway: limits/sizing for pushgateway

2022-06-18 Thread DerekLai Devops
We have been using Prometheus to monitor our AKS clusters for the various departments. Each has their own AKS cluster for ease of management/billing. Now the users would like to setup pushgateways to help with getting metrics on jobs which is shorted lived and can't be scraped. Management would

[prometheus-users] Guaranteed ingestion of metrics with historical timestamps

2022-06-18 Thread Jeremy Collette
Hello, We have written a custom exporter that exposes metrics with explicit timestamps, which Prometheus periodically scrapes. In the case where Prometheus becomes temporarily unavailable, these metric samples will be cached in the exporter until they are scraped, causing affected metrics to a

Re: [prometheus-users] Re: Does Prometheus recommend exposing 2M timeseries per scrape endpoint?

2022-06-18 Thread Matthias Rampke
One place where time series of this magnitude from a single target are unfortunately common is kube-state-metrics (KSM). On a large cluster, I see almost 1M metrics. Those are relatively cheap because they are nearly constant and compress well, but