Re: [prometheus-users] synthetic histograms in Prometheus

2022-08-07 Thread Johny
The application publishes metrics to a remote-write endpoint in a Prometheus shard at the moment; in future we've plans to migrate to pull model as much as possible after building service discovery for native deployments -- but for backward compatibility, we are adopting this approach currently

Re: [prometheus-users] synthetic histograms in Prometheus

2022-08-07 Thread Ben Kochie
To put it another way. If you can read every event raw from a log line, like every request has a "took X milliseconds", there are better ways to reconstruct metrics for your use case. On Sun, Aug 7, 2022 at 9:46 PM Ben Kochie wrote: > Right, but more basic, how do you get this information from t

Re: [prometheus-users] synthetic histograms in Prometheus

2022-08-07 Thread Ben Kochie
Right, but more basic, how do you get this information from the application right now? Are you reading logs? Does it emit statsd data? You're saying what, but not how. On Sun, Aug 7, 2022 at 7:15 PM Johny wrote: > Gauge contains most recent values of a metric, sampled every 1 min or so, > and e

Re: [prometheus-users] synthetic histograms in Prometheus

2022-08-07 Thread Johny
Thanks. While I understand the limitations with a gauge, the objective here is to backport existing reports with the new backend, integrate and optimize later. There is a period of time we need to continue backward compatibility due to high barrier to change in clients. The time window used to

Re: [prometheus-users] synthetic histograms in Prometheus

2022-08-07 Thread Stuart Clark
On 07/08/2022 18:14, Johny wrote: Gauge contains most recent values of a metric, sampled every 1 min or so, and exported by a user application, e.g. some latency sampled at 1 minute intervals by a client application. Lets presume this time series (scraped by Prometheus or sent via remote write)

Re: [prometheus-users] synthetic histograms in Prometheus

2022-08-07 Thread Johny
Gauge contains most recent values of a metric, sampled every 1 min or so, and exported by a user application, e.g. some latency sampled at 1 minute intervals by a client application. Lets presume this time series (scraped by Prometheus or sent via remote write) is absolute containing all the in

Re: [prometheus-users] synthetic histograms in Prometheus

2022-08-07 Thread Ben Kochie
So, let's take a step back and find out some more information, because this question is sounding a lot like an XY Problem. How are the current applications generating their metrics right now? How are you getting the data to create these histograms? On Sun, Aug 7, 2022 at 9:23 AM Johny wrote: >

Re: [prometheus-users] synthetic histograms in Prometheus

2022-08-07 Thread Stuart Clark
On 07/08/2022 08:23, Johny wrote: We are migrating telemetry backend from legacy database to Prometheus and require estimating percentiles on gauge metrics published by user applications. Estimating percentiles on a gauge metric in Prometheus is not feasible and for a number of reasons, client

[prometheus-users] synthetic histograms in Prometheus

2022-08-07 Thread Johny
We are migrating telemetry backend from legacy database to Prometheus and require estimating percentiles on gauge metrics published by user applications. Estimating percentiles on a gauge metric in Prometheus is not feasible and for a number of reasons, client applications will be difficult to