The application publishes metrics to a remote-write endpoint in a
Prometheus shard at the moment; in future we've plans to migrate to pull
model as much as possible after building service discovery for native
deployments -- but for backward compatibility, we are adopting this
approach currently.
On Sunday, August 7, 2022 at 3:48:44 PM UTC-4 sup...@gmail.com wrote:
> To put it another way. If you can read every event raw from a log line,
> like every request has a "took X milliseconds", there are better ways to
> reconstruct metrics for your use case.
>
> On Sun, Aug 7, 2022 at 9:46 PM Ben Kochie <sup...@gmail.com> wrote:
>
>> Right, but more basic, how do you get this information from the
>> application right now? Are you reading logs? Does it emit statsd data?
>>
>> You're saying what, but not how.
>>
>> On Sun, Aug 7, 2022 at 7:15 PM Johny <mailitt...@gmail.com> wrote:
>>
>>> Gauge contains most recent values of a metric, sampled every 1 min or
>>> so, and exported by a user application, e.g. some latency sampled at 1
>>> minute intervals by a client application. Lets presume this time series
>>> (scraped by Prometheus or sent via remote write) is absolute containing all
>>> the information we need for calculating derived statistics. In the most raw
>>> form, you can fetch the data points, sort them and calculate percentile.
>>> Incidentally, legacy backend has efficient mechanisms to calculate
>>> percentiles by scanning and reducing data using map-reduce.
>>>
>>>
>>>
>>> On Sunday, August 7, 2022 at 7:49:05 AM UTC-4 sup...@gmail.com wrote:
>>>
>>>> So, let's take a step back and find out some more information, because
>>>> this question is sounding a lot like an XY Problem.
>>>>
>>>> How are the current applications generating their metrics right now?
>>>> How are you getting the data to create these histograms?
>>>>
>>>> On Sun, Aug 7, 2022 at 9:23 AM Johny <mailitt...@gmail.com> wrote:
>>>>
>>>>> We are migrating telemetry backend from legacy database to Prometheus
>>>>> and require estimating percentiles on gauge metrics published by user
>>>>> applications. Estimating percentiles on a gauge metric in Prometheus is
>>>>> not
>>>>> feasible and for a number of reasons, client applications will be
>>>>> difficult
>>>>> to modify to start publishing histograms.
>>>>>
>>>>> I am exploring feasibility of creating a histogram in a recording rule
>>>>> in Prometheus based on the metrics published by users. The partial work
>>>>> put
>>>>> in so far seems inefficient, also illegible. Is there a recommended
>>>>> approach to solve this problem? As stated earlier, it will be extremely
>>>>> hard to solve the problem on the client side and I am looking for a
>>>>> solution within Prometheus.
>>>>>
>>>>> *Current metric is a gauge with with values representing request
>>>>> latency.*
>>>>> http_duration_milliseconds_gauge{instance="instance1:port1"}[1h]
>>>>> 1659752188 100
>>>>> 1659752068 120
>>>>> ..
>>>>> 1659751708 150
>>>>> 1659751588 160
>>>>>
>>>>> *Desired histogram after conversion -*
>>>>> http_duration_milliseconds_hist_bucket{instance="instance1:port1",
>>>>> le=100} 133
>>>>> http_duration_milliseconds_hist_bucket{instance="instance1:port1",
>>>>> le=120} 222
>>>>> http_duration_milliseconds_hist_bucket{instance="instance1:port1",
>>>>> le=140} 311
>>>>> http_duration_milliseconds_hist_bucket{instance="instance1:port1",
>>>>> le=160} 330
>>>>> http_duration_milliseconds_hist_bucket{instance="instance1:port1",
>>>>> le=180} 339
>>>>> http_duration_milliseconds_hist_bucket{instance="instance1:port1",
>>>>> le=200} 340
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "Prometheus Users" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to prometheus-use...@googlegroups.com.
>>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/prometheus-users/f95b5512-1c81-4e12-9670-7c7eb0d29f5en%40googlegroups.com
>>>>>
>>>>> <https://groups.google.com/d/msgid/prometheus-users/f95b5512-1c81-4e12-9670-7c7eb0d29f5en%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>>> .
>>>>>
>>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Prometheus Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to prometheus-use...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/prometheus-users/c8e3c184-e88d-4217-badc-f5f779b52af3n%40googlegroups.com
>>>
>>> <https://groups.google.com/d/msgid/prometheus-users/c8e3c184-e88d-4217-badc-f5f779b52af3n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/905b945d-ea00-4d4d-9f2b-89ed8a76fab8n%40googlegroups.com.
