Re: [prometheus-developers] Most recent Timeseries in case of GAUGE metrics
The pushgateway does use the prometheus pull mechanism. I'd really suggest
taking a look at it: https://github.com/prometheus/pushgateway
As for how to export a timestamp, The timestamp metric would just be a
sister metric to your main metric and you would update when you update the
main metric with the current epoch time. So if your metrics were:
my_gauge{}
my_guage_ts{...}
Then your query to return the most recent value would look something like:
my_guage and max(my_gauge_ts)
--Matt
On Tue, Sep 5, 2023, 12:40 sunil sahu wrote:
> Thanks for quick response
> Yes you understood it correctly. Earlier we were using push mechanism only
> for statsd format metrics from our custom/on promise instaces. But now want
> to use promql format with pull mechanism.
>
> It would be great help islf you can share example for exporting
> timestamps with gauge metrics? Would it be 1 metric/pod or 1 for each
> metrics?
>
>
> Regards,
> Sunil
>
>
> On Tue, 5 Sept 2023, 9:44 am Matt Doughty, wrote:
>
>> So if I'm understanding you correctly. You have multiple pods
>> exporting a gauge based on a remote request, and because it is load
>> balanced the request could go to one of many pods, and you are trying
>> to figure out which one is the most recent? This sounds like a
>> perfect use case for pushgateway as it will have the latest value, and
>> the metric will survive pod death.
>>
>> That said, if you are married to the current pattern you have set up,
>> the answer would be to export a corresponding timestamp metric so you
>> can select the most recently updated pod's gauge value. I hope this
>> helps.
>>
>> --Matt
>>
>> On Mon, Sep 4, 2023 at 3:23 PM Stuart Clark
>> wrote:
>> >
>> > On 25/08/2023 04:37, sunil sahu wrote:
>> >
>> > Hello All,
>> >
>> > I need help in my below use case.
>> >
>> > We have a centralized python application which continuously monitors
>> state for other applications by running tests. We creates GUAGE metrics
>> with a value (Ex. 100) and set to different value when state changes (Ex.
>> 200 or 300) using gauge's set function (python prometheus_client
>> instrumentation). This run frequently (1min, 5m etc) depending on tests.
>> >
>> > Our central application runs with multiples pods in load balancing. So
>> any pod can serve the request and set the metric value to new/existing one.
>> >
>> > We are scraping individual pods every minute using k8s discovery (role:
>> pods), now here any of the pod/instance can have latest/recent updated
>> value. Now my problem is how to get the metric with latest value out of all
>> instances' metrics.
>> >
>> > Aggregations on 'instance' label like sum/min or max won't be a ideal
>> choice here, may be `last` (which I don't find. I am aware of
>> last_over_time but that is kind of to fill the data-point gaps with
>> previous values)
>> >
>> > Currently I am living with `last_over_time(max without(instance)
>> (metric_name))` but as I explained it is giving wrong results for flaky
>> tests.
>> >
>> >
>> > I tried to explain as best as possible, still let me know if any
>> missing gaps to understand the scenario.
>> >
>> > So are you saying that the multiple pods which are all getting scraped
>> might give totally different values for this metric, but only one of those
>> is correct?
>> >
>> > If that is the case then you'd need to ensure the correct value is
>> returned by every pod, or you store that value elsewhere and scrape that
>> other system.
>> >
>> > --
>> > Stuart Clark
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> Groups "Prometheus Developers" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> an email to prometheus-developers+unsubscr...@googlegroups.com.
>> > To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-developers/37460f3a-1ef0-69b5-d3ad-02cf2f8e4154%40Jahingo.com
>> .
>>
>>
>>
>> --
>> --Matt
>>
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-developers+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-developers/CAGyBzczvoh8GFLYrHH2y31swyUqQyNhDhb3298BM7x_1KHcq%2BA%40mail.gmail.com.
Re: [prometheus-developers] Most recent Timeseries in case of GAUGE metrics
Thanks for quick response Yes you understood it correctly. Earlier we were using push mechanism only for statsd format metrics from our custom/on promise instaces. But now want to use promql format with pull mechanism. It would be great help islf you can share example for exporting timestamps with gauge metrics? Would it be 1 metric/pod or 1 for each metrics? Regards, Sunil On Tue, 5 Sept 2023, 9:44 am Matt Doughty, wrote: > So if I'm understanding you correctly. You have multiple pods > exporting a gauge based on a remote request, and because it is load > balanced the request could go to one of many pods, and you are trying > to figure out which one is the most recent? This sounds like a > perfect use case for pushgateway as it will have the latest value, and > the metric will survive pod death. > > That said, if you are married to the current pattern you have set up, > the answer would be to export a corresponding timestamp metric so you > can select the most recently updated pod's gauge value. I hope this > helps. > > --Matt > > On Mon, Sep 4, 2023 at 3:23 PM Stuart Clark > wrote: > > > > On 25/08/2023 04:37, sunil sahu wrote: > > > > Hello All, > > > > I need help in my below use case. > > > > We have a centralized python application which continuously monitors > state for other applications by running tests. We creates GUAGE metrics > with a value (Ex. 100) and set to different value when state changes (Ex. > 200 or 300) using gauge's set function (python prometheus_client > instrumentation). This run frequently (1min, 5m etc) depending on tests. > > > > Our central application runs with multiples pods in load balancing. So > any pod can serve the request and set the metric value to new/existing one. > > > > We are scraping individual pods every minute using k8s discovery (role: > pods), now here any of the pod/instance can have latest/recent updated > value. Now my problem is how to get the metric with latest value out of all > instances' metrics. > > > > Aggregations on 'instance' label like sum/min or max won't be a ideal > choice here, may be `last` (which I don't find. I am aware of > last_over_time but that is kind of to fill the data-point gaps with > previous values) > > > > Currently I am living with `last_over_time(max without(instance) > (metric_name))` but as I explained it is giving wrong results for flaky > tests. > > > > > > I tried to explain as best as possible, still let me know if any missing > gaps to understand the scenario. > > > > So are you saying that the multiple pods which are all getting scraped > might give totally different values for this metric, but only one of those > is correct? > > > > If that is the case then you'd need to ensure the correct value is > returned by every pod, or you store that value elsewhere and scrape that > other system. > > > > -- > > Stuart Clark > > > > -- > > You received this message because you are subscribed to the Google > Groups "Prometheus Developers" group. > > To unsubscribe from this group and stop receiving emails from it, send > an email to prometheus-developers+unsubscr...@googlegroups.com. > > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-developers/37460f3a-1ef0-69b5-d3ad-02cf2f8e4154%40Jahingo.com > . > > > > -- > --Matt > -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAP3yZ-rhpy%2BA0SGpWVkF1fZmnp6DGv5X9mbxR%3DXdXffv3OHy8w%40mail.gmail.com.
Re: [prometheus-developers] Most recent Timeseries in case of GAUGE metrics
So if I'm understanding you correctly. You have multiple pods exporting a gauge based on a remote request, and because it is load balanced the request could go to one of many pods, and you are trying to figure out which one is the most recent? This sounds like a perfect use case for pushgateway as it will have the latest value, and the metric will survive pod death. That said, if you are married to the current pattern you have set up, the answer would be to export a corresponding timestamp metric so you can select the most recently updated pod's gauge value. I hope this helps. --Matt On Mon, Sep 4, 2023 at 3:23 PM Stuart Clark wrote: > > On 25/08/2023 04:37, sunil sahu wrote: > > Hello All, > > I need help in my below use case. > > We have a centralized python application which continuously monitors state > for other applications by running tests. We creates GUAGE metrics with a > value (Ex. 100) and set to different value when state changes (Ex. 200 or > 300) using gauge's set function (python prometheus_client instrumentation). > This run frequently (1min, 5m etc) depending on tests. > > Our central application runs with multiples pods in load balancing. So any > pod can serve the request and set the metric value to new/existing one. > > We are scraping individual pods every minute using k8s discovery (role: > pods), now here any of the pod/instance can have latest/recent updated value. > Now my problem is how to get the metric with latest value out of all > instances' metrics. > > Aggregations on 'instance' label like sum/min or max won't be a ideal choice > here, may be `last` (which I don't find. I am aware of last_over_time but > that is kind of to fill the data-point gaps with previous values) > > Currently I am living with `last_over_time(max without(instance) > (metric_name))` but as I explained it is giving wrong results for flaky tests. > > > I tried to explain as best as possible, still let me know if any missing gaps > to understand the scenario. > > So are you saying that the multiple pods which are all getting scraped might > give totally different values for this metric, but only one of those is > correct? > > If that is the case then you'd need to ensure the correct value is returned > by every pod, or you store that value elsewhere and scrape that other system. > > -- > Stuart Clark > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Developers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to prometheus-developers+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-developers/37460f3a-1ef0-69b5-d3ad-02cf2f8e4154%40Jahingo.com. -- --Matt -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAGyBzcxTa6-UT7KHBKdoTnPbP9nDR%2BfUArdtGqtvftE9v9DMmQ%40mail.gmail.com.
Re: [prometheus-developers] Most recent Timeseries in case of GAUGE metrics
On 25/08/2023 04:37, sunil sahu wrote: Hello All, I need help in my below use case. We have a centralized python application which continuously monitors state for other applications by running tests. We creates *GUAGE* metrics with a value (Ex. 100) and set to different value when state changes (Ex. 200 or 300) using gauge's set function (python prometheus_client instrumentation). This run frequently (1min, 5m etc) depending on tests. Our central application runs with multiples pods in load balancing. So any pod can serve the request and set the metric value to new/existing one. We are scraping individual pods every minute using k8s discovery (role: pods), now here any of the pod/instance can have latest/recent updated value. N*ow my problem is how to get the metric with latest value out of all instances' metrics.* Aggregations on 'instance' label like sum/min or max won't be a ideal choice here, may be `last` (which I don't find. I am aware of last_over_time but that is kind of to fill the data-point gaps with previous values) Currently I am living with `last_over_time(max without(instance) (metric_name))` but as I explained it is giving wrong results for flaky tests. I tried to explain as best as possible, still let me know if any missing gaps to understand the scenario. So are you saying that the multiple pods which are all getting scraped might give totally different values for this metric, but only one of those is correct? If that is the case then you'd need to ensure the correct value is returned by every pod, or you store that value elsewhere and scrape that other system. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/37460f3a-1ef0-69b5-d3ad-02cf2f8e4154%40Jahingo.com.
[prometheus-developers] Most recent Timeseries in case of GAUGE metrics
Hello All, I need help in my below use case. We have a centralized python application which continuously monitors state for other applications by running tests. We creates *GUAGE* metrics with a value (Ex. 100) and set to different value when state changes (Ex. 200 or 300) using gauge's set function (python prometheus_client instrumentation). This run frequently (1min, 5m etc) depending on tests. Our central application runs with multiples pods in load balancing. So any pod can serve the request and set the metric value to new/existing one. We are scraping individual pods every minute using k8s discovery (role: pods), now here any of the pod/instance can have latest/recent updated value. N*ow my problem is how to get the metric with latest value out of all instances' metrics.* Aggregations on 'instance' label like sum/min or max won't be a ideal choice here, may be `last` (which I don't find. I am aware of last_over_time but that is kind of to fill the data-point gaps with previous values) Currently I am living with `last_over_time(max without(instance) (metric_name))` but as I explained it is giving wrong results for flaky tests. I tried to explain as best as possible, still let me know if any missing gaps to understand the scenario. Thanks in advance. Regards, Sunil -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/59baeea4-ce37-4d74-9f80-762f86e4cd38n%40googlegroups.com.
