Re: [prometheus-users] Counter metric resets

[Yaron Bialik]

Thanks!

On Thu, Apr 7, 2022, 19:04 Julius Volz  wrote:

> On Thu, Apr 7, 2022 at 4:52 PM Stuart Clark 
> wrote:
>
>> On 07/04/2022 14:04, Yaron B wrote:
>> > Hello,
>> > we have a counter metric that counts each time a pod is doing a
>> > specific action.
>> > I need to count how many times the pod (actually sum of all the pods
>> > from a certain deployment) did the action over 24 hours.
>> > problem is, the pod is on spot, and when it gets restarted, the
>> > counter resets, so the metric might be 20 at 1:00, but at 2:00 it
>> > might be 3, so when I try to do delta, or sum over time, I am getting
>> > wrong results..
>> > any ideas how can I get the real delta for the action in a 24 hours
>> range?
>>
>> Look at using rate() which handles counter resets. If you multiply the
>> value produced by the time period it is over you would get the number of
>> actions that occurred.
>>
>
> That sounds equivalent to just using increase() - increase() is identical
> to rate(), except that it does not convert the unit to be per-second, but
> keeps it per-whatever-time-interval-you-specified.
>
> But yep, with metrics and resets, this is only ever going to be an
> estimate, and both rate() and increase() do some extrapolation, see also
> https://promlabs.com/blog/2021/01/29/how-exactly-does-promql-calculate-rates
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAPe6rojb4mfFEeL0y_%2BmM-tu0wnSSuk_syfD%3DaDXqifYUkjNug%40mail.gmail.com.

Re: [prometheus-users] CMDB config

[Julius Volz]

Are you talking about config management databases in general, or some
specific one?

In either case, you likely will have to use the http_sd or file_sd custom
service discovery integrations to build your own discovery based on your
CMDB (unless you find something on Google for it already).

See:

* https://prometheus.io/docs/guides/file-sd/
* https://prometheus.io/blog/2018/07/05/implementing-custom-sd/
* https://prometheus.io/docs/prometheus/latest/http_sd/

On Thu, Apr 7, 2022 at 12:18 PM ritesh patel 
wrote:

> Hello Team,
>
> Anyone have CMDB config for Prometheus.yml for target’s.
>
> How i can add targets from CMDB directly?
>
> Thanks and regards
> Ritesh patel
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/CAPxUNF9HO%3DrXqDJ_7K5SHpn-nfWgTExGSt2VY8gY0FSZO2TePQ%40mail.gmail.com
> 
> .
>


-- 
Julius Volz
PromLabs - promlabs.com

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAObpH5yFph_QJUa3kdSM5J%3D%2Bkcgi8Jipqt%3Db7az%3D90U5%3D48F6w%40mail.gmail.com.

Re: [prometheus-users] Counter metric resets

[Julius Volz]

On Thu, Apr 7, 2022 at 4:52 PM Stuart Clark 
wrote:

> On 07/04/2022 14:04, Yaron B wrote:
> > Hello,
> > we have a counter metric that counts each time a pod is doing a
> > specific action.
> > I need to count how many times the pod (actually sum of all the pods
> > from a certain deployment) did the action over 24 hours.
> > problem is, the pod is on spot, and when it gets restarted, the
> > counter resets, so the metric might be 20 at 1:00, but at 2:00 it
> > might be 3, so when I try to do delta, or sum over time, I am getting
> > wrong results..
> > any ideas how can I get the real delta for the action in a 24 hours
> range?
>
> Look at using rate() which handles counter resets. If you multiply the
> value produced by the time period it is over you would get the number of
> actions that occurred.
>

That sounds equivalent to just using increase() - increase() is identical
to rate(), except that it does not convert the unit to be per-second, but
keeps it per-whatever-time-interval-you-specified.

But yep, with metrics and resets, this is only ever going to be an
estimate, and both rate() and increase() do some extrapolation, see also
https://promlabs.com/blog/2021/01/29/how-exactly-does-promql-calculate-rates
.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAObpH5yzyj93gzc6%2BnJnAARwM4Wdc-YG6JqBhJLLC7TbfLfMuA%40mail.gmail.com.

Re: [prometheus-users] Counter metric resets

[Stuart Clark]

On 07/04/2022 14:04, Yaron B wrote:

Hello,
we have a counter metric that counts each time a pod is doing a 
specific action.
I need to count how many times the pod (actually sum of all the pods 
from a certain deployment) did the action over 24 hours.
problem is, the pod is on spot, and when it gets restarted, the 
counter resets, so the metric might be 20 at 1:00, but at 2:00 it 
might be 3, so when I try to do delta, or sum over time, I am getting 
wrong results..

any ideas how can I get the real delta for the action in a 24 hours range?


Look at using rate() which handles counter resets. If you multiply the 
value produced by the time period it is over you would get the number of 
actions that occurred. Note that this will only ever be an estimate (for 
example you might not scrape a pod before it is destroyed, missing the 
detection of some actions) and will most likely not be an integer (due 
to the way interpolation happens).


--
Stuart Clark

--
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/2cf67da0-1566-3cdf-f467-8eda19ac7b9f%40Jahingo.com.

[prometheus-users] Counter metric resets

[Yaron B]

Hello, 
we have a counter metric that counts each time a pod is doing a specific 
action.
I need to count how many times the pod (actually sum of all the pods from a 
certain deployment) did the action over 24 hours.
problem is, the pod is on spot, and when it gets restarted, the counter 
resets, so the metric might be 20 at 1:00, but at 2:00 it might be 3, so 
when I try to do delta, or sum over time, I am getting wrong results..
any ideas how can I get the real delta for the action in a 24 hours range?

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/c750f114-f8b5-4cb8-9efb-d47f2f6807e3n%40googlegroups.com.

[prometheus-users] Re: Calculate 95th for network traffic per month

[Brian Candler]

Yes, see quantile_over_time() 

.

Here are some example recording rules, to calculate this continuously:

groups:
- name: bandwidth_percentiles_daily
  interval: 5m
  rules:
  - record: interface:in_octets:rate5m_95th_24h
expr: quantile_over_time(0.95, 
rate(ifHCInOctets{instance="",ifName=""}[10m])[24h:5m])
  - record: interface:out_octets:rate5m_95th_24h
expr: quantile_over_time(0.95, 
rate(ifHCOutOctets{instance="",ifName=""}[10m])[24h:5m])

- name: bandwidth_percentiles_monthly
  interval: 1h
  rules:
  - record: interface:in_octets:rate5m_95th_30d
expr: quantile_over_time(0.95, 
rate(ifHCInOctets{instance="",ifName=""}[10m])[30d:5m])
  - record: interface:out_octets:rate5m_95th_30d
expr: quantile_over_time(0.95, 
rate(ifHCOutOctets{instance="",ifName=""}[10m])[30d:5m])

Each metric gives the 95th-percentile of the rate taken at 5 minute 
intervals, over the preceding 24 hours or 30 days respectively.

You can of course use these expressions directly in the PromQL browser for 
ad-hoc queries.  You should be able to use the new modifier "@time 
" 
to perform the query at a given instant in time, so will calculate over the 
24 hours or 30 days before that time.

On Thursday, 7 April 2022 at 08:49:45 UTC+1 romu...@gmail.com wrote:

> Hello.
> Is it possible to calculate 95th of network traffic in Prometheus ?
> Thank you!
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/f4a083f4-0ede-4843-a0ed-f50da8858c7dn%40googlegroups.com.

[prometheus-users] CMDB config

[ritesh patel]

Hello Team,

Anyone have CMDB config for Prometheus.yml for target’s.

How i can add targets from CMDB directly?

Thanks and regards
Ritesh patel

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAPxUNF9HO%3DrXqDJ_7K5SHpn-nfWgTExGSt2VY8gY0FSZO2TePQ%40mail.gmail.com.

[prometheus-users] Calculate 95th for network traffic per month

[Roman Melnyk]

Hello.
Is it possible to calculate 95th of network traffic in Prometheus ?
Thank you!

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/4b275b2b-6e42-4c13-8b66-b75a9849ebc8n%40googlegroups.com.