Re: [prometheus-users] Counter metric resets
Thanks! On Thu, Apr 7, 2022, 19:04 Julius Volz wrote: > On Thu, Apr 7, 2022 at 4:52 PM Stuart Clark > wrote: > >> On 07/04/2022 14:04, Yaron B wrote: >> > Hello, >> > we have a counter metric that counts each time a pod is doing a >> > specific action. >> > I need to count how many times the pod (actually sum of all the pods >> > from a certain deployment) did the action over 24 hours. >> > problem is, the pod is on spot, and when it gets restarted, the >> > counter resets, so the metric might be 20 at 1:00, but at 2:00 it >> > might be 3, so when I try to do delta, or sum over time, I am getting >> > wrong results.. >> > any ideas how can I get the real delta for the action in a 24 hours >> range? >> >> Look at using rate() which handles counter resets. If you multiply the >> value produced by the time period it is over you would get the number of >> actions that occurred. >> > > That sounds equivalent to just using increase() - increase() is identical > to rate(), except that it does not convert the unit to be per-second, but > keeps it per-whatever-time-interval-you-specified. > > But yep, with metrics and resets, this is only ever going to be an > estimate, and both rate() and increase() do some extrapolation, see also > https://promlabs.com/blog/2021/01/29/how-exactly-does-promql-calculate-rates > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAPe6rojb4mfFEeL0y_%2BmM-tu0wnSSuk_syfD%3DaDXqifYUkjNug%40mail.gmail.com.
Re: [prometheus-users] Counter metric resets
On Thu, Apr 7, 2022 at 4:52 PM Stuart Clark wrote: > On 07/04/2022 14:04, Yaron B wrote: > > Hello, > > we have a counter metric that counts each time a pod is doing a > > specific action. > > I need to count how many times the pod (actually sum of all the pods > > from a certain deployment) did the action over 24 hours. > > problem is, the pod is on spot, and when it gets restarted, the > > counter resets, so the metric might be 20 at 1:00, but at 2:00 it > > might be 3, so when I try to do delta, or sum over time, I am getting > > wrong results.. > > any ideas how can I get the real delta for the action in a 24 hours > range? > > Look at using rate() which handles counter resets. If you multiply the > value produced by the time period it is over you would get the number of > actions that occurred. > That sounds equivalent to just using increase() - increase() is identical to rate(), except that it does not convert the unit to be per-second, but keeps it per-whatever-time-interval-you-specified. But yep, with metrics and resets, this is only ever going to be an estimate, and both rate() and increase() do some extrapolation, see also https://promlabs.com/blog/2021/01/29/how-exactly-does-promql-calculate-rates . -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAObpH5yzyj93gzc6%2BnJnAARwM4Wdc-YG6JqBhJLLC7TbfLfMuA%40mail.gmail.com.
Re: [prometheus-users] Counter metric resets
On 07/04/2022 14:04, Yaron B wrote: Hello, we have a counter metric that counts each time a pod is doing a specific action. I need to count how many times the pod (actually sum of all the pods from a certain deployment) did the action over 24 hours. problem is, the pod is on spot, and when it gets restarted, the counter resets, so the metric might be 20 at 1:00, but at 2:00 it might be 3, so when I try to do delta, or sum over time, I am getting wrong results.. any ideas how can I get the real delta for the action in a 24 hours range? Look at using rate() which handles counter resets. If you multiply the value produced by the time period it is over you would get the number of actions that occurred. Note that this will only ever be an estimate (for example you might not scrape a pod before it is destroyed, missing the detection of some actions) and will most likely not be an integer (due to the way interpolation happens). -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/2cf67da0-1566-3cdf-f467-8eda19ac7b9f%40Jahingo.com.
[prometheus-users] Counter metric resets
Hello, we have a counter metric that counts each time a pod is doing a specific action. I need to count how many times the pod (actually sum of all the pods from a certain deployment) did the action over 24 hours. problem is, the pod is on spot, and when it gets restarted, the counter resets, so the metric might be 20 at 1:00, but at 2:00 it might be 3, so when I try to do delta, or sum over time, I am getting wrong results.. any ideas how can I get the real delta for the action in a 24 hours range? -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/c750f114-f8b5-4cb8-9efb-d47f2f6807e3n%40googlegroups.com.
