Re: [prometheus-users] Re: missing data point make alert resolved

2020-06-08 Thread Brian Brazil
On Mon, 8 Jun 2020 at 08:47, Brian Candler wrote: > On 08/06/2020 08:23, Brian Brazil wrote: > > As Ben said this is a case for avg_over_time or max_over_time. Looking > > at just the last point would be too fragile, and once an alert fires > > adding additional semantics is only rearranging the

Re: [prometheus-users] Re: missing data point make alert resolved

2020-06-08 Thread Brian Candler
On 08/06/2020 08:23, Brian Brazil wrote: As Ben said this is a case for avg_over_time or max_over_time. Looking at just the last point would be too fragile, and once an alert fires adding additional semantics is only rearranging the deckchairs. See https://www.robustperception.io/alerting-on-ga

Re: [prometheus-users] Re: missing data point make alert resolved

2020-06-08 Thread Brian Brazil
On Mon, 8 Jun 2020 at 07:41, Rajesh Reddy Nachireddi < rajeshredd...@gmail.com> wrote: > Ok, thanks for the clarification. It is a necessary feature rather than a > good-to-have feature. > > @brian.bra...@robustperception.io > As prometheus usage is getting to multiple industries, would it be > p

Re: [prometheus-users] Re: missing data point make alert resolved

2020-06-07 Thread Brian Candler
On Monday, 8 June 2020 06:30:10 UTC+1, Rajesh Reddy Nachireddi wrote: > > it would be helpful to have timer based resolved similar to firing. > >> >> I agree: https://github.com/prometheus/alertmanager/issues/204#issuecomment-527522778 - although I posted that under alertmanager and really it bel

Re: [prometheus-users] Re: missing data point make alert resolved

2020-06-07 Thread Rajesh Reddy Nachireddi
Thanks Julien. I used the similar approach for testing but it is really difficult to write 3 to 4 rules to raise/clear an alert And another proposal was to check the number of times the condition matches FOR duration. This worlds for discrete data as well instead of streaming only. On Mon, Jun 8,

Re: [prometheus-users] Re: missing data point make alert resolved

2020-06-07 Thread Julien Pivotto
Hello This is a use case for a last_over_time function which does not exist and has been rejected in the past. Here is how I solved it: https://roidelapluie.be/blog/2019/02/21/prometheus-last/ Regards Le lun. 8 juin 2020 à 07:30, Rajesh Reddy Nachireddi < rajeshredd...@gmail.com> a écrit : > B

Re: [prometheus-users] Re: missing data point make alert resolved

2020-06-07 Thread Rajesh Reddy Nachireddi
But again, it always makes queries complex and doesn't give reliable results. If we use absent- queries are complex and not much useful when we are not expecting a boolean .. avg_over_time or max_over_time - it smoothens the data but its not reliable.. Do we have a way to distinguish data not p

[prometheus-users] Re: missing data point make alert resolved

2020-06-03 Thread Brian Candler
OK - then as Ben says, use avg_over_time or max_over_time. -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To

[prometheus-users] Re: missing data point make alert resolved

2020-06-03 Thread 林浩
Brain, I sure data is missing, if I use node_processes_state{state='D'} graph will look like this [image: prom1.jpg] 在 2020年6月3日星期三 UTC+8下午5:55:35,Brian Candler写道: > > Are you sure the data is missing? The query > > node_processes_state{state='D'} > 500 > > will only show values in the gr

[prometheus-users] Re: missing data point make alert resolved

2020-06-03 Thread 林浩
Brian, I sure data missing see graph here [image: prom1.jpg] 在 2020年6月3日星期三 UTC+8下午5:55:35,Brian Candler写道: > > Are you sure the data is missing? The query > > node_processes_state{state='D'} > 500 > > will only show values in the graph which are over 500; you will see gaps > when the value is

[prometheus-users] Re: missing data point make alert resolved

2020-06-03 Thread Brian Candler
Are you sure the data is missing? The query node_processes_state{state='D'} > 500 will only show values in the graph which are over 500; you will see gaps when the value is below 500. What do you see if you graph node_processes_state{state='D'} instead? -- You received this message because