On Tue, 10 Mar 2020 at 17:41, Rahul Hada <rh.8...@gmail.com> wrote: > Thanks for the quick response Brian, I was checking same. We are not using > Value as label, only using in annotations. Below is the alert config. > Please have a look and suggest what changes can help us achieve better > alert notifications. One more help i would request is, like below alert > expression, we have alert rule for warning(which is if usage reaches more > than 80%), we get both Critical & warning alert notification if value goes > to more than 90%. Would really appreciate help. >
That looks okay to me, though watch for https://www.robustperception.io/alerting-on-gauges-in-prometheus-2-0 Brian > > - alert: OutOfDiskSpace-Crit > expr: ((node_filesystem_size_bytes{fstype!~"nfs.*"} - > node_filesystem_avail_bytes{fstype!~"nfs.*"}) / > node_filesystem_size_bytes{fstype!~"nfs.*"} * 100 > 90) * on (instance) > group_left(nodename) node_uname_info > for: 5m > labels: > severity: "CRITICAL" > annotations: > summary: "High Disk Usage on *{{ $labels.instance }}* - *{{ > $labels.nodename }}* on the filesystem *{{ $labels.mountpoint }}*" > description: "Current Usage of {{ $labels.mountpoint }} = *{{ $value > | humanize }}%*" > identifier: "*Cluster:* `{{ $labels.cluster }}`, *node:* `{{ > $labels.node }}`, *Mountpoint:* `{{ $labels.mountpoint }}` " > > > On Tuesday, March 10, 2020 at 11:00:30 PM UTC+5:30, Brian Brazil wrote: >> >> On Tue, 10 Mar 2020 at 17:10, Rahul Hada <rh....@gmail.com> wrote: >> >>> I have alerts configured for various metrics, and has set send_resolved >>> : true, now once alert is in active state, we are getting [Firing] emails >>> as configured, but we are getting the [resolved] email even if there is a >>> slight change in the values. For ex:- we get Firing alert when disk usage >>> is 98% and the [resolved] emails comes in if the values goes to 97% or >>> anything. Please suggest how to eradicate this issue. >>> >> >> Sounds like it might be >> https://www.robustperception.io/dont-put-the-value-in-alert-labels >> >> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Prometheus Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to promethe...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-users/a9060291-9ea5-40a0-aec6-cea866d7a3e2%40googlegroups.com >>> <https://groups.google.com/d/msgid/prometheus-users/a9060291-9ea5-40a0-aec6-cea866d7a3e2%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> >> >> -- >> Brian Brazil >> www.robustperception.io >> > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to prometheus-users+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/7c692d96-f542-4c53-a5e2-61ee079d7588%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/7c692d96-f542-4c53-a5e2-61ee079d7588%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- Brian Brazil www.robustperception.io -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAHJKeLqgRJfOLci1qzSY4JkAvhT1fQc%2BuOaRudFWpUL-nfe6kw%40mail.gmail.com.
