You might want to read up on modern alerting practices so you can avoid this problem entirely.
https://docs.google.com/document/d/199PqyG3UsyXlwieHaqbGiWVa8eMWi8zzAn0YfcApr8Q/edit https://landing.google.com/sre/sre-book/chapters/monitoring-distributed-systems/ https://landing.google.com/sre/sre-book/chapters/practical-alerting/ http://www.brendangregg.com/usemethod.html https://grafana.com/blog/2018/08/02/the-red-method-how-to-instrument-your-services/ On Wed, Sep 9, 2020 at 3:59 PM [email protected] <[email protected]> wrote: > Hi, > > I tried to find proper soluion in this group but I still can't find > possible one. > > I am monitoring over 1000 AWS instances. > and I have 3 Alert rules for CPU, Memory and Disk. > > There is common threshold for all instances something like, > CPU : 0.7, Memory : 0.7, Disk: 0.7 > -> e.g. For Common (All instances) > - alert: CPU > expr: CPU > 0.7 > ... > - alert: MEM > expr: MEM > 0.7 > ... > > Now, I need to add different threshold for specific instances which it > want to use different threshold for CPU. > - alert: CPU > expr: CPU{instance="AAA"} > 0.8 > > Simplest approach like below, But If there are more instances with > specific case, file will be complexed. > # For specific one > - alert: CPU > expr: CPU{instance="AAA"} > 0.8 > - alert: CPU > expr: CPU{instance="BBB"} > 0.9 > # For common > - alert: CPU > expr: CPU{instance!~"AAA|BBB} > 0.7 > - alert: MEM > expr: MEM > 0.7 > > In this case, Isn't there more efficient way to make rules for specific > and common. > > Thanks for your answer in advance. > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/0cfd686c-106a-4941-b99b-7dc3dec7ba48n%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/0cfd686c-106a-4941-b99b-7dc3dec7ba48n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmqSO0Lcmpr4e_kc7kWRKY6johOLyyWXO3vs7Z%3DTJufx%3Dw%40mail.gmail.com.
