Ok I see problem, but how can i separate email subjects from alert and
alert-resolved? Can I somehow insert 2 templates of email or 2 templates of
subject into 1 alert? How alertmanager will know when to use 1st template
and second?
pt., 14 maj 2021 o 08:45 Sebastian Glock <sebagloc...@gmail.com> napisał(a):
> Ok so this one should look like this:
>
> - alert: CPU load is more than 80%!
> expr: 100 - (1 -avg(irate(windows_cpu_time_total{mode="user"}[10m]))
> by (instance)) * 100 >= 40
> for: 10s
> labels:
> severity: "[High]"
> annotations:
> summary: "CPU load is more than 80%!"
> description: "{{ humanize $value }}%"
>
>
> and alertmanager:
>
> - receiver: alert-emailer-cpu-1h-resolved
> repeat_interval: 1m
> match:
> severity: "[High-CPU]"
>
>
> - name: alert-emailer-cpu-1h-resolved
> email_configs:
> - to: 'sebastian.glock@ecom.software'
> send_resolved: true
> from: ****
> smarthost: ****
> auth_username: '****
> auth_password: ****
> auth_secret: ****
> auth_identity: ***
> html: '{{ template "email" .}}'
> headers:
> subject: "[CUK][RESOLVED]{{ .CommonLabels.severity }} {{
> .CommonLabels.instance }} {{ .CommonLabels.alertname }} | {{
> .CommonAnnotations.description }}"
>
> I'm right or something wrong? What if I want to configure send resolved
> for all alerts? I have many recivers based on severity in recivers and i
> want only to see mail with "[RESOLVED]" in subject.
>
> I think there's something wrong because it's not working :(
>
> śr., 31 mar 2021 o 22:31 Matthias Rampke <matth...@prometheus.io>
> napisał(a):
>
>> To Prometheus, "resolved" is not a separate first class event.
>>
>> As long as an alert is "firing" in Prometheus, it sends a constant stream
>> of "still firing" messages to Alertmanager. If there are multiple alerts
>> firing, or multiple label combinations for one alert, there's more
>> notifications. AM takes these streams of "still bad", deduplicates, and
>> groups them.
>>
>> This leads to an alert group with 1 or more alerts in it. In configurable
>> intervals, Alertmanager will send a notification to the configured
>> receiver(s). This is when the templates are evaluated. While a group
>> exists, individual alerts can be added and removed, this often triggers
>> additional notifications. Finally, when the last alert in a group is
>> resolved, another notification is fired.
>>
>> It is important here that this notification uses the same receiver, and
>> the same templates, as all the previous ones! The choice of Slack color is
>> only done based on the state of each notification, you can see how it's
>> done in the default value of the "color" setting:
>> https://prometheus.io/docs/alerting/latest/configuration/#slack_config
>>
>> You need only one receiver and set send_resolved: true on it. Then you
>> can use templating to make these emails have different subjects depending
>> on the contents of each notification.
>>
>> /MR
>>
>> On Wed, Mar 31, 2021, 09:19 Sebastian Glock <sebagloc...@gmail.com>
>> wrote:
>>
>>> I have something like this:
>>>
>>> global:
>>> route:
>>> receiver: alert-emailer-cpu-30m
>>>
>>> group_by: ['alertname', 'priority', 'instance']
>>> group_wait: 10s
>>> group_interval: 10s
>>> repeat_interval: 30m
>>> # resolve_timeout: 10s
>>> routes:
>>>
>>>
>>> #rebooted
>>>
>>>
>>> - receiver: alert-emailer-reboot-30m
>>> repeat_interval: 30m
>>> match:
>>> severity: "[Rebooted]"
>>>
>>>
>>> #CPU
>>>
>>> - receiver: alert-emailer-cpu-30m
>>> repeat_interval: 1m
>>> match:
>>> severity: "[Disaster-CPU]"
>>>
>>>
>>> - receiver: alert-emailer-cpu-1h
>>> repeat_interval: 1m
>>> match:
>>> severity: "[High-CPU]"
>>>
>>>
>>> - receiver: alert-emailer-cpu-3h
>>> repeat_interval: 1m
>>> match:
>>> severity: "[Average-CPU]"
>>>
>>>
>>> #resolved
>>>
>>> - receiver: alert-emailer-cpu-30m-resolved
>>> repeat_interval: 1m
>>> match:
>>> severity: "[Disaster-CPU]"
>>>
>>>
>>> - receiver: alert-emailer-cpu-1h-resolved
>>> repeat_interval: 1m
>>> match:
>>> severity: "[High-CPU]"
>>>
>>>
>>> - receiver: alert-emailer-cpu-3h-resolved
>>> repeat_interval: 1m
>>> match:
>>> severity: "[Average-CPU]"
>>>
>>>
>>> #cpu
>>>
>>> - name: alert-emailer-cpu-1h
>>> email_configs:
>>> - to: 'sebastian.glock@ecom.software'
>>> send_resolved: false
>>> from: '******'
>>> smarthost: '******'
>>> auth_username: '******'
>>> auth_password: '******'
>>> auth_secret: '******'
>>> auth_identity:'******'
>>> html: '{{ template "email" .}}'
>>> headers:
>>> subject: "[CUK]{{ .CommonLabels.severity }} {{
>>> .CommonLabels.instance }} {{ .CommonLabels.alertname }} | {{
>>> .CommonAnnotations.description }}"
>>>
>>> - name: alert-emailer-cpu-1h-resolved
>>> email_configs:
>>> - to: 'sebastian.glock@ecom.software'
>>> send_resolved: true
>>> from: '******'
>>> smarthost: '******'
>>> auth_username: '******'
>>> auth_password: '******'
>>> auth_secret: '******'
>>> auth_identity:'******'
>>> html: '{{ template "email" .}}'
>>> headers:
>>> subject: "[CUK][RESOLVED]{{ .CommonLabels.severity }} {{
>>> .CommonLabels.instance }} {{ .CommonLabels.alertname }} | {{
>>> .CommonAnnotations.description }}"
>>>
>>> - name: alert-emailer-cpu-30m
>>> email_configs:
>>> - to: 'sebastian.glock@ecom.software'
>>> send_resolved: false
>>> from: '******'
>>> smarthost: '******'
>>> auth_username: '******'
>>> auth_password: '******'
>>> auth_secret: '******'
>>> auth_identity:'******'
>>> html: '{{ template "email" .}}'
>>> headers:
>>> subject: "[CUK]{{ .CommonLabels.severity }} {{
>>> .CommonLabels.instance }} {{ .CommonLabels.alertname }} | {{
>>> .CommonAnnotations.description }}"
>>>
>>> - name: alert-emailer-cpu-30m-resolved
>>> email_configs:
>>> - to: 'sebastian.glock@ecom.software'
>>> send_resolved: true
>>> from: '******'
>>> smarthost: '******'
>>> auth_username: '******'
>>> auth_password: '******'
>>> auth_secret: '******'
>>> auth_identity:'******'
>>> html: '{{ template "email" .}}'
>>> headers:
>>> subject: "[CUK][RESOLVED]{{ .CommonLabels.severity }} {{
>>> .CommonLabels.instance }} {{ .CommonLabels.alertname }} | {{
>>> .CommonAnnotations.description }}"
>>>
>>>
>>> @Matthias Rampke <matth...@prometheus.io> so U think i should add
>>> .Alerts.Resolved: {{ if gt (len .Alerts.Firing) 0 }} to alert? Where
>>> prometheus hold's that alert is resolved? I thought in slack it will show
>>> that alert has been resolved, but in prometheus that alert is just green
>>> and show nothing.
>>>
>>> wt., 30 mar 2021 o 12:24 Matthias Rampke <matth...@prometheus.io>
>>> napisał(a):
>>>
>>>> I *think* you can have one email config, and toggle whether to show
>>>> [RESOLVED] as part of the template. Looking at the default templates
>>>> <https://github.com/prometheus/alertmanager/blob/master/template/default.tmpl>,
>>>> it seems like the way to do that is to check for the length of
>>>> .Alerts.Firing and .Alerts.Resolved: {{ if gt (len .Alerts.Firing) 0 }}
>>>> etc.
>>>>
>>>> /MR
>>>>
>>>> On Tue, Mar 30, 2021 at 9:02 AM Stuart Clark <stuart.cl...@jahingo.com>
>>>> wrote:
>>>>
>>>>> On 29/03/2021 14:20, Sebastian Glock wrote:
>>>>> > I tried to do something like this:
>>>>> >
>>>>> > ```
>>>>> > - name: alert-emailer-cpu-1h
>>>>> > email_configs:
>>>>> > - to: 'sebastian.glock@ecom.software'
>>>>> > send_resolved: false
>>>>> > from: '*****'
>>>>> > smarthost:'*****'
>>>>> > auth_username: '*****'
>>>>> > auth_password: '*****'
>>>>> > auth_secret: '*****'
>>>>> > auth_identity:'*****'
>>>>> > html: '{{ template "email" .}}'
>>>>> > headers:
>>>>> > subject: "[CUK]{{ .CommonLabels.severity }} {{
>>>>> > .CommonLabels.instance }} {{ .CommonLabels.alertname }} | {{
>>>>> > .CommonAnnotations.description }}"
>>>>> >
>>>>> > - name: alert-emailer-cpu-1h-resolved
>>>>> > email_configs:
>>>>> > - to: 'sebastian.glock@ecom.software'
>>>>> > send_resolved: true
>>>>> > from: '*****'
>>>>> > smarthost: '*****'
>>>>> > auth_username: '*****'
>>>>> > auth_password: '*****'
>>>>> > auth_secret: '*****'
>>>>> > auth_identity: '*****'
>>>>> > html: '{{ template "email" .}}'
>>>>> > headers:
>>>>> > subject: "[CUK][RESOLVED]{{ .CommonLabels.severity }} {{
>>>>> > .CommonLabels.instance }} {{ .CommonLabels.alertname }} | {{
>>>>> > .CommonAnnotations.description }}"
>>>>> > ```
>>>>> >
>>>>> > 1 alert without send resolved, second with send resolved: true. I
>>>>> > thought i'll get an email "alert-emailer-cpu-1h-resolved" after 5
>>>>> > minutes when 1st alert "alert-emailer-cpu-1h" goes out. Is there any
>>>>> > other way to solve that?
>>>>> >
>>>>> > Thanks for your help!
>>>>>
>>>>> Could you post the whole alertmanager config?
>>>>>
>>>>> --
>>>>> Stuart Clark
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "Prometheus Users" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to prometheus-users+unsubscr...@googlegroups.com.
>>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/prometheus-users/c6ee84ab-4b5b-9dde-db46-ae45f0dc8944%40Jahingo.com
>>>>> .
>>>>>
>>>>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/CAFn4LSqsPpg6FXpPttqFFR5CXi%3Dv5t3MpL7HC7GitybaFfpF-w%40mail.gmail.com.
