Thank you very much Brian! This will be my last post for now. I'm going to change company. I hope you can help my collegues as good as you helped me! This community is great, keep it up. Brian Candler schrieb am Sonntag, 18. Juni 2023 um 19:04:36 UTC+2:
> That will never work. All labels (including severity) are things set in > your alerting rules. You will never see a label with severity=resolved, > unless you actually make an alerting rule which sets severity=resolved when > it fires. > > (match_re does work though; it's the old syntax whereas "matchers" is the > new and preferred syntax) > > Moral: never trust what ChatGPT tells you. > > Off the top of my head, I can't think of any way to send a message when > alerts are resolved but *not* when alerts are firing. > > There are good reasons for not sending *any* resolved messages though: > > https://www.robustperception.io/running-into-burning-buildings-because-the-fire-alarm-stopped > > On Sunday, 18 June 2023 at 12:11:31 UTC Kolja Krückmann wrote: > >> Hi Brian, thanks here are the routes to the receivers: >> >> routes: >> - receiver: email_critical >> group_interval: 1m >> group_wait: 5m >> repeat_interval: 24h >> group_by: ['alertname', 'severity'] >> matchers: >> - severity="critical" >> - severity!="resolved" >> >> - receiver: email_warning >> group_interval: 10m >> group_wait: 5m >> repeat_interval: 24h >> group_by: ['alertname', 'severity'] >> matchers: >> - severity="warning" >> - severity!="resolved" >> >> - receiver: email_resolved >> match_re: >> severity: ^(resolved)$ >> >> >> the match_re part from resolved is from good ol' friend chatgpt >> >> Brian Candler schrieb am Samstag, 17. Juni 2023 um 00:48:14 UTC+2: >> >>> You'll need to show the rest of your alertmanager config. "Receivers" by >>> themselves don't do anything; you have to refer to them in routing rules. >>> >>> On Friday, 16 June 2023 at 09:23:56 UTC Kolja Krückmann wrote: >>> >>>> Or do I need to change it to explicitly say in the crit and warn mail >>>> to range over the .Alerts.Firing in order to get the third receiver to >>>> work? >>>> >>>> receivers: >>>> - name: 'email' >>>> email_configs: >>>> - to: 'sysw...@xxx.com' >>>> send_resolved: false >>>> headers: >>>> subject: 'Critical: {{ .GroupLabels.alertname }} - Instances: {{ >>>> range .Alerts.Firing }}{{ .Labels.instance }}, {{ end }}' >>>> >>>> >>>> - name: 'email_warning' >>>> email_configs: >>>> - to: 'sysw...@xxx.com' >>>> send_resolved: false >>>> headers: >>>> subject: 'Warning: {{ .GroupLabels.alertname }} - Instances: {{ >>>> range .Alerts.Firing }}{{ .Labels.instance }}, {{ end }}' >>>> >>>> >>>> - name: 'email_resolved' >>>> email_configs: >>>> - to: 'sysw...@xxx.com' >>>> send_resolved: true >>>> headers: >>>> subject: 'Resolved: {{ .GroupLabels.alertname }} - Instances: {{ >>>> range .Alerts.Resolved }}{{ .Labels.instance }}, {{ end }}' >>>> >>>> Kolja Krückmann schrieb am Freitag, 16. Juni 2023 um 11:21:59 UTC+2: >>>> >>>>> I changed the config to this: >>>>> >>>>> - name: 'email_resolved' >>>>> email_configs: >>>>> - to: 'sysw...@xxx.com' >>>>> send_resolved: true >>>>> headers: >>>>> subject: 'Resolved: {{ .GroupLabels.alertname }} - Instances: {{ >>>>> range .Alerts.Resolved }}{{ .Labels.instance }}, {{ end }}' >>>>> >>>>> But this is sending absolutly nothing. What did I do wrong? >>>>> >>>>> Brian Candler schrieb am Freitag, 9. Juni 2023 um 15:57:04 UTC+2: >>>>> >>>>>> Maybe you want .Alerts.Firing and .Alerts.Resolved. A single alert >>>>>> group is likely to contain a mixture of firing alerts and resolved >>>>>> alerts. >>>>>> >>>>>> Also, note that your "send_resolved: true" receiver doesn't only send >>>>>> resolved messages: it will send on firing *and* on resolved. Hence the >>>>>> subject "Resolved: ..." is likely to be very misleading. >>>>>> >>>>>> On Friday, 9 June 2023 at 13:42:13 UTC+1 Kolja Krückmann wrote: >>>>>> >>>>>>> Got it working like this: >>>>>>> >>>>>>> receivers: >>>>>>> - name: 'email' >>>>>>> email_configs: >>>>>>> - to: 'sysw...@xxx.com' >>>>>>> send_resolved: false >>>>>>> headers: >>>>>>> subject: 'Critical: {{ .GroupLabels.alertname }} - Instances: >>>>>>> {{ range .Alerts }}{{ .Labels.instance }}, {{ end }}' >>>>>>> >>>>>>> - name: 'email_warning' >>>>>>> email_configs: >>>>>>> - to: 'sysw...@xxx.com' >>>>>>> send_resolved: false >>>>>>> headers: >>>>>>> subject: 'Warning: {{ .GroupLabels.alertname }} - Instances: >>>>>>> {{ range .Alerts }}{{ .Labels.instance }}, {{ end }}' >>>>>>> >>>>>>> - name: 'email_resolved' >>>>>>> email_configs: >>>>>>> - to: 'sysw...@xxx.com' >>>>>>> send_resolved: true >>>>>>> headers: >>>>>>> subject: 'Resolved: {{ .GroupLabels.alertname }} - Instances: >>>>>>> {{ range .Alerts }}{{ .Labels.instance }}, {{ end }}' >>>>>>> >>>>>>> Kolja Krückmann schrieb am Donnerstag, 8. Juni 2023 um 15:03:14 >>>>>>> UTC+2: >>>>>>> >>>>>>>> [EDIT] >>>>>>>> Also I want to split the firing and resolved alerts. >>>>>>>> Kolja Krückmann schrieb am Donnerstag, 8. Juni 2023 um 14:48:40 >>>>>>>> UTC+2: >>>>>>>> >>>>>>>>> Hi all, I tried it now for two days straight and I just don't get >>>>>>>>> it working. >>>>>>>>> >>>>>>>>> First of all where do I actually find the expressions I can use to >>>>>>>>> define the subject under the header? I am still missing a template I >>>>>>>>> can >>>>>>>>> use? I don't really have one in my Prom/Alert-files. >>>>>>>>> Second of all my current receivers are configured like so: >>>>>>>>> >>>>>>>>> receivers: >>>>>>>>> - name: 'email' >>>>>>>>> email_configs: >>>>>>>>> - to: 'x...@xxx.com' >>>>>>>>> send_resolved: true >>>>>>>>> headers: >>>>>>>>> Subject: "{{ .Alerts.alertname }} - {{ range .Alerts }} {{ >>>>>>>>> .Alerts.instance }} {{ end }}" >>>>>>>>> >>>>>>>>> Like Brian said, I want to iterate in the second half of that >>>>>>>>> "statement" (I don't actually know how these {{ .xyz }} are called) >>>>>>>>> through >>>>>>>>> all instances and the first one should just say which alert is now >>>>>>>>> firing. >>>>>>>>> I also tried it as followed: >>>>>>>>> >>>>>>>>> Subject: "{{ .GroupLabels.alertname }} - {{ range .Alerts }} {{ >>>>>>>>> .Alerts.instance }} {{ end }}" >>>>>>>>> and >>>>>>>>> Subject: "{{ .Labels.alertname }} - {{ range .Alerts }} {{ >>>>>>>>> .Alerts.instance }} {{ end }}" >>>>>>>>> >>>>>>>>> but nothing is really working. >>>>>>>>> >>>>>>>>> All I really want is the Subject like so e.x.: >>>>>>>>> InstanceDown - SVR-DS01 SVR-DC11 >>>>>>>>> ^^^^^^^^ ^^^^^^^^ ^^^^^^^^ >>>>>>>>> Alertname - Instance1 Instance2 (if it is also possible >>>>>>>>> to seperate the instances via comma would be perfect) >>>>>>>>> Brian Candler schrieb am Dienstag, 6. Juni 2023 um 13:16:53 UTC+2: >>>>>>>>> >>>>>>>>>> As its name implies, "CommonLabels" contains only those labels >>>>>>>>>> which are common to all alerts in the group. If there are multiple >>>>>>>>>> instances, then the "instance" label is not common to all alerts, so >>>>>>>>>> won't >>>>>>>>>> be in this object. >>>>>>>>>> >>>>>>>>>> You want to iterate over "Alerts" (or "Alerts.Firing") and look >>>>>>>>>> at the Labels.instance within each alert. You will find an example >>>>>>>>>> showing >>>>>>>>>> how to do that here: >>>>>>>>>> >>>>>>>>>> https://prometheus.io/docs/alerting/latest/notification_examples/#ranging-over-all-received-alerts >>>>>>>>>> >>>>>>>>>> Additional references: >>>>>>>>>> https://prometheus.io/docs/alerting/latest/notifications/#data >>>>>>>>>> >>>>>>>>>> https://prometheus.io/docs/prometheus/latest/configuration/template_reference/ >>>>>>>>>> >>>>>>>>>> On Tuesday, 6 June 2023 at 12:03:03 UTC+1 Kolja Krückmann wrote: >>>>>>>>>> >>>>>>>>>>> Currently my alertmanager.yml looks like this: >>>>>>>>>>> >>>>>>>>>>> receivers: >>>>>>>>>>> - name: 'email' >>>>>>>>>>> email_configs: >>>>>>>>>>> - to: 'x...@company.com' >>>>>>>>>>> send_resolved: true >>>>>>>>>>> headers: >>>>>>>>>>> Subject: '{{ .CommonLabels.alertname }} - {{ >>>>>>>>>>> .CommonLabels.instance }}' >>>>>>>>>>> >>>>>>>>>>> Unfortunately the subject in the mail now only displays both >>>>>>>>>>> labels if only there is an alert for a single instance (like High >>>>>>>>>>> CPU Usage >>>>>>>>>>> on core 0,1,2,3 on one instance (see screenshot attatched)) As soon >>>>>>>>>>> as the >>>>>>>>>>> alert is for more then just one instance the instance lable is >>>>>>>>>>> completly >>>>>>>>>>> missing... Why is it missing? And how can I change it to display >>>>>>>>>>> all >>>>>>>>>>> instances affected by that alert. >>>>>>>>>>> >>>>>>>>>>> Kind regards. >>>>>>>>>>> Kolja >>>>>>>>>>> >>>>>>>>>>> [image: mail.png] >>>>>>>>>>> >>>>>>>>>>> Kolja Krückmann schrieb am Montag, 5. Juni 2023 um 09:21:22 >>>>>>>>>>> UTC+2: >>>>>>>>>>> >>>>>>>>>>> Hi y'all >>>>>>>>>>> I'm trying to change the way the mail from my alertmanger looks >>>>>>>>>>> and behaves. >>>>>>>>>>> My goal is to have the subject list the alertname and the >>>>>>>>>>> targets detected by that alert. (I need it like that for future >>>>>>>>>>> todo's) >>>>>>>>>>> My first problem is, i dont have any templates in my files. Do I >>>>>>>>>>> need to clone one from git? How does the prom know, I have a >>>>>>>>>>> template? >>>>>>>>>>> The second one would be to change the subject in order for my >>>>>>>>>>> requirements to fit. >>>>>>>>>>> >>>>>>>>>>> Kind regards, >>>>>>>>>>> Kolja >>>>>>>>>>> >>>>>>>>>>> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/8e36f83b-03bb-4295-be42-9884391846dfn%40googlegroups.com.
