Hi all,
Currently we have different environments where Prometheus and Alertmanager
are used. These environments are also protected by certain network
settings, so we need to use PushProx. To collect data from all
environments, we use a "global" Prometheus server that federates with the
environments Prometheus server.
Alarms are emailed directly from the environments. This means that the
alarm rules are also configured per environment. Now we would like to send
the alerts from all environments to one of our Slack channels as well.
Since the environments are protected by certain network settings, it is not
possible to send the alarms from the environments directly to our Slack
channel. One of our ideas is to configure an alert in the "global" alert
manager that looks like this:
- alert: AlertName
expr: ALERTS{alertname!="AlertName",alertstate="firing",severity="critical"}
labels:
severity: critical
annotations:
alertname: '{{ $labels.alertname }}'
summary: Alert {{ $labels.alertname }} fired.
This seems to work, but unfortunately this is not the best solution as
alarms often switch between firing/resolved state and also the summary is
not available here.
Does anyone have any ideas on how we can improve this?
Thanks!
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/CAFERo3qk1zjB3krqNePeomzgvBXdVzTyTVd%3DkD9SbCJuM3WWhA%40mail.gmail.com.
