On Thu, Jun 4, 2020, at 11:09, anya...@gmail.com wrote: > Hi, colleagues! > I setup and configured Alertmanager for getting email notifications, but all > of my alerts about different environments are gathered in one email and > sorted by alphabetical order by job_name. In test environment there are > approx. 250 alerts in one email. It doesn't work for urgents reaction. > Prometheus rules have 3 rules per 1 env. Now I'm testing 7 envs.
It's not useful to have 250 alerts firing at once in testing or in a real outage. You need to find something less noisy to alert on. As an example, if you have a fleet of 100 servers, you might want to send a single alert if more than 10 of them are down; then use a dashboard to figure out which ones (and why). As others have mentioned you'll also want to group your alerts; at the very least separate them by environment. -- Harald -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/22c45cf4-7000-4405-9532-c2bfbceaf601%40www.fastmail.com.
