On Fri, 3 Dec 2021 at 19:08, Matt Miller <msm9...@gmail.com> wrote: > Hi, > > In the documentation under the HA section, it mentions: > "It's important not to load balance traffic between Prometheus and its > Alertmanagers, but instead, point Prometheus to a list of all > Alertmanagers." > > I'm curious if this is strictly for high availability and network > partitioning concerns, or if there is a more functional reason that every > Alertmanager member needs to receive the alerts. >
Both are true. > > What prompted this question, is that in our three member HA alertmanager > cluster that we've been sending alerts to via a load balancer (from > multiple prometheus instances), we've observed that alerts stored on each > cluster member may have pretty drastically different endsAt times for a > single given alert (one to two minutes). We believe that this may be > contributing to random flapping alerts, that prometheus indicates has been > firing the entire duration. > This is what can happen if you don't follow the documentation. Alerts are not passed between alertmanagers, you need to send them to all of them. Brian > > Thanks. > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to prometheus-users+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/4b4e8a88-1243-4591-b2d7-43bd550738een%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/4b4e8a88-1243-4591-b2d7-43bd550738een%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- Brian Brazil www.robustperception.io -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAHJKeLrCcajp83XoZ4k6RZ34xXPO%2BYzYmsWEcLjuHYfDVSTxvw%40mail.gmail.com.
