What is your webhook receiver? Are any of the resolve messages getting through? Are the requests succeeding?
I think Alertmanager will retry failed webhooks, not sure for how long. This would keep them in the queue, leading to what you observe in Alertmanager. /MR On Thu, Jan 6, 2022, 07:14 shivakumar sajjan <shivusajjan...@gmail.com> wrote: > Hi, > > I have single instance cluster for AlertManager and I see below warning in > AlertManager > > > *container level=warn ts=2021-11-03T08:50:44.528Z caller=delegate.go:272 > component=cluster msg="dropping messages because too many are queued" > current=4125 limit=4096* > > *Alert Manager Version information:* > Branch: HEAD > BuildDate: 20190708-14:31:49 > BuildUser: root@868685ed3ed0 > GoVersion: go1.12.6 > Revision: 1ace0f76b7101cccc149d7298022df36039858ca > Version: 0.18.0 > > *AlertManager metrics* > # HELP alertmanager_cluster_members Number indicating current number of > members in cluster. > # TYPE alertmanager_cluster_members gauge alertmanager_cluster_members 1 > # HELP alertmanager_cluster_messages_pruned_total Total number of cluster > messages pruned. > # TYPE alertmanager_cluster_messages_pruned_total counter > alertmanager_cluster_messages_pruned_total 23020 > # HELP alertmanager_cluster_messages_queued Number of cluster messages > which are queued. > # TYPE alertmanager_cluster_messages_queued gauge > alertmanager_cluster_messages_queued 4125 > > I am new to alerting. Could you please answer for the below questions? > > > - > > Why are messages queueing up due to this alertmanager is not sending > alert resolve information to webhook instance? > - > > What is the solution for the above issue? > - > > How do we see those queued messages in AlertManager? > - > > Do we lose alerts when messages are dropped because of too many queued > ? > - > > Why are messages queued even though there is logic to prune messages > on regular interval i.e 15 minutes ? > - > > Do we lose alerts when AlertManager pruned messages on regular > interval? > > > Thanks, > > Shiva > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to prometheus-users+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/ea293a79-9f3f-42f1-b9a4-66ff7353cb16n%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/ea293a79-9f3f-42f1-b9a4-66ff7353cb16n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAMV%3D_gYiqhGs%2BLZiEBLRyrVaNBZ2pJG-Kby1MmTDbpiexzoXhQ%40mail.gmail.com.
