In theory Alertmanager keeps resolved alerts around for a generally
short interval, between when Prometheus tells Alertmanager that the
alert is resolved and when the alert group's group_interval allows
Alertmanager to successfully send a notification about it, but I don't
know if Alertmanager shows such alerts in the API. In any case, even if
they briefly show up in Alertmanager's API, as Brian Candler says it's
not a useful way to see what alerts Alertmanager sent 'resolved'
notifications about because they don't stick around for the long term.
(These lingering resolved alerts sometimes can have interesting
interactions with inhibitions and I believe it's one cause of getting a
'alert resolved' notification when you never saw the corresponding
'alert firing'.)
In general if you want to know the history of past Alertmanager
activities, I think you should set up a webhook that gets included in
every notification group and either logs the data or puts it in a
database (this is what we eventually wound up doing). If you just want
what Prometheus alerts were firing, you can get that from Prometheus
with suitable queries, although the queries get tricky if you want to
know things like how many times an alert triggered and was resolved.
- cks
> AFAIK, alertmanager simply notices that the alert has disappeared, and
> sends a notification when that happens. That is, it retains some internal
> state about previously firing alerts (which it needs to for other reasons
> too, e.g. repeat_interval) and compares current to previous. But once it's
> gone, it's gone; it doesn't hang around in a "resolved" state.
>
> On Wednesday, 1 January 2025 at 22:15:40 UTC Basil Lalli wrote:
>
> > The ALERTS table doesn't show alerts in a resolved state. They simply
> > disappear from the list. I realize this is sorta how it works (alerts just
> > disappear)... but alertmanager sends resolved alerts so clearly it's doing
> > the job of noticing an alert is gone, and I was hoping it exposes this in
> > the API somewhere.
> >
> > On Wednesday, January 1, 2025 at 11:13:10 AM UTC-8 Bryan Boreham wrote:
> >
> >> You could query the ALERTS time-series.
> >>
> >> On Wednesday, 1 January 2025 at 19:11:18 UTC [email protected] wrote:
> >>
> >>> Is there any API call that can provide information on resolved alerts?
> >>> I assumed `api/v2/status` would display resolved alerts for some time
> >>> period afterwards but that doesn't seem to be the case.
> >>
> >>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To view this discussion visit
> https://groups.google.com/d/msgid/prometheus-users/2f4d06b6-7028-4781-86ba-4406ca4d7d97n%40googlegroups.com.
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion visit
https://groups.google.com/d/msgid/prometheus-users/2868445.1735837021%40apps0.cs.toronto.edu.
