I don't know, but you could look at the source code of projects like https://github.com/prymitive/karma and alerta.io to see how they interface with the API.
Also: https://stackoverflow.com/questions/75283114/prometheus-alertmanager-api-to-manage-silences On Thursday, 16 January 2025 at 11:50:53 UTC Saar Zur wrote: > Thanks for your response. > > Another question - there is a way to delete expired silences via the api ? > > ב-יום שני, 13 בינואר 2025 בשעה 18:16:36 UTC+2, Brian Candler כתב/ה: > >> > from a look in the AlertManager UI no silence was created, and i got >> resolved notification after 5 minutes since the fired notification. >> ... >> > I wonder why the silence wasn't able to create? (not the first time it >> happens) >> > Maybe it's some kind of a race condition? we can't silence alerts which >> are not in fired state right? >> >> That's not true - you can certainly create silences which don't match any >> active alerts. This allows you, for example, to create silences before >> maintenance starts, to suppress the alerts you expect. >> >> If the silences aren't being created (i.e. not visible in the GUI), then >> you need to look deeper into the code which creates them, and perhaps >> tcpdump the API to alertmanager to see if you're passing valid parameters. >> >> On Monday, 13 January 2025 at 15:22:54 UTC Saar Zur wrote: >> >>> Hi, >>> >>> i am using the amtool client in a Job inside my cluster. >>> >>> An alert was fired and we got notification in our slack channel, i used >>> the cli (in code that runs inside docker image from the Job) to create a >>> silence according to `alertname` matcher and there was no failure. >>> >>> from a look in the AlertManager UI no silence was created, and i got >>> resolved notification after 5 minutes since the fired notification. >>> >>> After ~10 minutes the alert was fired and resolved again (5 minutes >>> difference). >>> >>> I wonder why the silence wasn't able to create? (not the first time it >>> happens) >>> Maybe it's some kind of a race condition? we can't silence alerts which >>> are not in fired state right? (although the alert was in fired state while >>> i tried to create the silence) >>> >>> The Alert rule: >>> name: Orchestrator GRPC Failures for ExternalProcessor Service >>> <http://localhost:9090/graph?g0.expr=ALERTS%7Balertname%3D%22Orchestrator%20GRPC%20Failures%20for%20ExternalProcessor%20Service%22%7D&g0.tab=1&g0.display_mode=lines&g0.show_exemplars=0.g0.range_input=1h.> >>> expr: >>> sum(increase(grpc_server_handled_total{grpc_code!~"OK|Canceled",grpc_service="envoy.service.ext_proc.v3.ExternalProcessor"}[5m])) >>> >>> > 0 >>> <http://localhost:9090/graph?g0.expr=sum(increase(grpc_server_handled_total%7Bgrpc_code!~%22OK%7CCanceled%22%2Cgrpc_service%3D%22envoy.service.ext_proc.v3.ExternalProcessor%22%7D%5B5m%5D))%20%3E%200&g0.tab=1&g0.display_mode=lines&g0.show_exemplars=0.g0.range_input=1h.> >>> for: 5m >>> labels: >>> severity: WARNING >>> annotations: >>> dashboard_url: p-R7Hw1Iz >>> runbook_url: extension-orchestrator-dashboard >>> summary: Failed gRPC calls detected in the Envoy External Processor >>> within the last 5 minutes. <!subteam^S06E0CPPC5S> >>> >>> The code for creating the silence: >>> func postSilence(amCli amclient.Client, matchers []*models.Matcher) >>> error { >>> startsAt := strfmt.DateTime(silenceStart) >>> endsAt := strfmt.DateTime(silenceStart.Add(silenceDuration)) >>> createdBy := creatorType >>> comment := silenceComment >>> silenceParams := silence.NewPostSilencesParams().WithSilence( >>> &models.PostableSilence{ >>> Silence: models.Silence{ >>> Matchers: matchers, >>> StartsAt: &startsAt, >>> EndsAt: &endsAt, >>> CreatedBy: &createdBy, >>> Comment: &comment, >>> }, >>> }, >>> ) >>> >>> err := amCli.PostSilence(silenceParams) >>> if err != nil { >>> return fmt.Errorf("failed on post silence: %w", err) >>> } >>> log.Print("Silence posted successfully") >>> >>> return nil >>> } >>> >>> Thank in advance, >>> Saar Zur SAP Labs >>> >> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion visit https://groups.google.com/d/msgid/prometheus-users/80fa5075-356d-435c-8185-3eadbd84cf8bn%40googlegroups.com.
