On 24/07/2022 11:10, Milad Devops wrote:
hi all
I use Prometheus to create alert rules and hook alerts using alertmanager.
My scenario is as follows:
- The log publishing service sends logs to Prometheus Exporter
- Prometheus takes the logs every second and matches them with our rules
- If the log applies to our rules, the alertmanager sends an alert to
the frontend application. It also saves the alert in the elastic
My problem is that when sending each alert, all the previous alerts
are also stored in Elastic in the form of a single log and sent to my
front service as a notification (web hook).
Is there a way I can change the alert status to resolved after the
hook so that it won't be sent again on subsequent hooks?
Or delete the previous logs completely after the hook from Prometheus
Or any other suggested way you have
Thank you in advance
I'm not sure I really understand what you are asking due to your
mentioning of logs.
Are you saying that you are using an exporter (for example mtail) which
is consuming logs and then generating metrics?
When you create an alerting rule in Prometheus it performs the PromQL
query given, and if there are any results an alert is fired. Once the
PromQL query stops returning results (or has a different set of time
series being returned) the alert is resolved.
So for example if you had a simple query that said "alert if the number
of error logs [stored in a counter metric] increases by 5 or more in the
last 5 minutes" as soon as the metric returned an increase of at least 5
over the last 5 minutes it would fire. It would then continue to fire
until that is no longer true - so if the counter kept recording error
log lines such that the increase was still over 5 per 5 minutes it would
keep firing. It would only resolve once there were no more than 5 new
long lines recorded over the past 5 minutes.
Alertmanager just routes alerts that are generated within Prometheus to
other notification/processing systems, such as email or webhooks. It
would normally fire the webhook once the alert starts firing, and then
periodically (if it keeps firing, at a configurable interval) and then
finally (optionally) once it resolves. This is a one-way process -
nothing about the notification has any impact on the alert firing or
not. Only the PromQL query controls the alert.
I'm not sure if that helps.
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/040d084b-4046-6bbf-3691-5c9bedd51343%40Jahingo.com.
