Re: [prometheus-users] deactive alert after hook
Events are not metrics.
For the behaviour you want, you should be looking at an event-based system
like Loki.
On Thursday, 28 July 2022 at 12:14:39 UTC+1 ariap...@gmail.com wrote:
> Hello Stuart
> I'm sorry I couldn't ask my question properly
> Actually, I use prometheus/alertmanager as an event pipeline to alert
> every event that occurs.
>
> For example, I show two of the rules that are exported from the log
> exporter service:
> - name: plate
> rules:
> - alert: "plate"
> expr: 'plate_log {plate_number="123456877Lay"}'
> for: 1s
> annotations:
> title: "plate detection {{ $labels.model_camera_id }}"
> description: "plate detection with confidence : {{
> $labels.confidence }} "
> labels:
> severity: "critical"
> type: "plate"
>
> - name: human
> rules:
> - alert: "human"
> expr: 'number_of_Human > 15'
> for: 1s
> annotations:
> title: "human detection {{ $labels.model_camera_id }}"
> description: "human detection with confidence : {{
> $labels.confidence }} "
> labels:
> severity: "critical"
> type: "human"
>
>
> Also, the alertmanager configuration is as follows:
>
> global:
>
> route:
> receiver: webhook
> group_by: ["alertname"]
> group_wait: 1s
> group_interval: 1s
> # repeat_interval: 6d
> routes:
> - receiver: webhook
> continue: true
> Receivers:
> - name: webhook
> webhook_configs:
> - send_resolved: false
> http_config: {}
> url: "http://192.168.10.20:7000/visual;
> max_alerts: 0
> - url: "http://192.168.10.20:9200/alerts/_doc;
> send_resolved: false
>
> My problem is exactly that if 5 alerts are hooked at different times, for
> the sixth log, all the previous 5 logs are also hooked.
> I felt that because the logs are still in firing mode after the hook, they
> are sent again to Front and Elastic with new logs.
>
> Stuart Clark در تاریخ یکشنبه ۲۴ ژوئیهٔ ۲۰۲۲ ساعت ۱۹:۵۹:۳۵ (UTC+4:30) نوشت:
>
>> On 24/07/2022 11:10, Milad Devops wrote:
>> > hi all
>> > I use Prometheus to create alert rules and hook alerts using
>> alertmanager.
>> > My scenario is as follows:
>> > - The log publishing service sends logs to Prometheus Exporter
>> > - Prometheus takes the logs every second and matches them with our
>> rules
>> > - If the log applies to our rules, the alertmanager sends an alert to
>> > the frontend application. It also saves the alert in the elastic
>> >
>> > My problem is that when sending each alert, all the previous alerts
>> > are also stored in Elastic in the form of a single log and sent to my
>> > front service as a notification (web hook).
>> >
>> > Is there a way I can change the alert status to resolved after the
>> > hook so that it won't be sent again on subsequent hooks?
>> > Or delete the previous logs completely after the hook from Prometheus
>> > Or any other suggested way you have
>> > Thank you in advance
>>
>> I'm not sure I really understand what you are asking due to your
>> mentioning of logs.
>>
>> Are you saying that you are using an exporter (for example mtail) which
>> is consuming logs and then generating metrics?
>>
>> When you create an alerting rule in Prometheus it performs the PromQL
>> query given, and if there are any results an alert is fired. Once the
>> PromQL query stops returning results (or has a different set of time
>> series being returned) the alert is resolved.
>>
>> So for example if you had a simple query that said "alert if the number
>> of error logs [stored in a counter metric] increases by 5 or more in the
>> last 5 minutes" as soon as the metric returned an increase of at least 5
>> over the last 5 minutes it would fire. It would then continue to fire
>> until that is no longer true - so if the counter kept recording error
>> log lines such that the increase was still over 5 per 5 minutes it would
>> keep firing. It would only resolve once there were no more than 5 new
>> long lines recorded over the past 5 minutes.
>>
>> Alertmanager just routes alerts that are generated within Prometheus to
>> other notification/processing systems, such as email or webhooks. It
>> would normally fire the webhook once the alert starts firing, and then
>> periodically (if it keeps firing, at a configurable interval) and then
>> finally (optionally) once it resolves. This is a one-way process -
>> nothing about the notification has any impact on the alert firing or
>> not. Only the PromQL query controls the alert.
>>
>> I'm not sure if that helps.
>>
>> --
>> Stuart Clark
>>
>>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
Re: [prometheus-users] deactive alert after hook
Hello Stuart
I'm sorry I couldn't ask my question properly
Actually, I use prometheus/alertmanager as an event pipeline to alert every
event that occurs.
For example, I show two of the rules that are exported from the log
exporter service:
- name: plate
rules:
- alert: "plate"
expr: 'plate_log {plate_number="123456877Lay"}'
for: 1s
annotations:
title: "plate detection {{ $labels.model_camera_id }}"
description: "plate detection with confidence : {{
$labels.confidence }} "
labels:
severity: "critical"
type: "plate"
- name: human
rules:
- alert: "human"
expr: 'number_of_Human > 15'
for: 1s
annotations:
title: "human detection {{ $labels.model_camera_id }}"
description: "human detection with confidence : {{
$labels.confidence }} "
labels:
severity: "critical"
type: "human"
Also, the alertmanager configuration is as follows:
global:
route:
receiver: webhook
group_by: ["alertname"]
group_wait: 1s
group_interval: 1s
# repeat_interval: 6d
routes:
- receiver: webhook
continue: true
Receivers:
- name: webhook
webhook_configs:
- send_resolved: false
http_config: {}
url: "http://192.168.10.20:7000/visual;
max_alerts: 0
- url: "http://192.168.10.20:9200/alerts/_doc;
send_resolved: false
My problem is exactly that if 5 alerts are hooked at different times, for
the sixth log, all the previous 5 logs are also hooked.
I felt that because the logs are still in firing mode after the hook, they
are sent again to Front and Elastic with new logs.
Stuart Clark در تاریخ یکشنبه ۲۴ ژوئیهٔ ۲۰۲۲ ساعت ۱۹:۵۹:۳۵ (UTC+4:30) نوشت:
> On 24/07/2022 11:10, Milad Devops wrote:
> > hi all
> > I use Prometheus to create alert rules and hook alerts using
> alertmanager.
> > My scenario is as follows:
> > - The log publishing service sends logs to Prometheus Exporter
> > - Prometheus takes the logs every second and matches them with our rules
> > - If the log applies to our rules, the alertmanager sends an alert to
> > the frontend application. It also saves the alert in the elastic
> >
> > My problem is that when sending each alert, all the previous alerts
> > are also stored in Elastic in the form of a single log and sent to my
> > front service as a notification (web hook).
> >
> > Is there a way I can change the alert status to resolved after the
> > hook so that it won't be sent again on subsequent hooks?
> > Or delete the previous logs completely after the hook from Prometheus
> > Or any other suggested way you have
> > Thank you in advance
>
> I'm not sure I really understand what you are asking due to your
> mentioning of logs.
>
> Are you saying that you are using an exporter (for example mtail) which
> is consuming logs and then generating metrics?
>
> When you create an alerting rule in Prometheus it performs the PromQL
> query given, and if there are any results an alert is fired. Once the
> PromQL query stops returning results (or has a different set of time
> series being returned) the alert is resolved.
>
> So for example if you had a simple query that said "alert if the number
> of error logs [stored in a counter metric] increases by 5 or more in the
> last 5 minutes" as soon as the metric returned an increase of at least 5
> over the last 5 minutes it would fire. It would then continue to fire
> until that is no longer true - so if the counter kept recording error
> log lines such that the increase was still over 5 per 5 minutes it would
> keep firing. It would only resolve once there were no more than 5 new
> long lines recorded over the past 5 minutes.
>
> Alertmanager just routes alerts that are generated within Prometheus to
> other notification/processing systems, such as email or webhooks. It
> would normally fire the webhook once the alert starts firing, and then
> periodically (if it keeps firing, at a configurable interval) and then
> finally (optionally) once it resolves. This is a one-way process -
> nothing about the notification has any impact on the alert firing or
> not. Only the PromQL query controls the alert.
>
> I'm not sure if that helps.
>
> --
> Stuart Clark
>
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/d68448de-7a30-469c-bb7c-38b2720c213fn%40googlegroups.com.
Re: [prometheus-users] deactive alert after hook
On 24/07/2022 11:10, Milad Devops wrote: hi all I use Prometheus to create alert rules and hook alerts using alertmanager. My scenario is as follows: - The log publishing service sends logs to Prometheus Exporter - Prometheus takes the logs every second and matches them with our rules - If the log applies to our rules, the alertmanager sends an alert to the frontend application. It also saves the alert in the elastic My problem is that when sending each alert, all the previous alerts are also stored in Elastic in the form of a single log and sent to my front service as a notification (web hook). Is there a way I can change the alert status to resolved after the hook so that it won't be sent again on subsequent hooks? Or delete the previous logs completely after the hook from Prometheus Or any other suggested way you have Thank you in advance I'm not sure I really understand what you are asking due to your mentioning of logs. Are you saying that you are using an exporter (for example mtail) which is consuming logs and then generating metrics? When you create an alerting rule in Prometheus it performs the PromQL query given, and if there are any results an alert is fired. Once the PromQL query stops returning results (or has a different set of time series being returned) the alert is resolved. So for example if you had a simple query that said "alert if the number of error logs [stored in a counter metric] increases by 5 or more in the last 5 minutes" as soon as the metric returned an increase of at least 5 over the last 5 minutes it would fire. It would then continue to fire until that is no longer true - so if the counter kept recording error log lines such that the increase was still over 5 per 5 minutes it would keep firing. It would only resolve once there were no more than 5 new long lines recorded over the past 5 minutes. Alertmanager just routes alerts that are generated within Prometheus to other notification/processing systems, such as email or webhooks. It would normally fire the webhook once the alert starts firing, and then periodically (if it keeps firing, at a configurable interval) and then finally (optionally) once it resolves. This is a one-way process - nothing about the notification has any impact on the alert firing or not. Only the PromQL query controls the alert. I'm not sure if that helps. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/040d084b-4046-6bbf-3691-5c9bedd51343%40Jahingo.com.
[prometheus-users] deactive alert after hook
hi all I use Prometheus to create alert rules and hook alerts using alertmanager. My scenario is as follows: - The log publishing service sends logs to Prometheus Exporter - Prometheus takes the logs every second and matches them with our rules - If the log applies to our rules, the alertmanager sends an alert to the frontend application. It also saves the alert in the elastic My problem is that when sending each alert, all the previous alerts are also stored in Elastic in the form of a single log and sent to my front service as a notification (web hook). Is there a way I can change the alert status to resolved after the hook so that it won't be sent again on subsequent hooks? Or delete the previous logs completely after the hook from Prometheus Or any other suggested way you have Thank you in advance -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/c7f6f592-8e6c-41d7-bd14-8e9baf55b682n%40googlegroups.com.
