Dear Support Team,
I am currently experiencing some discrepancies and unexpected behavior with
the configuration of Prometheus AlertManager. Despite configuring alerts to
fire every 4 hours, I am encountering issues with the actual number of
alerts sent and received by AlertManager. There seems to be a significant
disparity between the expected and observed counts.
Furthermore, even when no conditions are satisfied for triggering alerts,
Prometheus continues to send alerts every 4 to 5 hours, and AlertManager
subsequently forwards the payload to the configured webhook. I have noticed
this behavior persisting even after a prolonged period of no data changes –
in one instance, alerts were still being received 20 hours after data had
ceased to change.
Below are the relevant configuration files: prometheus.yml: Scrape
Interval: 2m
*rules.yml:* groups:
- name: recording-rule
interval: 1h
rules:
- record: myRecord
expr: [expression calculating ratio by dividing two metrics,
greater than a specified value]
- name: alerting-rule
interval: 4h
rules:
- alert: myAlert
expr: max_over_time(myRecord[4h])
labels:
severity: warning
annotations:
summary: "Summary"
* alertmanager.yml:* group_by: ['alertname']
group_wait: 30s
group_interval: 1h
repeat_interval: 4h
I would appreciate your assistance in resolving the following issues:
1. Identifying and rectifying the discrepancies in the number of alerts
between Prometheus and AlertManager.
2. Investigating and addressing the persistence of alerts being fired by
Prometheus and forwarded by AlertManager, even in the absence of new data
or changes.
3. Resolving the issue of receiving alerts at regular intervals, despite
a prolonged period of no new data.
*It's worth noting that a substantial amount of data is being processed for
alerts, more than 25,000 unique metrics.*
Your prompt attention to these matters would be greatly appreciated.
Thank you for your assistance.
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/32cc9bd5-868b-4631-8c8b-018023e29c34n%40googlegroups.com.
