On 23/11/2022 15:28, Nenad Rakonjac wrote:
Hello,
Does everyone have clue how much time need for prometheus metrics to
go from application to alertmenager? Can this time be bigger than one
minute? --
Metrics don't go to Alertmanager. Instead you create alerting rules
which query metrics to produce alerts.
How long something changing to an alert being fired depend on lots of
different factors:
How often you scrape the application (so Prometheus has the latest metrics)
The query you are using in your alerting rule (for example you might be
alerting based on an average rate over the last few minutes, so a sudden
spike wouldn't immediately trigger an alert)
If you have a "for" clause in the alert rule (which is generally
recommended so as not to send an alert for something that goes away very
quickly, such as a transient spike)
In general the "speed" of alerting isn't particularly critical. Instead
what is important is producing useful actionable alerts. Sending an
alert as soon as a resource goes above 90% isn't particularly useful if
a second later it drops to 10% - nothing bad happened and there is
nothing to be addressed. In general it would actually be more useful to
alert if that threshold is breached for over say 5 minutes, or even more
usefully when an SLO is failed or is projected to fail within the next
30 minutes.
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/ef6ed871-c01c-56c1-b7f3-68074fffb745%40Jahingo.com.
