On 2020-12-14 17:05, Al wrote:
Hi
I realize alert conditions in a Promertheus ecosystem should be
triggered from a prometheus instance itself although there is the
"amtool alert add" command that can be used to manually trigger an
alert. Is this something which is commonly used in production
use-cases? I can see a benefit to using this command as I could still
allow users to trigger alerts in a standardized way, but without
having to have specific pre-defined alerting conditions. There may
also be situations where there is no metric collected but only an
alert to be triggered in the situation a specific event occurs.
From my understanding, when prometheus fires an alert, it will send
the payload to all instances of alert manager with in the cluster and
then they will handle which instance will actually route the alert to
the final destination (e.g.: Victorops, email, webook, etc). If this
is in fact correct, does this mean that amtool should also send the
alert to all alertmanager instances within the cluster?
I appreciate any clarification you can provide me with.
That command is only intended for testing. Alerts aren't a one-off API
call from Prometheus to Alertmanager. Instead Prometheus will repeatedly
call every single Alertmanager periodically until the alert is cleared.
If Alertmanager stops receiving these updates it will mark the alert as
resolved.
Alerts in the Prometheus world are triggered based on the evaluation of
alerting rules, which themselves are queries which interrogate metrics.
Therefore every alert would be based on some sort of source metric
(there are a few exceptions, such as having an alert which always fires
to check the alerting pipeline for example).
For one of the example use cases you gave you said an alert should be
triggered if an event happens. Prometheus itself isn't an event system,
but you can create metrics from events. So in that case you'd have a
metric that could be a counter of the number of events that have
happened. Then your alert would fire when that value increases (for
example).
Are you able to give some more information on what use cases you are
trying to handle?
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/3ebbb88423946a3dbca81f81b8d2b635%40Jahingo.com.
