Re: [prometheus-users] Routes and receivers

2021-01-23 Thread Julius Volz
Great. Other than that, we see escalation as the responsibility of a system after Alertmanager in the chain. For example, PagerDuty (one of the built-in notifier mechanisms in AM) allows setting up both a schedule and an escalation policy for notifications. On Fri, Jan 22, 2021 at 8:51 PM steve wa

Re: [prometheus-users] Routes and receivers

2021-01-22 Thread steve waldron
I was looking for a way to escalate the alert after it was firing for a period of time. Looks like defining the alert twice with different for and label will do the trick. Thanks for you assistance Julius. On Friday, January 22, 2021 at 1:16:17 PM UTC-5 juliu...@promlabs.com wrote: > If your

Re: [prometheus-users] Routes and receivers

2021-01-22 Thread Julius Volz
If your goal is to send an alert notification only after it has been firing for some time, the usual way of doing that in Prometheus would be to use the "for" duration field in the alerting rule to define how long the alert has to be active before it transitions from "pending" into "firing" state (

Re: [prometheus-users] Routes and receivers

2021-01-21 Thread steve waldron
That makes sense which leads to a couple of other questions: 1. Can an alert rule annotation definition contain a promql query to define? 2. if 1. is true, can the annotation label be used in alertmanager config route matching? Thanks, Steve On Thursday, January 21, 2021 at 12:04:52 PM UTC-5 S

Re: [prometheus-users] Routes and receivers

2021-01-21 Thread Stuart Clark
On 21/01/2021 16:58, steve waldron wrote: Is it possible to add a label with a dynamic value? for example - the amount of time an alert has been firing?  if so, any examples of PROMQL queries to accomplish this? I'm trying to determine if routes can be created using a label defined by length

[prometheus-users] Routes and receivers

2021-01-21 Thread steve waldron
Is it possible to add a label with a dynamic value? for example - the amount of time an alert has been firing? if so, any examples of PROMQL queries to accomplish this? I'm trying to determine if routes can be created using a label defined by length of time an alert has been firing. Thanks ,