Re: [prometheus-users] Null value in alerts
Hi Team
On Sunday, 11 December 2022 at 23:20:24 UTC+5:30 matt...@prometheus.io
wrote:
> When you say "the value is missing", what condition exactly do you want to
> alert on?
>
> To detect that there is *no* metric matching your selector, you can use
> the absent(…) function. It returns 1 when … is nothing.
>
> It gets more complicated and difficult if you want to detect that a single
> series has disappeared. In this case, you need to very specific in telling
> Prometheus which series *should* exist. Common ways to do this are
>
> - listing them all out with separate absent(x) clauses and specific
> positive matchers
> - comparing to a previous time (x offset 15m unless x)
> - use some other metric that lets you determine what should be there
> - generate recording rules to create such a metric
>
> The fundamental challenge here is to distinguish between "this went
> missing" and "this went away because of expected changes".
>
> In general, I prefer splitting "metric indicates there is a problem " and
> "metric is missing" into two different alerts with separate names and
> descriptions. To the one investigating, the difference matters.
> Additionally using absent() often results in different label sets because
> it cannot know labels for a time series that is absent. This causes trouble
> with templating that you sidestep by using separate alert definitions to
> begin with.
>
> /MR
>
>
> On Fri, 9 Dec 2022, 08:31 Sebastian Glock, wrote:
>
>> Hi,
>>
>> I'm having trouble setting up an alert that will send a notification when
>> a value is different from 0 and the value is missing (i.e. null).
>>
>> expression:
>> windows_mscluster_resourcegroup_state {name!~"Available Storage"} != 0 or
>> on() vector(0)
>>
>> The alert goes off non-stop. How can I set the metric to send an alert
>> when the value is different from 0 and is null?
>>
>> I tried with sum() but not working anyway:
>> sum(windows_mscluster_resourcegroup_state {name!~"Available Storage"} !=
>> 0) or on() vector(0)
>>
>> Thanks for replies!
>>
>> --
>>
> You received this message because you are subscribed to the Google Groups
>> "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to prometheus-use...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-users/9fbc7d5d-c7ce-4b93-b653-733cac798956n%40googlegroups.com
>>
>>
>> .
>>
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/06c25a97-ca2f-4fd7-bd1a-82902864bd84n%40googlegroups.com.
Re: [prometheus-users] Null value in alerts
When you say "the value is missing", what condition exactly do you want to
alert on?
To detect that there is *no* metric matching your selector, you can use the
absent(…) function. It returns 1 when … is nothing.
It gets more complicated and difficult if you want to detect that a single
series has disappeared. In this case, you need to very specific in telling
Prometheus which series *should* exist. Common ways to do this are
- listing them all out with separate absent(x) clauses and specific
positive matchers
- comparing to a previous time (x offset 15m unless x)
- use some other metric that lets you determine what should be there
- generate recording rules to create such a metric
The fundamental challenge here is to distinguish between "this went
missing" and "this went away because of expected changes".
In general, I prefer splitting "metric indicates there is a problem " and
"metric is missing" into two different alerts with separate names and
descriptions. To the one investigating, the difference matters.
Additionally using absent() often results in different label sets because
it cannot know labels for a time series that is absent. This causes trouble
with templating that you sidestep by using separate alert definitions to
begin with.
/MR
On Fri, 9 Dec 2022, 08:31 Sebastian Glock, wrote:
> Hi,
>
> I'm having trouble setting up an alert that will send a notification when
> a value is different from 0 and the value is missing (i.e. null).
>
> expression:
> windows_mscluster_resourcegroup_state {name!~"Available Storage"} != 0 or
> on() vector(0)
>
> The alert goes off non-stop. How can I set the metric to send an alert
> when the value is different from 0 and is null?
>
> I tried with sum() but not working anyway:
> sum(windows_mscluster_resourcegroup_state {name!~"Available Storage"} !=
> 0) or on() vector(0)
>
> Thanks for replies!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/9fbc7d5d-c7ce-4b93-b653-733cac798956n%40googlegroups.com
>
> .
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/CAMV%3D_gYFKXKTubtfPnF9irhj-_12G5GwtCS28FYO_woBVxprVw%40mail.gmail.com.
Re: [prometheus-users] Null value in alerts
On 09/12/2022 08:49, sebagloc...@gmail.com wrote:
Thanks for advice,
So in this case I just need to use absent like this In alert?:
- alert: Resource group in cluster is down
expr: absent(windows_mscluster_resourcegroup_state
{name!~"Available Storage"}) == 1
You aren't listing a metric here as you are using !~. You need to ensure
you are only using = in any labels.
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/02603cd3-ccdd-9a73-cd3e-1aa9ed55e093%40Jahingo.com.
RE: [prometheus-users] Null value in alerts
Thanks for advice,
So in this case I just need to use absent like this In alert?:
- alert: Resource group in cluster is down
expr: absent(windows_mscluster_resourcegroup_state {name!~"Available
Storage"}) == 1
for: 10s
labels:
severity: "[Cluster]"
annotations:
summary: "Resource group in cluster is down!"
description: "{{ humanize $value }}"
This one will send message, when metric is missing?
From: Matthias Rampke
Sent: Friday, December 9, 2022 8:57 AM
To: Sebastian Glock
Cc: Prometheus Users
Subject: Re: [prometheus-users] Null value in alerts
When you say "the value is missing", what condition exactly do you want to
alert on?
To detect that there is *no* metric matching your selector, you can use the
absent(…) function. It returns 1 when … is nothing.
It gets more complicated and difficult if you want to detect that a single
series has disappeared. In this case, you need to very specific in telling
Prometheus which series *should* exist. Common ways to do this are
- listing them all out with separate absent(x) clauses and specific positive
matchers
- comparing to a previous time (x offset 15m unless x)
- use some other metric that lets you determine what should be there
- generate recording rules to create such a metric
The fundamental challenge here is to distinguish between "this went missing"
and "this went away because of expected changes".
In general, I prefer splitting "metric indicates there is a problem " and
"metric is missing" into two different alerts with separate names and
descriptions. To the one investigating, the difference matters. Additionally
using absent() often results in different label sets because it cannot know
labels for a time series that is absent. This causes trouble with templating
that you sidestep by using separate alert definitions to begin with.
/MR
On Fri, 9 Dec 2022, 08:31 Sebastian Glock, mailto:sebagloc...@gmail.com> > wrote:
Hi,
I'm having trouble setting up an alert that will send a notification when a
value is different from 0 and the value is missing (i.e. null).
expression:
windows_mscluster_resourcegroup_state {name!~"Available Storage"} != 0 or on()
vector(0)
The alert goes off non-stop. How can I set the metric to send an alert when the
value is different from 0 and is null?
I tried with sum() but not working anyway:
sum(windows_mscluster_resourcegroup_state {name!~"Available Storage"} != 0) or
on() vector(0)
Thanks for replies!
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com
<mailto:prometheus-users+unsubscr...@googlegroups.com> .
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/9fbc7d5d-c7ce-4b93-b653-733cac798956n%40googlegroups.com
<https://groups.google.com/d/msgid/prometheus-users/9fbc7d5d-c7ce-4b93-b653-733cac798956n%40googlegroups.com?utm_medium=email_source=footer>
.
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/04e501d90bab%242c5659b0%2485030d10%24%40gmail.com.
[prometheus-users] Null value in alerts
Hi,
I'm having trouble setting up an alert that will send a notification when a
value is different from 0 and the value is missing (i.e. null).
expression:
windows_mscluster_resourcegroup_state {name!~"Available Storage"} != 0 or
on() vector(0)
The alert goes off non-stop. How can I set the metric to send an alert when
the value is different from 0 and is null?
I tried with sum() but not working anyway:
sum(windows_mscluster_resourcegroup_state {name!~"Available Storage"} != 0)
or on() vector(0)
Thanks for replies!
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/9fbc7d5d-c7ce-4b93-b653-733cac798956n%40googlegroups.com.
