Hi list,
For a monitored system we setup a rule as follows:
absent(our_metric{environment="pro",service="bar",stack="foo"}) and
on(stack, environment) up{service="bar",source="app"} == 1
This is one of the few absence rules we have in our ruleset. This is also a
bit special because the exporter uses the absence of the metric to indicate
a problem - something that is discouraged from guidelines. But that goes
beyond my question anyway.
Using a binary AND operator seems to work fine, cutting out the cases in
which the node is not scrapable. However this morning the node went
missing. We had probably a misconfiguration in our provisioning which we
are currently investigating.
As the node went missing the second operand of the binary operator could
not be evaluated, simply because it was neither `1`, nor `0`. Or, in other
words, the following was holding true:
absent(up{service="bar",source="app"}) = 1
I understand an alert can resolve if the related metric goes stale but I'm
not sure how the logic should translate in this case. On the surface I
would not expect the AND expression to fire as we are not able to say the
"up" metric is really 1.
But maybe I'm missing the point here?
Thanks in advance,
F.
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/CA%2B%2Bt_8MB0fdKVtshOewGaHtSarBB3ySHG3eFbGT0kYq2nS6pdQ%40mail.gmail.com.
