Since you don't care about the status, the typical thing to do is us a
sum() aggregator to remove the label.
sum without (status) (increase(response_total{status!="200"}[10m])) / sum
without (status) (increase(response_total{status="200"}[10m]))
On Tue, Dec 29, 2020 at 11:36 PM Alex K <[email protected]> wrote:
> I have a counter metric called response_total. It has labels source,
> status, and service, plus a few more, but those are the important ones for
> this question.
>
> response_total{status="200", source="foo", service="bar"} is the counter
> for successful requests from a service or job called "foo" to a service
> called "bar".
> response_total{status!="200", source="foo", service="bar"} is the counter
> for failed requests from a service or job called "foo" to a service called
> "bar".
>
> I'm trying to define an alert that will trigger if there's a sudden
> increase of non-200 requests from a specific source to a specific service
> relative the increase of 200 requests for the same (source, service). E.g.,
> if the increase of non-200 requests over the last 10 minutes is 10x greater
> than the increase of 200 requests, trigger an alert.
>
> I'm a bit stuck on how to define this as an expression. So far I've
> converged on something along these lines:
>
> *increase(response_total{status!="200"}[10m])) /
> increase(response_total{status="200"}[10m]) > 10*
>
> This doesn't seem to work, and it's not particularly surprising. I'm not
> sure how prometheus should "know" that it should be comparing
> response_total{status!="200", source="foo", service="bar"} to
> response_total{status="200", source="foo", service="bar"}.
>
> I could define the service up-front, but the sources are defined by our
> cluster manager, so I can't enumerate them all up-front.
>
> I appreciate any help!
>
> Thanks,
> Alex
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/d5f90eb5-f7f5-4049-bbd6-50d530edc545n%40googlegroups.com
> <https://groups.google.com/d/msgid/prometheus-users/d5f90eb5-f7f5-4049-bbd6-50d530edc545n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/CABbyFmpgc%3DnqsAe5v4kNJdciRXH5Xas2pQOeMFDoBg5HhPaqGQ%40mail.gmail.com.
