Almost forgot, the problem behaves in a strange way.
There are statements in logs:
...
level=error ts=2021-02-10T10:47:20.270Z caller=notify.go:372
component=dispatcher msg="Error on notify" err="cancelling notify retry for
\"pageduty\" due to unrecoverable error: unexpected status code 422:
{\"message\":\"Request body is not processable. Please check the
errors.\",\"errors\":{\"message\":\"Message can not be
empty.\"},\"took\":0.001,\"requestId\":\"ee7f2788-b567-4134-bcde-96080025b392\"}"
context_err=null
level=error ts=2021-02-10T10:47:20.271Z caller=dispatch.go:301
component=dispatcher msg="Notify for alerts failed" num_alerts=246
err="cancelling notify retry for \"pageduty\" due to unrecoverable error:
unexpected status code 422: {\"message\":\"Request body is not processable.
Please check the errors.\",\"errors\":{\"message\":\"Message can not be
empty.\"},\"took\":0.001,\"requestId\":\"ee7f2788-b567-4134-bcde-96080025b392\"}"
...
PagerDuty answers that he receives an empty message and he can't process
such a request.
This is damn weird. How my config affects the message body and the whole
payload of the alert?
P.S. I'm already tried these regexps:
.*
.+
.*$
^(.*)$
^([a-zA-Z0-9-]*)&
and many more :(
It works only if I set 'pagerduty' as a default receiver and the unwanted
alerts are being routed via 'blackhole' configured with a looooong
'match_re" expression, but this is madness. I don't want to stalk every new
infrastructure alert and add it to the expression :(
On Wednesday, February 10, 2021 at 1:16:21 AM UTC+3 Anton Tokarev wrote:
> Hey, Björn! 0/
>
> I'll try to describe the issue.
>
> I have a bunch of K8s clusters that sends alerts to my Alertmanager. Every
> cluster has its own Prometheus in it. Infrastructure guys sharing these
> Prometheuses between devs and send alerts to two Alertmanager
> simultaneously (weird, I know, but there is no chance to set up devs their
> own Prometheus instance in a cluster).
>
> The thing is that I want to separate devs alerts from infrastructure ones
> and send infrastructure alerts to "/dev/null/" while devs alerts should be
> routed to a certain receiver.
>
> For this case, all the devs alert marked with a special label:
> team_name="team1|team2|teamN".
>
> Devs teams count will grow without any control, so I can't define the
> static regexp rules for `match_re`.
>
> And here's the main issue - I can't set a regexp rule for `match_re` like
> `team_name: '.*'`. It's just doesn't work.
>
> Here's my Alertmanager config:
>
> global:
> ...
> route:
> group_by:
> - '...'
> group_wait: 30s
> group_interval: 5m
> repeat_interval: 12h
> receiver: blackhole
> routes:
> - match_re:
> team_name: '.*'
> receiver: pageduty
> receivers:
> - name: blackhole
> - name: pageduty
> ...
>
> Thanks in advance!
> On Thursday, January 7, 2021 at 6:57:02 PM UTC+3 [email protected] wrote:
>
>> On 30.12.20 07:30, Anton Tokarev wrote:
>> >
>> > I've tried to use regex 'my_label: .+' but seems it doesn't work.
>> >
>> > ```
>> > ...
>> > route:
>> > group_by:
>> > - '...'
>> > group_wait: 30s
>> > group_interval: 5m
>> > repeat_interval: 12h
>> > receiver: 'blackhole'
>> > routes:
>> > - match_re:
>> > opsgenie_team_id: '.+'
>> > receiver: 'opsgenie'
>> > receivers:
>> > - name: 'blackhole'
>> > - name: 'opsgenie'
>> > ...
>> > ```
>> >
>> > Could you guys please help?
>>
>> At first glance, I'd say it should work.
>>
>> Could you describe more precisely what you are seeing? Like show us
>> ladies an example alert that goes down a route that you didn't expect?
>>
>> --
>> Björn Rabenstein
>> [PGP-ID] 0x851C3DA17D748D03
>> [email] [email protected]
>>
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/4ce3ff03-cad2-406b-9a68-a34aab10d3efn%40googlegroups.com.
