You should remove the two sections that I highlighted. The following is a
valid config, and is accepted:
---- 8< ----
global:
smtp_hello: 'example.com'
route:
group_wait: 1m
group_interval: 5m
repeat_interval: 12h
group_by: ["alertname", "severity"]
# Default receiver (for all alerts that don't match any specific route)
receiver: "default-receiver"
routes:
- receiver: "default-receiver" # This is for other alerts, not
EndpointDown or dev-team alerts
continue: true
- match:
service: "EndpointDown"
receiver: "custom-receiver"
inhibit_rules:
- source_matchers:
- severity="critical"
target_matchers:
- severity="warning"
equal: ['alertname', 'instance']
receivers:
# Default receiver for other alerts
- name: "default-receiver"
email_configs:
- to: "[email protected], [email protected]"
from: '[email protected]'
smarthost: 'smtp-relay.example.com:587'
send_resolved: true
# Receiver for custom alerts (both EndpointDown and dev-team alerts)
- name: "custom-receiver"
email_configs:
- to: '[email protected]'
from: '[email protected]'
smarthost: 'smtp-relay.example.com:587'
send_resolved: true
---- 8< ----
But if you tried to use the config you posted, and you think it was
accepted, then it must have been reading a different config file.
Note: the config you have written will do the following:
* ALL alerts (including EndpointDown) will be sent to default-receiver
* EndpointDown alerts will additionally be sent to custom-receiver
The receiver under the comment "# Default receiver (for all alerts that
don't match any specific route)" is never actually used, since one of the
child routes always matches - but you still have to provide it.
On Tuesday, 7 January 2025 at 14:22:27 UTC Chrysale Tchako wrote:
> What modifications would you recommend making to the file? Or better yet,
> how should it be formatted?
>
> On Tuesday, January 7, 2025 at 3:55:27 AM UTC-5 Brian Candler wrote:
>
>> There are two major problems with that confgi:
>> - there is a "routes" block nested under "inhibit_rules", where it is not
>> allowed
>> - there is a "matchers" block nested under "receivers", where it is not
>> allowed
>>
>> And if I try to start alertmanager 0.25.0 (the version you say you're
>> using) with that exact configuration, it aborts.
>>
>> root@prometheus:~# /opt/alertmanager-0.25.0.linux-amd64/alertmanager
>> --config.file=amtest.conf --web.listen-address=:9193
>> --cluster.listen-address=0.0.0.0:9194
>> ts=2025-01-07T08:51:23.716Z caller=main.go:240 level=info msg="Starting
>> Alertmanager" version="(version=0.25.0, branch=HEAD,
>> revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)"
>> ts=2025-01-07T08:51:23.716Z caller=main.go:241 level=info
>> build_context="(go=go1.19.4, user=root@abe866dd5717,
>> date=20221222-14:51:36)"
>> ts=2025-01-07T08:51:23.723Z caller=cluster.go:185 level=info
>> component=cluster msg="setting advertise address explicitly"
>> addr=10.12.255.33 port=9194
>> ts=2025-01-07T08:51:23.726Z caller=cluster.go:681 level=info
>> component=cluster msg="Waiting for gossip to settle..." interval=2s
>> ts=2025-01-07T08:51:23.761Z caller=coordinator.go:113 level=info
>> component=configuration msg="Loading configuration file" file=amtest.conf
>> *ts=2025-01-07T08:51:23.761Z caller=coordinator.go:118 level=error
>> component=configuration msg="Loading configuration file failed"
>> file=amtest.conf err="yaml: unmarshal errors:\n line 27: field routes not
>> found in type config.plain\n line 43: field matchers not found in type
>> config.plain"*
>> ts=2025-01-07T08:51:23.761Z caller=cluster.go:690 level=info
>> component=cluster msg="gossip not settled but continuing anyway" polls=0
>> elapsed=35.105627ms
>>
>> You cannot possibly be running a working alertmanager with this config.
>>
>> On Monday, 6 January 2025 at 18:40:09 UTC Chrysale Tchako wrote:
>>
>>> This is the exact configuration of my alermanager.yml file (some values
>>> were substituted for privacy/security) do you notice any discrepancies?
>>>
>>> global:
>>> smtp_hello: 'example.com'
>>>
>>>
>>> route:
>>> group_wait: 1m
>>> group_interval: 5m
>>> repeat_interval: 12h
>>> group_by: ["alertname", "severity"]
>>> # Default receiver (for all alerts that don't match any specific route)
>>> receiver: "default-receiver"
>>>
>>> routes:
>>> - receiver: "default-receiver" # This is for other alerts, not
>>> EndpointDown or dev-team alerts
>>> continue: true
>>> - match:
>>> service: "EndpointDown"
>>> receiver: "custom-receiver"
>>>
>>>
>>> inhibit_rules:
>>> - source_matchers:
>>> - severity="critical"
>>> target_matchers:
>>> - severity="warning"
>>> equal: ['alertname', 'instance']
>>> routes:
>>> - {receiver: default-receiver, continue: true}
>>> - {receiver: custom-receiver}
>>>
>>>
>>> receivers:
>>> # Default receiver for other alerts
>>> - name: "default-receiver"
>>> email_configs:
>>> - to: "[email protected], [email protected]"
>>> from: '[email protected]'
>>> smarthost: 'smtp-relay.example.com:587'
>>> send_resolved: true
>>>
>>> # Receiver for custom alerts (both EndpointDown and dev-team alerts)
>>> - name: "custom-receiver"
>>> matchers:
>>> - service: "EndpointDown"
>>> email_configs:
>>> - to: '[email protected]'
>>> from: '[email protected]'
>>> smarthost: 'smtp-relay.example.com:587'
>>> send_resolved: true
>>>
>>> On Monday, January 6, 2025 at 1:03:08 PM UTC-5 Brian Candler wrote:
>>>
>>>> Then something strange is going on. Are you 100% sure that you pasted
>>>> *exactly* the configuration file you're using, with *exactly* the correct
>>>> contents and indentation?
>>>>
>>>> You can add labels in your service discovery, e.g. if you are using
>>>> file_sd_configs then inside the targets file you can put
>>>>
>>>> - labels:
>>>> owner: teamA
>>>> targets:
>>>> - foo
>>>> - bar
>>>> - baz
>>>> - labels:
>>>> owner: teamB
>>>> targets:
>>>> - qux
>>>>
>>>> In this case, all timeseries which are scraped from the hosts will have
>>>> those labels added as attributes of every timeseries.
>>>>
>>>> Or you can add labels in your alerting rules:
>>>>
>>>> groups:
>>>> - name: UpDown
>>>> rules:
>>>> - alert: UpDownTeamA
>>>> expr: up{instance=~"foo|bar|baz"} == 0
>>>> for: 3m
>>>> keep_firing_for: 3m
>>>> labels:
>>>> owner: teamA
>>>> - alert: UpDownTeamB
>>>> expr: up{instance=~"qux"} == 0
>>>> for: 3m
>>>> keep_firing_for: 3m
>>>> labels:
>>>> owner: teamB
>>>>
>>>> It's up to you on whether you want to label the timeseries themselves
>>>> (which is simpler but less flexible), or do the labelling in the alerting
>>>> rules (which means you may need to look at other labels to decide how to
>>>> route things, but allows you to change your alert routing without changing
>>>> the underlying timeseries)
>>>>
>>>> In alertmanager you'd then match labels to decide where to deliver.
>>>>
>>>> route:
>>>>
>>>> # Default receiver (for all alerts that don't match any specific
>>>> route)
>>>> receiver: "everyone"
>>>>
>>>> routes:
>>>> - matchers:
>>>> - owner=teamA
>>>> receiver: "gmail"
>>>> - matchers:
>>>> - owner=teamB
>>>> receiver: "dev-receiver"
>>>>
>>>> This simple example says alerts with owner=teamA go to "gmail", alerts
>>>> with owner=teamB go to "dev-receiver", and everything else goes to
>>>> "everyone". (The first match stops any subsequent processing, unless that
>>>> rule has "continue: true")
>>>>
>>>> There is a tool you can test your alertmanager config with here:
>>>> https://prometheus.io/webtools/alerting/routing-tree-editor/
>>>>
>>>> Try pasting in the above block, then click "Draw routing tree". Then
>>>> next to Match Label Set, enter
>>>> {instance="foo",owner="teamA"}
>>>> and click Match Label Set.
>>>>
>>>> HTH.
>>>>
>>>> On Monday, 6 January 2025 at 16:39:49 UTC Chrysale Tchako wrote:
>>>>
>>>>>
>>>>> I believe this is the correct config file because when I mute an email
>>>>> from one of the receivers that email is no longer included in any
>>>>> notification that gets sent out. As far as the labels you mentioned
>>>>> earlier, could you share an example of how to implement or format those
>>>>> in
>>>>> my alertmanager.yml and alert.rules.yml files.
>>>>> On Friday, January 3, 2025 at 11:15:20 AM UTC-5 Brian Candler wrote:
>>>>>
>>>>>> I have retested this with alertmanager 0.25.0, and it still rejects
>>>>>> the config file you supplied as invalid, refusing to start. Therefore, I
>>>>>> think you're *not* running with the configuration file you think you
>>>>>> are.
>>>>>> This is the first issue to resolve.
>>>>>>
>>>>>> You can prove this to yourself by running a separate instance of
>>>>>> alertmanager on the command line, pointing it explicitly at the config
>>>>>> file
>>>>>> in question:
>>>>>>
>>>>>> # /opt/alertmanager-0.25.0.linux-amd64/alertmanager
>>>>>> --config.file=amtest.conf.orig --web.listen-address=:9193
>>>>>> --cluster.listen-address=0.0.0.0:9194
>>>>>> ts=2025-01-03T16:10:12.215Z caller=main.go:240 level=info
>>>>>> msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD,
>>>>>> revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)"
>>>>>> ts=2025-01-03T16:10:12.215Z caller=main.go:241 level=info
>>>>>> build_context="(go=go1.19.4, user=root@abe866dd5717,
>>>>>> date=20221222-14:51:36)"
>>>>>> ts=2025-01-03T16:10:12.217Z caller=cluster.go:185 level=info
>>>>>> component=cluster msg="setting advertise address explicitly"
>>>>>> addr=10.12.255.33 port=9194
>>>>>> ts=2025-01-03T16:10:12.219Z caller=cluster.go:681 level=info
>>>>>> component=cluster msg="Waiting for gossip to settle..." interval=2s
>>>>>> ts=2025-01-03T16:10:12.249Z caller=coordinator.go:113 level=info
>>>>>> component=configuration msg="Loading configuration file"
>>>>>> file=amtest.conf.orig
>>>>>> ts=2025-01-03T16:10:12.249Z caller=coordinator.go:118 level=error
>>>>>> component=configuration *msg="Loading configuration file failed"
>>>>>> file=amtest.conf.orig err="yaml: unmarshal errors:\n line 12: field
>>>>>> routes
>>>>>> not found in type config.plain\n line 23: field routes not found in
>>>>>> type
>>>>>> config.plain"*
>>>>>> ts=2025-01-03T16:10:12.249Z caller=cluster.go:690 level=info
>>>>>> component=cluster msg="gossip not settled but continuing anyway" polls=0
>>>>>> elapsed=30.684346ms
>>>>>>
>>>>>> (The default config file is "alertmanager.yml", but as there is no
>>>>>> absolute path, it will be in whatever the current working directory is
>>>>>> when
>>>>>> alertmanager is started. This may not be where you think it is, and in
>>>>>> particular, is not necessarily the same directory as alertmanager itself)
>>>>>>
>>>>>> > Let's say I have Team A receiver for Team A resources and Team B
>>>>>> receiver for Team B resources, I want to isolate the notifications for
>>>>>> each
>>>>>> team. Neither team should be aware of or get notified about the other
>>>>>> teams
>>>>>> resources. How can I accomplish that?
>>>>>>
>>>>>> You set labels on the Team A / Team B alerts, and use those labels in
>>>>>> routing rules in alertmanager to route them appropriately. But there's
>>>>>> no
>>>>>> point trying to do that until you've worked out why the system isn't
>>>>>> using
>>>>>> the configuration file you think it is.
>>>>>>
>>>>>> On Friday, 3 January 2025 at 15:43:30 UTC Chrysale Tchako wrote:
>>>>>>
>>>>>>> Currently I am running alertmanager version 0.25.0, and I am making
>>>>>>> changes/modifications directly in the alertmanager.yml file, this is
>>>>>>> the
>>>>>>> file containing the information I shared in my initial inquiry. Or are
>>>>>>> referring to another file? I updated my routes config to match what you
>>>>>>> recommended but, email notifications are still getting sent to all
>>>>>>> emails
>>>>>>> in the alertmanager.yml file regardless of the receiver they are
>>>>>>> associated
>>>>>>> with. What I am trying to accomplish is a custom alert/notification for
>>>>>>> each receiver.
>>>>>>>
>>>>>>> Let's say I have Team A receiver for Team A resources and Team B
>>>>>>> receiver for Team B resources, I want to isolate the notifications for
>>>>>>> each
>>>>>>> team. Neither team should be aware of or get notified about the other
>>>>>>> teams
>>>>>>> resources. How can I accomplish that? What modifications should I make
>>>>>>> to
>>>>>>> my alertmanager.yml file? Also, do I need to make any modifications to
>>>>>>> my alert.rules.yml file or any other file that is?
>>>>>>>
>>>>>>> Please let me know if you need any additional information.
>>>>>>>
>>>>>>> On Thursday, January 2, 2025 at 11:56:08 AM UTC-5 Brian Candler
>>>>>>> wrote:
>>>>>>>
>>>>>>>> What version of alertmanager are you using? Your configuration is
>>>>>>>> invalid, so I don't know how you're even getting alertmanager to start.
>>>>>>>>
>>>>>>>> You have a top-level "routes:" block (before "inhibit_rules"), and
>>>>>>>> you have another "routes:" block nested under inhibit_rules, but
>>>>>>>> neither of
>>>>>>>> these are allowed by Alertmanager. If I try to run your config with
>>>>>>>> alertmanager-0.27.0 I get an explicit error:
>>>>>>>>
>>>>>>>> ts=2025-01-02T16:41:05.905Z caller=coordinator.go:118 level=error
>>>>>>>> component=configuration msg="Loading configuration file failed"
>>>>>>>> file=amtest.conf err="yaml: unmarshal errors:\n* line 12: field
>>>>>>>> routes not found in type config.plain\n line 23: field routes not
>>>>>>>> found in
>>>>>>>> type config.plain*"
>>>>>>>>
>>>>>>>> It would be valid to nest "routes" under "route", like this:
>>>>>>>>
>>>>>>>> route:
>>>>>>>> group_wait: 1m
>>>>>>>> group_interval: 5m
>>>>>>>> repeat_interval: 12h
>>>>>>>> group_by: ["alertname", "severity"]
>>>>>>>> # Default receiver (for all alerts that don't match any specific
>>>>>>>> route)
>>>>>>>> receiver: "gmail"
>>>>>>>>
>>>>>>>> routes:
>>>>>>>> - receiver: "gmail"
>>>>>>>> continue: true
>>>>>>>> - receiver: "dev-receiver"
>>>>>>>>
>>>>>>>> Note the spacing: "routes:" is indented to line up with other
>>>>>>>> attributes like "receiver", not at the top level. Furthermore, if you
>>>>>>>> did,
>>>>>>>> it could explain the behaviour you're seeing, since this configuration
>>>>>>>> requests all messages to be sent to both receivers. But since you say
>>>>>>>> that's *not* your actual configuration, I think we have to start
>>>>>>>> somewhere
>>>>>>>> else.
>>>>>>>>
>>>>>>>> How are you getting your config into alertmanager? Are you editing
>>>>>>>> the alertmanager configuration directly, or is it coming via some
>>>>>>>> frontend
>>>>>>>> like a helm chart which spits out the config? If so, can you show the
>>>>>>>> generated config, which is the actual file that alertmanager consumes?
>>>>>>>>
>>>>>>>> > What changes can I make so that each receiver gets it's own
>>>>>>>> individual email notification instead of grouping them together?
>>>>>>>>
>>>>>>>> What do you mean by "individual email notification instead of
>>>>>>>> grouping them together"? Do you want some alerts to go to one
>>>>>>>> receiver,
>>>>>>>> and some alerts to go to a different receiver? If so, how do you want
>>>>>>>> to
>>>>>>>> choose which alerts go where?
>>>>>>>>
>>>>>>>> On Thursday, 2 January 2025 at 15:16:26 UTC Chrysale Tchako wrote:
>>>>>>>>
>>>>>>>>> This is my current alertmanager.yml config, but all notifications
>>>>>>>>> sent out get sent to all emails added in the "to" section of the
>>>>>>>>> email
>>>>>>>>> configs. What changes can I make so that each receiver gets it's own
>>>>>>>>> individual email notification instead of grouping them together? Note
>>>>>>>>> the
>>>>>>>>> "from" email is the same for both receivers.
>>>>>>>>>
>>>>>>>>> global:
>>>>>>>>> smtp_hello: '*******.net'
>>>>>>>>>
>>>>>>>>> route:
>>>>>>>>> group_wait: 1m
>>>>>>>>> group_interval: 5m
>>>>>>>>> repeat_interval: 12h
>>>>>>>>> group_by: ["alertname", "severity"]
>>>>>>>>> # Default receiver (for all alerts that don't match any specific
>>>>>>>>> route)
>>>>>>>>> receiver: "gmail"
>>>>>>>>>
>>>>>>>>> routes:
>>>>>>>>> - receiver: "gmail"
>>>>>>>>> continue: true
>>>>>>>>> - receiver: "dev-receiver"
>>>>>>>>>
>>>>>>>>> inhibit_rules:
>>>>>>>>> - source_matchers:
>>>>>>>>> - severity="critical"
>>>>>>>>> target_matchers:
>>>>>>>>> - severity="warning"
>>>>>>>>> equal: ['alertname', 'instance']
>>>>>>>>> routes:
>>>>>>>>> - receiver: "gmail"
>>>>>>>>> continue: true
>>>>>>>>> - receiver: "dev-receiver"
>>>>>>>>>
>>>>>>>>> receivers:
>>>>>>>>> # Default receiver for other alerts
>>>>>>>>> - name: "gmail"
>>>>>>>>> email_configs:
>>>>>>>>> - to: "email1.net, email2.net"
>>>>>>>>> from: '*******.net'
>>>>>>>>> smarthost: 'smtp-relay.gmail.com:587'
>>>>>>>>> send_resolved: true
>>>>>>>>>
>>>>>>>>> - name: "dev-receiver"
>>>>>>>>> email_configs:
>>>>>>>>> - to: 'email3.net'
>>>>>>>>> from: '*******.net'
>>>>>>>>> smarthost: 'smtp-relay.gmail.com:587'
>>>>>>>>> send_resolved: true
>>>>>>>>>
>>>>>>>>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion visit
https://groups.google.com/d/msgid/prometheus-users/2b7aa424-2694-44c9-a8bf-243fed596fa2n%40googlegroups.com.
