There are two major problems with that confgi:
- there is a "routes" block nested under "inhibit_rules", where it is not
allowed
- there is a "matchers" block nested under "receivers", where it is not
allowed
And if I try to start alertmanager 0.25.0 (the version you say you're
using) with that exact configuration, it aborts.
root@prometheus:~# /opt/alertmanager-0.25.0.linux-amd64/alertmanager
--config.file=amtest.conf --web.listen-address=:9193
--cluster.listen-address=0.0.0.0:9194
ts=2025-01-07T08:51:23.716Z caller=main.go:240 level=info msg="Starting
Alertmanager" version="(version=0.25.0, branch=HEAD,
revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)"
ts=2025-01-07T08:51:23.716Z caller=main.go:241 level=info
build_context="(go=go1.19.4, user=root@abe866dd5717,
date=20221222-14:51:36)"
ts=2025-01-07T08:51:23.723Z caller=cluster.go:185 level=info
component=cluster msg="setting advertise address explicitly"
addr=10.12.255.33 port=9194
ts=2025-01-07T08:51:23.726Z caller=cluster.go:681 level=info
component=cluster msg="Waiting for gossip to settle..." interval=2s
ts=2025-01-07T08:51:23.761Z caller=coordinator.go:113 level=info
component=configuration msg="Loading configuration file" file=amtest.conf
*ts=2025-01-07T08:51:23.761Z caller=coordinator.go:118 level=error
component=configuration msg="Loading configuration file failed"
file=amtest.conf err="yaml: unmarshal errors:\n line 27: field routes not
found in type config.plain\n line 43: field matchers not found in type
config.plain"*
ts=2025-01-07T08:51:23.761Z caller=cluster.go:690 level=info
component=cluster msg="gossip not settled but continuing anyway" polls=0
elapsed=35.105627ms
You cannot possibly be running a working alertmanager with this config.
On Monday, 6 January 2025 at 18:40:09 UTC Chrysale Tchako wrote:
> This is the exact configuration of my alermanager.yml file (some values
> were substituted for privacy/security) do you notice any discrepancies?
>
> global:
> smtp_hello: 'example.com'
>
>
> route:
> group_wait: 1m
> group_interval: 5m
> repeat_interval: 12h
> group_by: ["alertname", "severity"]
> # Default receiver (for all alerts that don't match any specific route)
> receiver: "default-receiver"
>
> routes:
> - receiver: "default-receiver" # This is for other alerts, not
> EndpointDown or dev-team alerts
> continue: true
> - match:
> service: "EndpointDown"
> receiver: "custom-receiver"
>
>
> inhibit_rules:
> - source_matchers:
> - severity="critical"
> target_matchers:
> - severity="warning"
> equal: ['alertname', 'instance']
> routes:
> - {receiver: default-receiver, continue: true}
> - {receiver: custom-receiver}
>
>
> receivers:
> # Default receiver for other alerts
> - name: "default-receiver"
> email_configs:
> - to: "[email protected], [email protected]"
> from: '[email protected]'
> smarthost: 'smtp-relay.example.com:587'
> send_resolved: true
>
> # Receiver for custom alerts (both EndpointDown and dev-team alerts)
> - name: "custom-receiver"
> matchers:
> - service: "EndpointDown"
> email_configs:
> - to: '[email protected]'
> from: '[email protected]'
> smarthost: 'smtp-relay.example.com:587'
> send_resolved: true
>
> On Monday, January 6, 2025 at 1:03:08 PM UTC-5 Brian Candler wrote:
>
>> Then something strange is going on. Are you 100% sure that you pasted
>> *exactly* the configuration file you're using, with *exactly* the correct
>> contents and indentation?
>>
>> You can add labels in your service discovery, e.g. if you are using
>> file_sd_configs then inside the targets file you can put
>>
>> - labels:
>> owner: teamA
>> targets:
>> - foo
>> - bar
>> - baz
>> - labels:
>> owner: teamB
>> targets:
>> - qux
>>
>> In this case, all timeseries which are scraped from the hosts will have
>> those labels added as attributes of every timeseries.
>>
>> Or you can add labels in your alerting rules:
>>
>> groups:
>> - name: UpDown
>> rules:
>> - alert: UpDownTeamA
>> expr: up{instance=~"foo|bar|baz"} == 0
>> for: 3m
>> keep_firing_for: 3m
>> labels:
>> owner: teamA
>> - alert: UpDownTeamB
>> expr: up{instance=~"qux"} == 0
>> for: 3m
>> keep_firing_for: 3m
>> labels:
>> owner: teamB
>>
>> It's up to you on whether you want to label the timeseries themselves
>> (which is simpler but less flexible), or do the labelling in the alerting
>> rules (which means you may need to look at other labels to decide how to
>> route things, but allows you to change your alert routing without changing
>> the underlying timeseries)
>>
>> In alertmanager you'd then match labels to decide where to deliver.
>>
>> route:
>>
>> # Default receiver (for all alerts that don't match any specific route)
>> receiver: "everyone"
>>
>> routes:
>> - matchers:
>> - owner=teamA
>> receiver: "gmail"
>> - matchers:
>> - owner=teamB
>> receiver: "dev-receiver"
>>
>> This simple example says alerts with owner=teamA go to "gmail", alerts
>> with owner=teamB go to "dev-receiver", and everything else goes to
>> "everyone". (The first match stops any subsequent processing, unless that
>> rule has "continue: true")
>>
>> There is a tool you can test your alertmanager config with here:
>> https://prometheus.io/webtools/alerting/routing-tree-editor/
>>
>> Try pasting in the above block, then click "Draw routing tree". Then next
>> to Match Label Set, enter
>> {instance="foo",owner="teamA"}
>> and click Match Label Set.
>>
>> HTH.
>>
>> On Monday, 6 January 2025 at 16:39:49 UTC Chrysale Tchako wrote:
>>
>>>
>>> I believe this is the correct config file because when I mute an email
>>> from one of the receivers that email is no longer included in any
>>> notification that gets sent out. As far as the labels you mentioned
>>> earlier, could you share an example of how to implement or format those in
>>> my alertmanager.yml and alert.rules.yml files.
>>> On Friday, January 3, 2025 at 11:15:20 AM UTC-5 Brian Candler wrote:
>>>
>>>> I have retested this with alertmanager 0.25.0, and it still rejects the
>>>> config file you supplied as invalid, refusing to start. Therefore, I think
>>>> you're *not* running with the configuration file you think you are. This
>>>> is
>>>> the first issue to resolve.
>>>>
>>>> You can prove this to yourself by running a separate instance of
>>>> alertmanager on the command line, pointing it explicitly at the config
>>>> file
>>>> in question:
>>>>
>>>> # /opt/alertmanager-0.25.0.linux-amd64/alertmanager
>>>> --config.file=amtest.conf.orig --web.listen-address=:9193
>>>> --cluster.listen-address=0.0.0.0:9194
>>>> ts=2025-01-03T16:10:12.215Z caller=main.go:240 level=info msg="Starting
>>>> Alertmanager" version="(version=0.25.0, branch=HEAD,
>>>> revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)"
>>>> ts=2025-01-03T16:10:12.215Z caller=main.go:241 level=info
>>>> build_context="(go=go1.19.4, user=root@abe866dd5717,
>>>> date=20221222-14:51:36)"
>>>> ts=2025-01-03T16:10:12.217Z caller=cluster.go:185 level=info
>>>> component=cluster msg="setting advertise address explicitly"
>>>> addr=10.12.255.33 port=9194
>>>> ts=2025-01-03T16:10:12.219Z caller=cluster.go:681 level=info
>>>> component=cluster msg="Waiting for gossip to settle..." interval=2s
>>>> ts=2025-01-03T16:10:12.249Z caller=coordinator.go:113 level=info
>>>> component=configuration msg="Loading configuration file"
>>>> file=amtest.conf.orig
>>>> ts=2025-01-03T16:10:12.249Z caller=coordinator.go:118 level=error
>>>> component=configuration *msg="Loading configuration file failed"
>>>> file=amtest.conf.orig err="yaml: unmarshal errors:\n line 12: field
>>>> routes
>>>> not found in type config.plain\n line 23: field routes not found in type
>>>> config.plain"*
>>>> ts=2025-01-03T16:10:12.249Z caller=cluster.go:690 level=info
>>>> component=cluster msg="gossip not settled but continuing anyway" polls=0
>>>> elapsed=30.684346ms
>>>>
>>>> (The default config file is "alertmanager.yml", but as there is no
>>>> absolute path, it will be in whatever the current working directory is
>>>> when
>>>> alertmanager is started. This may not be where you think it is, and in
>>>> particular, is not necessarily the same directory as alertmanager itself)
>>>>
>>>> > Let's say I have Team A receiver for Team A resources and Team B
>>>> receiver for Team B resources, I want to isolate the notifications for
>>>> each
>>>> team. Neither team should be aware of or get notified about the other
>>>> teams
>>>> resources. How can I accomplish that?
>>>>
>>>> You set labels on the Team A / Team B alerts, and use those labels in
>>>> routing rules in alertmanager to route them appropriately. But there's no
>>>> point trying to do that until you've worked out why the system isn't using
>>>> the configuration file you think it is.
>>>>
>>>> On Friday, 3 January 2025 at 15:43:30 UTC Chrysale Tchako wrote:
>>>>
>>>>> Currently I am running alertmanager version 0.25.0, and I am making
>>>>> changes/modifications directly in the alertmanager.yml file, this is the
>>>>> file containing the information I shared in my initial inquiry. Or are
>>>>> referring to another file? I updated my routes config to match what you
>>>>> recommended but, email notifications are still getting sent to all emails
>>>>> in the alertmanager.yml file regardless of the receiver they are
>>>>> associated
>>>>> with. What I am trying to accomplish is a custom alert/notification for
>>>>> each receiver.
>>>>>
>>>>> Let's say I have Team A receiver for Team A resources and Team B
>>>>> receiver for Team B resources, I want to isolate the notifications for
>>>>> each
>>>>> team. Neither team should be aware of or get notified about the other
>>>>> teams
>>>>> resources. How can I accomplish that? What modifications should I make to
>>>>> my alertmanager.yml file? Also, do I need to make any modifications to
>>>>> my alert.rules.yml file or any other file that is?
>>>>>
>>>>> Please let me know if you need any additional information.
>>>>>
>>>>> On Thursday, January 2, 2025 at 11:56:08 AM UTC-5 Brian Candler wrote:
>>>>>
>>>>>> What version of alertmanager are you using? Your configuration is
>>>>>> invalid, so I don't know how you're even getting alertmanager to start.
>>>>>>
>>>>>> You have a top-level "routes:" block (before "inhibit_rules"), and
>>>>>> you have another "routes:" block nested under inhibit_rules, but neither
>>>>>> of
>>>>>> these are allowed by Alertmanager. If I try to run your config with
>>>>>> alertmanager-0.27.0 I get an explicit error:
>>>>>>
>>>>>> ts=2025-01-02T16:41:05.905Z caller=coordinator.go:118 level=error
>>>>>> component=configuration msg="Loading configuration file failed"
>>>>>> file=amtest.conf err="yaml: unmarshal errors:\n* line 12: field
>>>>>> routes not found in type config.plain\n line 23: field routes not found
>>>>>> in
>>>>>> type config.plain*"
>>>>>>
>>>>>> It would be valid to nest "routes" under "route", like this:
>>>>>>
>>>>>> route:
>>>>>> group_wait: 1m
>>>>>> group_interval: 5m
>>>>>> repeat_interval: 12h
>>>>>> group_by: ["alertname", "severity"]
>>>>>> # Default receiver (for all alerts that don't match any specific
>>>>>> route)
>>>>>> receiver: "gmail"
>>>>>>
>>>>>> routes:
>>>>>> - receiver: "gmail"
>>>>>> continue: true
>>>>>> - receiver: "dev-receiver"
>>>>>>
>>>>>> Note the spacing: "routes:" is indented to line up with other
>>>>>> attributes like "receiver", not at the top level. Furthermore, if you
>>>>>> did,
>>>>>> it could explain the behaviour you're seeing, since this configuration
>>>>>> requests all messages to be sent to both receivers. But since you say
>>>>>> that's *not* your actual configuration, I think we have to start
>>>>>> somewhere
>>>>>> else.
>>>>>>
>>>>>> How are you getting your config into alertmanager? Are you editing
>>>>>> the alertmanager configuration directly, or is it coming via some
>>>>>> frontend
>>>>>> like a helm chart which spits out the config? If so, can you show the
>>>>>> generated config, which is the actual file that alertmanager consumes?
>>>>>>
>>>>>> > What changes can I make so that each receiver gets it's own
>>>>>> individual email notification instead of grouping them together?
>>>>>>
>>>>>> What do you mean by "individual email notification instead of
>>>>>> grouping them together"? Do you want some alerts to go to one receiver,
>>>>>> and some alerts to go to a different receiver? If so, how do you want to
>>>>>> choose which alerts go where?
>>>>>>
>>>>>> On Thursday, 2 January 2025 at 15:16:26 UTC Chrysale Tchako wrote:
>>>>>>
>>>>>>> This is my current alertmanager.yml config, but all notifications
>>>>>>> sent out get sent to all emails added in the "to" section of the email
>>>>>>> configs. What changes can I make so that each receiver gets it's own
>>>>>>> individual email notification instead of grouping them together? Note
>>>>>>> the
>>>>>>> "from" email is the same for both receivers.
>>>>>>>
>>>>>>> global:
>>>>>>> smtp_hello: '*******.net'
>>>>>>>
>>>>>>> route:
>>>>>>> group_wait: 1m
>>>>>>> group_interval: 5m
>>>>>>> repeat_interval: 12h
>>>>>>> group_by: ["alertname", "severity"]
>>>>>>> # Default receiver (for all alerts that don't match any specific
>>>>>>> route)
>>>>>>> receiver: "gmail"
>>>>>>>
>>>>>>> routes:
>>>>>>> - receiver: "gmail"
>>>>>>> continue: true
>>>>>>> - receiver: "dev-receiver"
>>>>>>>
>>>>>>> inhibit_rules:
>>>>>>> - source_matchers:
>>>>>>> - severity="critical"
>>>>>>> target_matchers:
>>>>>>> - severity="warning"
>>>>>>> equal: ['alertname', 'instance']
>>>>>>> routes:
>>>>>>> - receiver: "gmail"
>>>>>>> continue: true
>>>>>>> - receiver: "dev-receiver"
>>>>>>>
>>>>>>> receivers:
>>>>>>> # Default receiver for other alerts
>>>>>>> - name: "gmail"
>>>>>>> email_configs:
>>>>>>> - to: "email1.net, email2.net"
>>>>>>> from: '*******.net'
>>>>>>> smarthost: 'smtp-relay.gmail.com:587'
>>>>>>> send_resolved: true
>>>>>>>
>>>>>>> - name: "dev-receiver"
>>>>>>> email_configs:
>>>>>>> - to: 'email3.net'
>>>>>>> from: '*******.net'
>>>>>>> smarthost: 'smtp-relay.gmail.com:587'
>>>>>>> send_resolved: true
>>>>>>>
>>>>>>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion visit
https://groups.google.com/d/msgid/prometheus-users/324a6655-6199-4baa-bf50-4b7b4d77dab8n%40googlegroups.com.
