What modifications would you recommend making to the file? Or better yet,
how should it be formatted?
On Tuesday, January 7, 2025 at 3:55:27 AM UTC-5 Brian Candler wrote:
> There are two major problems with that confgi:
> - there is a "routes" block nested under "inhibit_rules", where it is not
> allowed
> - there is a "matchers" block nested under "receivers", where it is not
> allowed
>
> And if I try to start alertmanager 0.25.0 (the version you say you're
> using) with that exact configuration, it aborts.
>
> root@prometheus:~# /opt/alertmanager-0.25.0.linux-amd64/alertmanager
> --config.file=amtest.conf --web.listen-address=:9193
> --cluster.listen-address=0.0.0.0:9194
> ts=2025-01-07T08:51:23.716Z caller=main.go:240 level=info msg="Starting
> Alertmanager" version="(version=0.25.0, branch=HEAD,
> revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)"
> ts=2025-01-07T08:51:23.716Z caller=main.go:241 level=info
> build_context="(go=go1.19.4, user=root@abe866dd5717,
> date=20221222-14:51:36)"
> ts=2025-01-07T08:51:23.723Z caller=cluster.go:185 level=info
> component=cluster msg="setting advertise address explicitly"
> addr=10.12.255.33 port=9194
> ts=2025-01-07T08:51:23.726Z caller=cluster.go:681 level=info
> component=cluster msg="Waiting for gossip to settle..." interval=2s
> ts=2025-01-07T08:51:23.761Z caller=coordinator.go:113 level=info
> component=configuration msg="Loading configuration file" file=amtest.conf
> *ts=2025-01-07T08:51:23.761Z caller=coordinator.go:118 level=error
> component=configuration msg="Loading configuration file failed"
> file=amtest.conf err="yaml: unmarshal errors:\n line 27: field routes not
> found in type config.plain\n line 43: field matchers not found in type
> config.plain"*
> ts=2025-01-07T08:51:23.761Z caller=cluster.go:690 level=info
> component=cluster msg="gossip not settled but continuing anyway" polls=0
> elapsed=35.105627ms
>
> You cannot possibly be running a working alertmanager with this config.
>
> On Monday, 6 January 2025 at 18:40:09 UTC Chrysale Tchako wrote:
>
>> This is the exact configuration of my alermanager.yml file (some values
>> were substituted for privacy/security) do you notice any discrepancies?
>>
>> global:
>> smtp_hello: 'example.com'
>>
>>
>> route:
>> group_wait: 1m
>> group_interval: 5m
>> repeat_interval: 12h
>> group_by: ["alertname", "severity"]
>> # Default receiver (for all alerts that don't match any specific route)
>> receiver: "default-receiver"
>>
>> routes:
>> - receiver: "default-receiver" # This is for other alerts, not
>> EndpointDown or dev-team alerts
>> continue: true
>> - match:
>> service: "EndpointDown"
>> receiver: "custom-receiver"
>>
>>
>> inhibit_rules:
>> - source_matchers:
>> - severity="critical"
>> target_matchers:
>> - severity="warning"
>> equal: ['alertname', 'instance']
>> routes:
>> - {receiver: default-receiver, continue: true}
>> - {receiver: custom-receiver}
>>
>>
>> receivers:
>> # Default receiver for other alerts
>> - name: "default-receiver"
>> email_configs:
>> - to: "[email protected], [email protected]"
>> from: '[email protected]'
>> smarthost: 'smtp-relay.example.com:587'
>> send_resolved: true
>>
>> # Receiver for custom alerts (both EndpointDown and dev-team alerts)
>> - name: "custom-receiver"
>> matchers:
>> - service: "EndpointDown"
>> email_configs:
>> - to: '[email protected]'
>> from: '[email protected]'
>> smarthost: 'smtp-relay.example.com:587'
>> send_resolved: true
>>
>> On Monday, January 6, 2025 at 1:03:08 PM UTC-5 Brian Candler wrote:
>>
>>> Then something strange is going on. Are you 100% sure that you pasted
>>> *exactly* the configuration file you're using, with *exactly* the correct
>>> contents and indentation?
>>>
>>> You can add labels in your service discovery, e.g. if you are using
>>> file_sd_configs then inside the targets file you can put
>>>
>>> - labels:
>>> owner: teamA
>>> targets:
>>> - foo
>>> - bar
>>> - baz
>>> - labels:
>>> owner: teamB
>>> targets:
>>> - qux
>>>
>>> In this case, all timeseries which are scraped from the hosts will have
>>> those labels added as attributes of every timeseries.
>>>
>>> Or you can add labels in your alerting rules:
>>>
>>> groups:
>>> - name: UpDown
>>> rules:
>>> - alert: UpDownTeamA
>>> expr: up{instance=~"foo|bar|baz"} == 0
>>> for: 3m
>>> keep_firing_for: 3m
>>> labels:
>>> owner: teamA
>>> - alert: UpDownTeamB
>>> expr: up{instance=~"qux"} == 0
>>> for: 3m
>>> keep_firing_for: 3m
>>> labels:
>>> owner: teamB
>>>
>>> It's up to you on whether you want to label the timeseries themselves
>>> (which is simpler but less flexible), or do the labelling in the alerting
>>> rules (which means you may need to look at other labels to decide how to
>>> route things, but allows you to change your alert routing without changing
>>> the underlying timeseries)
>>>
>>> In alertmanager you'd then match labels to decide where to deliver.
>>>
>>> route:
>>>
>>> # Default receiver (for all alerts that don't match any specific route)
>>> receiver: "everyone"
>>>
>>> routes:
>>> - matchers:
>>> - owner=teamA
>>> receiver: "gmail"
>>> - matchers:
>>> - owner=teamB
>>> receiver: "dev-receiver"
>>>
>>> This simple example says alerts with owner=teamA go to "gmail", alerts
>>> with owner=teamB go to "dev-receiver", and everything else goes to
>>> "everyone". (The first match stops any subsequent processing, unless that
>>> rule has "continue: true")
>>>
>>> There is a tool you can test your alertmanager config with here:
>>> https://prometheus.io/webtools/alerting/routing-tree-editor/
>>>
>>> Try pasting in the above block, then click "Draw routing tree". Then
>>> next to Match Label Set, enter
>>> {instance="foo",owner="teamA"}
>>> and click Match Label Set.
>>>
>>> HTH.
>>>
>>> On Monday, 6 January 2025 at 16:39:49 UTC Chrysale Tchako wrote:
>>>
>>>>
>>>> I believe this is the correct config file because when I mute an email
>>>> from one of the receivers that email is no longer included in any
>>>> notification that gets sent out. As far as the labels you mentioned
>>>> earlier, could you share an example of how to implement or format those in
>>>> my alertmanager.yml and alert.rules.yml files.
>>>> On Friday, January 3, 2025 at 11:15:20 AM UTC-5 Brian Candler wrote:
>>>>
>>>>> I have retested this with alertmanager 0.25.0, and it still rejects
>>>>> the config file you supplied as invalid, refusing to start. Therefore, I
>>>>> think you're *not* running with the configuration file you think you are.
>>>>> This is the first issue to resolve.
>>>>>
>>>>> You can prove this to yourself by running a separate instance of
>>>>> alertmanager on the command line, pointing it explicitly at the config
>>>>> file
>>>>> in question:
>>>>>
>>>>> # /opt/alertmanager-0.25.0.linux-amd64/alertmanager
>>>>> --config.file=amtest.conf.orig --web.listen-address=:9193
>>>>> --cluster.listen-address=0.0.0.0:9194
>>>>> ts=2025-01-03T16:10:12.215Z caller=main.go:240 level=info
>>>>> msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD,
>>>>> revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)"
>>>>> ts=2025-01-03T16:10:12.215Z caller=main.go:241 level=info
>>>>> build_context="(go=go1.19.4, user=root@abe866dd5717,
>>>>> date=20221222-14:51:36)"
>>>>> ts=2025-01-03T16:10:12.217Z caller=cluster.go:185 level=info
>>>>> component=cluster msg="setting advertise address explicitly"
>>>>> addr=10.12.255.33 port=9194
>>>>> ts=2025-01-03T16:10:12.219Z caller=cluster.go:681 level=info
>>>>> component=cluster msg="Waiting for gossip to settle..." interval=2s
>>>>> ts=2025-01-03T16:10:12.249Z caller=coordinator.go:113 level=info
>>>>> component=configuration msg="Loading configuration file"
>>>>> file=amtest.conf.orig
>>>>> ts=2025-01-03T16:10:12.249Z caller=coordinator.go:118 level=error
>>>>> component=configuration *msg="Loading configuration file failed"
>>>>> file=amtest.conf.orig err="yaml: unmarshal errors:\n line 12: field
>>>>> routes
>>>>> not found in type config.plain\n line 23: field routes not found in type
>>>>> config.plain"*
>>>>> ts=2025-01-03T16:10:12.249Z caller=cluster.go:690 level=info
>>>>> component=cluster msg="gossip not settled but continuing anyway" polls=0
>>>>> elapsed=30.684346ms
>>>>>
>>>>> (The default config file is "alertmanager.yml", but as there is no
>>>>> absolute path, it will be in whatever the current working directory is
>>>>> when
>>>>> alertmanager is started. This may not be where you think it is, and in
>>>>> particular, is not necessarily the same directory as alertmanager itself)
>>>>>
>>>>> > Let's say I have Team A receiver for Team A resources and Team B
>>>>> receiver for Team B resources, I want to isolate the notifications for
>>>>> each
>>>>> team. Neither team should be aware of or get notified about the other
>>>>> teams
>>>>> resources. How can I accomplish that?
>>>>>
>>>>> You set labels on the Team A / Team B alerts, and use those labels in
>>>>> routing rules in alertmanager to route them appropriately. But there's no
>>>>> point trying to do that until you've worked out why the system isn't
>>>>> using
>>>>> the configuration file you think it is.
>>>>>
>>>>> On Friday, 3 January 2025 at 15:43:30 UTC Chrysale Tchako wrote:
>>>>>
>>>>>> Currently I am running alertmanager version 0.25.0, and I am making
>>>>>> changes/modifications directly in the alertmanager.yml file, this is the
>>>>>> file containing the information I shared in my initial inquiry. Or are
>>>>>> referring to another file? I updated my routes config to match what you
>>>>>> recommended but, email notifications are still getting sent to all
>>>>>> emails
>>>>>> in the alertmanager.yml file regardless of the receiver they are
>>>>>> associated
>>>>>> with. What I am trying to accomplish is a custom alert/notification for
>>>>>> each receiver.
>>>>>>
>>>>>> Let's say I have Team A receiver for Team A resources and Team B
>>>>>> receiver for Team B resources, I want to isolate the notifications for
>>>>>> each
>>>>>> team. Neither team should be aware of or get notified about the other
>>>>>> teams
>>>>>> resources. How can I accomplish that? What modifications should I make
>>>>>> to
>>>>>> my alertmanager.yml file? Also, do I need to make any modifications to
>>>>>> my alert.rules.yml file or any other file that is?
>>>>>>
>>>>>> Please let me know if you need any additional information.
>>>>>>
>>>>>> On Thursday, January 2, 2025 at 11:56:08 AM UTC-5 Brian Candler wrote:
>>>>>>
>>>>>>> What version of alertmanager are you using? Your configuration is
>>>>>>> invalid, so I don't know how you're even getting alertmanager to start.
>>>>>>>
>>>>>>> You have a top-level "routes:" block (before "inhibit_rules"), and
>>>>>>> you have another "routes:" block nested under inhibit_rules, but
>>>>>>> neither of
>>>>>>> these are allowed by Alertmanager. If I try to run your config with
>>>>>>> alertmanager-0.27.0 I get an explicit error:
>>>>>>>
>>>>>>> ts=2025-01-02T16:41:05.905Z caller=coordinator.go:118 level=error
>>>>>>> component=configuration msg="Loading configuration file failed"
>>>>>>> file=amtest.conf err="yaml: unmarshal errors:\n* line 12: field
>>>>>>> routes not found in type config.plain\n line 23: field routes not
>>>>>>> found in
>>>>>>> type config.plain*"
>>>>>>>
>>>>>>> It would be valid to nest "routes" under "route", like this:
>>>>>>>
>>>>>>> route:
>>>>>>> group_wait: 1m
>>>>>>> group_interval: 5m
>>>>>>> repeat_interval: 12h
>>>>>>> group_by: ["alertname", "severity"]
>>>>>>> # Default receiver (for all alerts that don't match any specific
>>>>>>> route)
>>>>>>> receiver: "gmail"
>>>>>>>
>>>>>>> routes:
>>>>>>> - receiver: "gmail"
>>>>>>> continue: true
>>>>>>> - receiver: "dev-receiver"
>>>>>>>
>>>>>>> Note the spacing: "routes:" is indented to line up with other
>>>>>>> attributes like "receiver", not at the top level. Furthermore, if you
>>>>>>> did,
>>>>>>> it could explain the behaviour you're seeing, since this configuration
>>>>>>> requests all messages to be sent to both receivers. But since you say
>>>>>>> that's *not* your actual configuration, I think we have to start
>>>>>>> somewhere
>>>>>>> else.
>>>>>>>
>>>>>>> How are you getting your config into alertmanager? Are you editing
>>>>>>> the alertmanager configuration directly, or is it coming via some
>>>>>>> frontend
>>>>>>> like a helm chart which spits out the config? If so, can you show the
>>>>>>> generated config, which is the actual file that alertmanager consumes?
>>>>>>>
>>>>>>> > What changes can I make so that each receiver gets it's own
>>>>>>> individual email notification instead of grouping them together?
>>>>>>>
>>>>>>> What do you mean by "individual email notification instead of
>>>>>>> grouping them together"? Do you want some alerts to go to one
>>>>>>> receiver,
>>>>>>> and some alerts to go to a different receiver? If so, how do you want
>>>>>>> to
>>>>>>> choose which alerts go where?
>>>>>>>
>>>>>>> On Thursday, 2 January 2025 at 15:16:26 UTC Chrysale Tchako wrote:
>>>>>>>
>>>>>>>> This is my current alertmanager.yml config, but all notifications
>>>>>>>> sent out get sent to all emails added in the "to" section of the email
>>>>>>>> configs. What changes can I make so that each receiver gets it's own
>>>>>>>> individual email notification instead of grouping them together? Note
>>>>>>>> the
>>>>>>>> "from" email is the same for both receivers.
>>>>>>>>
>>>>>>>> global:
>>>>>>>> smtp_hello: '*******.net'
>>>>>>>>
>>>>>>>> route:
>>>>>>>> group_wait: 1m
>>>>>>>> group_interval: 5m
>>>>>>>> repeat_interval: 12h
>>>>>>>> group_by: ["alertname", "severity"]
>>>>>>>> # Default receiver (for all alerts that don't match any specific
>>>>>>>> route)
>>>>>>>> receiver: "gmail"
>>>>>>>>
>>>>>>>> routes:
>>>>>>>> - receiver: "gmail"
>>>>>>>> continue: true
>>>>>>>> - receiver: "dev-receiver"
>>>>>>>>
>>>>>>>> inhibit_rules:
>>>>>>>> - source_matchers:
>>>>>>>> - severity="critical"
>>>>>>>> target_matchers:
>>>>>>>> - severity="warning"
>>>>>>>> equal: ['alertname', 'instance']
>>>>>>>> routes:
>>>>>>>> - receiver: "gmail"
>>>>>>>> continue: true
>>>>>>>> - receiver: "dev-receiver"
>>>>>>>>
>>>>>>>> receivers:
>>>>>>>> # Default receiver for other alerts
>>>>>>>> - name: "gmail"
>>>>>>>> email_configs:
>>>>>>>> - to: "email1.net, email2.net"
>>>>>>>> from: '*******.net'
>>>>>>>> smarthost: 'smtp-relay.gmail.com:587'
>>>>>>>> send_resolved: true
>>>>>>>>
>>>>>>>> - name: "dev-receiver"
>>>>>>>> email_configs:
>>>>>>>> - to: 'email3.net'
>>>>>>>> from: '*******.net'
>>>>>>>> smarthost: 'smtp-relay.gmail.com:587'
>>>>>>>> send_resolved: true
>>>>>>>>
>>>>>>>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion visit
https://groups.google.com/d/msgid/prometheus-users/a10cb9a2-38d3-4e6b-ab7a-96f0bb523e5fn%40googlegroups.com.
