This is the exact configuration of my alermanager.yml file (some values
were substituted for privacy/security) do you notice any discrepancies?
global:
smtp_hello: 'example.com'
route:
group_wait: 1m
group_interval: 5m
repeat_interval: 12h
group_by: ["alertname", "severity"]
# Default receiver (for all alerts that don't match any specific route)
receiver: "default-receiver"
routes:
- receiver: "default-receiver" # This is for other alerts, not
EndpointDown or dev-team alerts
continue: true
- match:
service: "EndpointDown"
receiver: "custom-receiver"
inhibit_rules:
- source_matchers:
- severity="critical"
target_matchers:
- severity="warning"
equal: ['alertname', 'instance']
routes:
- {receiver: default-receiver, continue: true}
- {receiver: custom-receiver}
receivers:
# Default receiver for other alerts
- name: "default-receiver"
email_configs:
- to: "[email protected], [email protected]"
from: '[email protected]'
smarthost: 'smtp-relay.example.com:587'
send_resolved: true
# Receiver for custom alerts (both EndpointDown and dev-team alerts)
- name: "custom-receiver"
matchers:
- service: "EndpointDown"
email_configs:
- to: '[email protected]'
from: '[email protected]'
smarthost: 'smtp-relay.example.com:587'
send_resolved: true
On Monday, January 6, 2025 at 1:03:08 PM UTC-5 Brian Candler wrote:
> Then something strange is going on. Are you 100% sure that you pasted
> *exactly* the configuration file you're using, with *exactly* the correct
> contents and indentation?
>
> You can add labels in your service discovery, e.g. if you are using
> file_sd_configs then inside the targets file you can put
>
> - labels:
> owner: teamA
> targets:
> - foo
> - bar
> - baz
> - labels:
> owner: teamB
> targets:
> - qux
>
> In this case, all timeseries which are scraped from the hosts will have
> those labels added as attributes of every timeseries.
>
> Or you can add labels in your alerting rules:
>
> groups:
> - name: UpDown
> rules:
> - alert: UpDownTeamA
> expr: up{instance=~"foo|bar|baz"} == 0
> for: 3m
> keep_firing_for: 3m
> labels:
> owner: teamA
> - alert: UpDownTeamB
> expr: up{instance=~"qux"} == 0
> for: 3m
> keep_firing_for: 3m
> labels:
> owner: teamB
>
> It's up to you on whether you want to label the timeseries themselves
> (which is simpler but less flexible), or do the labelling in the alerting
> rules (which means you may need to look at other labels to decide how to
> route things, but allows you to change your alert routing without changing
> the underlying timeseries)
>
> In alertmanager you'd then match labels to decide where to deliver.
>
> route:
>
> # Default receiver (for all alerts that don't match any specific route)
> receiver: "everyone"
>
> routes:
> - matchers:
> - owner=teamA
> receiver: "gmail"
> - matchers:
> - owner=teamB
> receiver: "dev-receiver"
>
> This simple example says alerts with owner=teamA go to "gmail", alerts
> with owner=teamB go to "dev-receiver", and everything else goes to
> "everyone". (The first match stops any subsequent processing, unless that
> rule has "continue: true")
>
> There is a tool you can test your alertmanager config with here:
> https://prometheus.io/webtools/alerting/routing-tree-editor/
>
> Try pasting in the above block, then click "Draw routing tree". Then next
> to Match Label Set, enter
> {instance="foo",owner="teamA"}
> and click Match Label Set.
>
> HTH.
>
> On Monday, 6 January 2025 at 16:39:49 UTC Chrysale Tchako wrote:
>
>>
>> I believe this is the correct config file because when I mute an email
>> from one of the receivers that email is no longer included in any
>> notification that gets sent out. As far as the labels you mentioned
>> earlier, could you share an example of how to implement or format those in
>> my alertmanager.yml and alert.rules.yml files.
>> On Friday, January 3, 2025 at 11:15:20 AM UTC-5 Brian Candler wrote:
>>
>>> I have retested this with alertmanager 0.25.0, and it still rejects the
>>> config file you supplied as invalid, refusing to start. Therefore, I think
>>> you're *not* running with the configuration file you think you are. This is
>>> the first issue to resolve.
>>>
>>> You can prove this to yourself by running a separate instance of
>>> alertmanager on the command line, pointing it explicitly at the config file
>>> in question:
>>>
>>> # /opt/alertmanager-0.25.0.linux-amd64/alertmanager
>>> --config.file=amtest.conf.orig --web.listen-address=:9193
>>> --cluster.listen-address=0.0.0.0:9194
>>> ts=2025-01-03T16:10:12.215Z caller=main.go:240 level=info msg="Starting
>>> Alertmanager" version="(version=0.25.0, branch=HEAD,
>>> revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)"
>>> ts=2025-01-03T16:10:12.215Z caller=main.go:241 level=info
>>> build_context="(go=go1.19.4, user=root@abe866dd5717,
>>> date=20221222-14:51:36)"
>>> ts=2025-01-03T16:10:12.217Z caller=cluster.go:185 level=info
>>> component=cluster msg="setting advertise address explicitly"
>>> addr=10.12.255.33 port=9194
>>> ts=2025-01-03T16:10:12.219Z caller=cluster.go:681 level=info
>>> component=cluster msg="Waiting for gossip to settle..." interval=2s
>>> ts=2025-01-03T16:10:12.249Z caller=coordinator.go:113 level=info
>>> component=configuration msg="Loading configuration file"
>>> file=amtest.conf.orig
>>> ts=2025-01-03T16:10:12.249Z caller=coordinator.go:118 level=error
>>> component=configuration *msg="Loading configuration file failed"
>>> file=amtest.conf.orig err="yaml: unmarshal errors:\n line 12: field routes
>>> not found in type config.plain\n line 23: field routes not found in type
>>> config.plain"*
>>> ts=2025-01-03T16:10:12.249Z caller=cluster.go:690 level=info
>>> component=cluster msg="gossip not settled but continuing anyway" polls=0
>>> elapsed=30.684346ms
>>>
>>> (The default config file is "alertmanager.yml", but as there is no
>>> absolute path, it will be in whatever the current working directory is when
>>> alertmanager is started. This may not be where you think it is, and in
>>> particular, is not necessarily the same directory as alertmanager itself)
>>>
>>> > Let's say I have Team A receiver for Team A resources and Team B
>>> receiver for Team B resources, I want to isolate the notifications for each
>>> team. Neither team should be aware of or get notified about the other teams
>>> resources. How can I accomplish that?
>>>
>>> You set labels on the Team A / Team B alerts, and use those labels in
>>> routing rules in alertmanager to route them appropriately. But there's no
>>> point trying to do that until you've worked out why the system isn't using
>>> the configuration file you think it is.
>>>
>>> On Friday, 3 January 2025 at 15:43:30 UTC Chrysale Tchako wrote:
>>>
>>>> Currently I am running alertmanager version 0.25.0, and I am making
>>>> changes/modifications directly in the alertmanager.yml file, this is the
>>>> file containing the information I shared in my initial inquiry. Or are
>>>> referring to another file? I updated my routes config to match what you
>>>> recommended but, email notifications are still getting sent to all emails
>>>> in the alertmanager.yml file regardless of the receiver they are
>>>> associated
>>>> with. What I am trying to accomplish is a custom alert/notification for
>>>> each receiver.
>>>>
>>>> Let's say I have Team A receiver for Team A resources and Team B
>>>> receiver for Team B resources, I want to isolate the notifications for
>>>> each
>>>> team. Neither team should be aware of or get notified about the other
>>>> teams
>>>> resources. How can I accomplish that? What modifications should I make to
>>>> my alertmanager.yml file? Also, do I need to make any modifications to
>>>> my alert.rules.yml file or any other file that is?
>>>>
>>>> Please let me know if you need any additional information.
>>>>
>>>> On Thursday, January 2, 2025 at 11:56:08 AM UTC-5 Brian Candler wrote:
>>>>
>>>>> What version of alertmanager are you using? Your configuration is
>>>>> invalid, so I don't know how you're even getting alertmanager to start.
>>>>>
>>>>> You have a top-level "routes:" block (before "inhibit_rules"), and you
>>>>> have another "routes:" block nested under inhibit_rules, but neither of
>>>>> these are allowed by Alertmanager. If I try to run your config with
>>>>> alertmanager-0.27.0 I get an explicit error:
>>>>>
>>>>> ts=2025-01-02T16:41:05.905Z caller=coordinator.go:118 level=error
>>>>> component=configuration msg="Loading configuration file failed"
>>>>> file=amtest.conf err="yaml: unmarshal errors:\n* line 12: field
>>>>> routes not found in type config.plain\n line 23: field routes not found
>>>>> in
>>>>> type config.plain*"
>>>>>
>>>>> It would be valid to nest "routes" under "route", like this:
>>>>>
>>>>> route:
>>>>> group_wait: 1m
>>>>> group_interval: 5m
>>>>> repeat_interval: 12h
>>>>> group_by: ["alertname", "severity"]
>>>>> # Default receiver (for all alerts that don't match any specific
>>>>> route)
>>>>> receiver: "gmail"
>>>>>
>>>>> routes:
>>>>> - receiver: "gmail"
>>>>> continue: true
>>>>> - receiver: "dev-receiver"
>>>>>
>>>>> Note the spacing: "routes:" is indented to line up with other
>>>>> attributes like "receiver", not at the top level. Furthermore, if you
>>>>> did,
>>>>> it could explain the behaviour you're seeing, since this configuration
>>>>> requests all messages to be sent to both receivers. But since you say
>>>>> that's *not* your actual configuration, I think we have to start
>>>>> somewhere
>>>>> else.
>>>>>
>>>>> How are you getting your config into alertmanager? Are you editing the
>>>>> alertmanager configuration directly, or is it coming via some frontend
>>>>> like
>>>>> a helm chart which spits out the config? If so, can you show the
>>>>> generated
>>>>> config, which is the actual file that alertmanager consumes?
>>>>>
>>>>> > What changes can I make so that each receiver gets it's own
>>>>> individual email notification instead of grouping them together?
>>>>>
>>>>> What do you mean by "individual email notification instead of grouping
>>>>> them together"? Do you want some alerts to go to one receiver, and some
>>>>> alerts to go to a different receiver? If so, how do you want to choose
>>>>> which alerts go where?
>>>>>
>>>>> On Thursday, 2 January 2025 at 15:16:26 UTC Chrysale Tchako wrote:
>>>>>
>>>>>> This is my current alertmanager.yml config, but all notifications
>>>>>> sent out get sent to all emails added in the "to" section of the email
>>>>>> configs. What changes can I make so that each receiver gets it's own
>>>>>> individual email notification instead of grouping them together? Note
>>>>>> the
>>>>>> "from" email is the same for both receivers.
>>>>>>
>>>>>> global:
>>>>>> smtp_hello: '*******.net'
>>>>>>
>>>>>> route:
>>>>>> group_wait: 1m
>>>>>> group_interval: 5m
>>>>>> repeat_interval: 12h
>>>>>> group_by: ["alertname", "severity"]
>>>>>> # Default receiver (for all alerts that don't match any specific
>>>>>> route)
>>>>>> receiver: "gmail"
>>>>>>
>>>>>> routes:
>>>>>> - receiver: "gmail"
>>>>>> continue: true
>>>>>> - receiver: "dev-receiver"
>>>>>>
>>>>>> inhibit_rules:
>>>>>> - source_matchers:
>>>>>> - severity="critical"
>>>>>> target_matchers:
>>>>>> - severity="warning"
>>>>>> equal: ['alertname', 'instance']
>>>>>> routes:
>>>>>> - receiver: "gmail"
>>>>>> continue: true
>>>>>> - receiver: "dev-receiver"
>>>>>>
>>>>>> receivers:
>>>>>> # Default receiver for other alerts
>>>>>> - name: "gmail"
>>>>>> email_configs:
>>>>>> - to: "email1.net, email2.net"
>>>>>> from: '*******.net'
>>>>>> smarthost: 'smtp-relay.gmail.com:587'
>>>>>> send_resolved: true
>>>>>>
>>>>>> - name: "dev-receiver"
>>>>>> email_configs:
>>>>>> - to: 'email3.net'
>>>>>> from: '*******.net'
>>>>>> smarthost: 'smtp-relay.gmail.com:587'
>>>>>> send_resolved: true
>>>>>>
>>>>>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion visit
https://groups.google.com/d/msgid/prometheus-users/81f52b33-6e8a-4c0d-b528-cccb9086eedcn%40googlegroups.com.
