[prometheus-users] Re: change mail template on windows
Thank you very much Brian!
This will be my last post for now. I'm going to change company. I hope you
can help my collegues as good as you helped me!
This community is great, keep it up.
Brian Candler schrieb am Sonntag, 18. Juni 2023 um 19:04:36 UTC+2:
> That will never work. All labels (including severity) are things set in
> your alerting rules. You will never see a label with severity=resolved,
> unless you actually make an alerting rule which sets severity=resolved when
> it fires.
>
> (match_re does work though; it's the old syntax whereas "matchers" is the
> new and preferred syntax)
>
> Moral: never trust what ChatGPT tells you.
>
> Off the top of my head, I can't think of any way to send a message when
> alerts are resolved but *not* when alerts are firing.
>
> There are good reasons for not sending *any* resolved messages though:
>
> https://www.robustperception.io/running-into-burning-buildings-because-the-fire-alarm-stopped
>
> On Sunday, 18 June 2023 at 12:11:31 UTC Kolja Krückmann wrote:
>
>> Hi Brian, thanks here are the routes to the receivers:
>>
>> routes:
>> - receiver: email_critical
>> group_interval: 1m
>> group_wait: 5m
>> repeat_interval: 24h
>> group_by: ['alertname', 'severity']
>> matchers:
>> - severity="critical"
>> - severity!="resolved"
>>
>> - receiver: email_warning
>> group_interval: 10m
>> group_wait: 5m
>> repeat_interval: 24h
>> group_by: ['alertname', 'severity']
>> matchers:
>> - severity="warning"
>> - severity!="resolved"
>>
>> - receiver: email_resolved
>> match_re:
>> severity: ^(resolved)$
>>
>>
>> the match_re part from resolved is from good ol' friend chatgpt
>>
>> Brian Candler schrieb am Samstag, 17. Juni 2023 um 00:48:14 UTC+2:
>>
>>> You'll need to show the rest of your alertmanager config. "Receivers" by
>>> themselves don't do anything; you have to refer to them in routing rules.
>>>
>>> On Friday, 16 June 2023 at 09:23:56 UTC Kolja Krückmann wrote:
>>>
>>>> Or do I need to change it to explicitly say in the crit and warn mail
>>>> to range over the .Alerts.Firing in order to get the third receiver to
>>>> work?
>>>>
>>>> receivers:
>>>> - name: 'email'
>>>> email_configs:
>>>> - to: 'sysw...@xxx.com'
>>>> send_resolved: false
>>>> headers:
>>>> subject: 'Critical: {{ .GroupLabels.alertname }} - Instances: {{
>>>> range .Alerts.Firing }}{{ .Labels.instance }}, {{ end }}'
>>>>
>>>>
>>>> - name: 'email_warning'
>>>> email_configs:
>>>> - to: 'sysw...@xxx.com'
>>>> send_resolved: false
>>>> headers:
>>>> subject: 'Warning: {{ .GroupLabels.alertname }} - Instances: {{
>>>> range .Alerts.Firing }}{{ .Labels.instance }}, {{ end }}'
>>>>
>>>>
>>>> - name: 'email_resolved'
>>>> email_configs:
>>>> - to: 'sysw...@xxx.com'
>>>> send_resolved: true
>>>> headers:
>>>> subject: 'Resolved: {{ .GroupLabels.alertname }} - Instances: {{
>>>> range .Alerts.Resolved }}{{ .Labels.instance }}, {{ end }}'
>>>>
>>>> Kolja Krückmann schrieb am Freitag, 16. Juni 2023 um 11:21:59 UTC+2:
>>>>
>>>>> I changed the config to this:
>>>>>
>>>>> - name: 'email_resolved'
>>>>> email_configs:
>>>>> - to: 'sysw...@xxx.com'
>>>>> send_resolved: true
>>>>> headers:
>>>>> subject: 'Resolved: {{ .GroupLabels.alertname }} - Instances: {{
>>>>> range .Alerts.Resolved }}{{ .Labels.instance }}, {{ end }}'
>>>>>
>>>>> But this is sending absolutly nothing. What did I do wrong?
>>>>>
>>>>> Brian Candler schrieb am Freitag, 9. Juni 2023 um 15:57:04 UTC+2:
>>>>>
>>>>>> Maybe you want .Alerts.Firing and .Alerts.Resolved. A single alert
>>>>>> group is likely to contain a mixture of firing alerts and resolved
>>>>>> alerts.
>>>>>>
>>>>>> Also, note that your "send_resolved: true" receiver doesn't only send
>>>>>> resolved messages: it will send on firing *and* on resolved. Hence the
[prometheus-users] Re: change mail template on windows
Hi Brian, thanks here are the routes to the receivers:
routes:
- receiver: email_critical
group_interval: 1m
group_wait: 5m
repeat_interval: 24h
group_by: ['alertname', 'severity']
matchers:
- severity="critical"
- severity!="resolved"
- receiver: email_warning
group_interval: 10m
group_wait: 5m
repeat_interval: 24h
group_by: ['alertname', 'severity']
matchers:
- severity="warning"
- severity!="resolved"
- receiver: email_resolved
match_re:
severity: ^(resolved)$
the match_re part from resolved is from good ol' friend chatgpt
Brian Candler schrieb am Samstag, 17. Juni 2023 um 00:48:14 UTC+2:
> You'll need to show the rest of your alertmanager config. "Receivers" by
> themselves don't do anything; you have to refer to them in routing rules.
>
> On Friday, 16 June 2023 at 09:23:56 UTC Kolja Krückmann wrote:
>
>> Or do I need to change it to explicitly say in the crit and warn mail to
>> range over the .Alerts.Firing in order to get the third receiver to work?
>>
>> receivers:
>> - name: 'email'
>> email_configs:
>> - to: 'sysw...@xxx.com'
>> send_resolved: false
>> headers:
>> subject: 'Critical: {{ .GroupLabels.alertname }} - Instances: {{
>> range .Alerts.Firing }}{{ .Labels.instance }}, {{ end }}'
>>
>>
>> - name: 'email_warning'
>> email_configs:
>> - to: 'sysw...@xxx.com'
>> send_resolved: false
>> headers:
>> subject: 'Warning: {{ .GroupLabels.alertname }} - Instances: {{
>> range .Alerts.Firing }}{{ .Labels.instance }}, {{ end }}'
>>
>>
>> - name: 'email_resolved'
>> email_configs:
>> - to: 'sysw...@xxx.com'
>> send_resolved: true
>> headers:
>> subject: 'Resolved: {{ .GroupLabels.alertname }} - Instances: {{
>> range .Alerts.Resolved }}{{ .Labels.instance }}, {{ end }}'
>>
>> Kolja Krückmann schrieb am Freitag, 16. Juni 2023 um 11:21:59 UTC+2:
>>
>>> I changed the config to this:
>>>
>>> - name: 'email_resolved'
>>> email_configs:
>>> - to: 'sysw...@xxx.com'
>>> send_resolved: true
>>> headers:
>>> subject: 'Resolved: {{ .GroupLabels.alertname }} - Instances: {{
>>> range .Alerts.Resolved }}{{ .Labels.instance }}, {{ end }}'
>>>
>>> But this is sending absolutly nothing. What did I do wrong?
>>>
>>> Brian Candler schrieb am Freitag, 9. Juni 2023 um 15:57:04 UTC+2:
>>>
>>>> Maybe you want .Alerts.Firing and .Alerts.Resolved. A single alert
>>>> group is likely to contain a mixture of firing alerts and resolved alerts.
>>>>
>>>> Also, note that your "send_resolved: true" receiver doesn't only send
>>>> resolved messages: it will send on firing *and* on resolved. Hence the
>>>> subject "Resolved: ..." is likely to be very misleading.
>>>>
>>>> On Friday, 9 June 2023 at 13:42:13 UTC+1 Kolja Krückmann wrote:
>>>>
>>>>> Got it working like this:
>>>>>
>>>>> receivers:
>>>>> - name: 'email'
>>>>> email_configs:
>>>>> - to: 'sysw...@xxx.com'
>>>>> send_resolved: false
>>>>> headers:
>>>>> subject: 'Critical: {{ .GroupLabels.alertname }} - Instances: {{
>>>>> range .Alerts }}{{ .Labels.instance }}, {{ end }}'
>>>>>
>>>>> - name: 'email_warning'
>>>>> email_configs:
>>>>> - to: 'sysw...@xxx.com'
>>>>> send_resolved: false
>>>>> headers:
>>>>> subject: 'Warning: {{ .GroupLabels.alertname }} - Instances: {{
>>>>> range .Alerts }}{{ .Labels.instance }}, {{ end }}'
>>>>>
>>>>> - name: 'email_resolved'
>>>>> email_configs:
>>>>> - to: 'sysw...@xxx.com'
>>>>> send_resolved: true
>>>>> headers:
>>>>> subject: 'Resolved: {{ .GroupLabels.alertname }} - Instances: {{
>>>>> range .Alerts }}{{ .Labels.instance }}, {{ end }}'
>>>>>
>>>>> Kolja Krückmann schrieb am Donnerstag, 8. Juni 2023 um 15:03:14 UTC+2:
>>>>>
>>>>>> [EDIT]
>>>>>> Also I want to split the firing and resolved alerts.
>>>>>> Kolja Krückmann schrieb am Donnerstag, 8. Juni 2023 um 14:48:40 UTC+2:
>>>>>>
>>>>>>>
[prometheus-users] Re: change mail template on windows
Or do I need to change it to explicitly say in the crit and warn mail to
range over the .Alerts.Firing in order to get the third receiver to work?
receivers:
- name: 'email'
email_configs:
- to: 'syswa...@xxx.com'
send_resolved: false
headers:
subject: 'Critical: {{ .GroupLabels.alertname }} - Instances: {{
range .Alerts.Firing }}{{ .Labels.instance }}, {{ end }}'
- name: 'email_warning'
email_configs:
- to: 'syswa...@xxx.com'
send_resolved: false
headers:
subject: 'Warning: {{ .GroupLabels.alertname }} - Instances: {{ range
.Alerts.Firing }}{{ .Labels.instance }}, {{ end }}'
- name: 'email_resolved'
email_configs:
- to: 'syswa...@xxx.com'
send_resolved: true
headers:
subject: 'Resolved: {{ .GroupLabels.alertname }} - Instances: {{
range .Alerts.Resolved }}{{ .Labels.instance }}, {{ end }}'
Kolja Krückmann schrieb am Freitag, 16. Juni 2023 um 11:21:59 UTC+2:
> I changed the config to this:
>
> - name: 'email_resolved'
> email_configs:
> - to: 'sysw...@xxx.com'
> send_resolved: true
> headers:
> subject: 'Resolved: {{ .GroupLabels.alertname }} - Instances: {{
> range .Alerts.Resolved }}{{ .Labels.instance }}, {{ end }}'
>
> But this is sending absolutly nothing. What did I do wrong?
>
> Brian Candler schrieb am Freitag, 9. Juni 2023 um 15:57:04 UTC+2:
>
>> Maybe you want .Alerts.Firing and .Alerts.Resolved. A single alert group
>> is likely to contain a mixture of firing alerts and resolved alerts.
>>
>> Also, note that your "send_resolved: true" receiver doesn't only send
>> resolved messages: it will send on firing *and* on resolved. Hence the
>> subject "Resolved: ..." is likely to be very misleading.
>>
>> On Friday, 9 June 2023 at 13:42:13 UTC+1 Kolja Krückmann wrote:
>>
>>> Got it working like this:
>>>
>>> receivers:
>>> - name: 'email'
>>> email_configs:
>>> - to: 'sysw...@xxx.com'
>>> send_resolved: false
>>> headers:
>>> subject: 'Critical: {{ .GroupLabels.alertname }} - Instances: {{
>>> range .Alerts }}{{ .Labels.instance }}, {{ end }}'
>>>
>>> - name: 'email_warning'
>>> email_configs:
>>> - to: 'sysw...@xxx.com'
>>> send_resolved: false
>>> headers:
>>> subject: 'Warning: {{ .GroupLabels.alertname }} - Instances: {{
>>> range .Alerts }}{{ .Labels.instance }}, {{ end }}'
>>>
>>> - name: 'email_resolved'
>>> email_configs:
>>> - to: 'sysw...@xxx.com'
>>> send_resolved: true
>>> headers:
>>> subject: 'Resolved: {{ .GroupLabels.alertname }} - Instances: {{
>>> range .Alerts }}{{ .Labels.instance }}, {{ end }}'
>>>
>>> Kolja Krückmann schrieb am Donnerstag, 8. Juni 2023 um 15:03:14 UTC+2:
>>>
>>>> [EDIT]
>>>> Also I want to split the firing and resolved alerts.
>>>> Kolja Krückmann schrieb am Donnerstag, 8. Juni 2023 um 14:48:40 UTC+2:
>>>>
>>>>> Hi all, I tried it now for two days straight and I just don't get it
>>>>> working.
>>>>>
>>>>> First of all where do I actually find the expressions I can use to
>>>>> define the subject under the header? I am still missing a template I can
>>>>> use? I don't really have one in my Prom/Alert-files.
>>>>> Second of all my current receivers are configured like so:
>>>>>
>>>>> receivers:
>>>>> - name: 'email'
>>>>> email_configs:
>>>>> - to: 'x...@xxx.com'
>>>>> send_resolved: true
>>>>> headers:
>>>>> Subject: "{{ .Alerts.alertname }} - {{ range .Alerts }} {{
>>>>> .Alerts.instance }} {{ end }}"
>>>>>
>>>>> Like Brian said, I want to iterate in the second half of that
>>>>> "statement" (I don't actually know how these {{ .xyz }} are called)
>>>>> through
>>>>> all instances and the first one should just say which alert is now firing.
>>>>> I also tried it as followed:
>>>>>
>>>>> Subject: "{{ .GroupLabels.alertname }} - {{ range .Alerts }} {{
>>>>> .Alerts.instance }} {{ end }}"
>>>>> and
>>>>> Subject: "{{ .Labels.alertname }} - {{ range .Alerts }} {{
>>>>> .Alerts.instance }} {{ end }}"
>>>>>
>>>>> but nothing is really working.
>>>>>
>>>>> All I really wa
[prometheus-users] Re: change mail template on windows
I changed the config to this:
- name: 'email_resolved'
email_configs:
- to: 'syswa...@xxx.com'
send_resolved: true
headers:
subject: 'Resolved: {{ .GroupLabels.alertname }} - Instances: {{
range .Alerts.Resolved }}{{ .Labels.instance }}, {{ end }}'
But this is sending absolutly nothing. What did I do wrong?
Brian Candler schrieb am Freitag, 9. Juni 2023 um 15:57:04 UTC+2:
> Maybe you want .Alerts.Firing and .Alerts.Resolved. A single alert group
> is likely to contain a mixture of firing alerts and resolved alerts.
>
> Also, note that your "send_resolved: true" receiver doesn't only send
> resolved messages: it will send on firing *and* on resolved. Hence the
> subject "Resolved: ..." is likely to be very misleading.
>
> On Friday, 9 June 2023 at 13:42:13 UTC+1 Kolja Krückmann wrote:
>
>> Got it working like this:
>>
>> receivers:
>> - name: 'email'
>> email_configs:
>> - to: 'sysw...@xxx.com'
>> send_resolved: false
>> headers:
>> subject: 'Critical: {{ .GroupLabels.alertname }} - Instances: {{
>> range .Alerts }}{{ .Labels.instance }}, {{ end }}'
>>
>> - name: 'email_warning'
>> email_configs:
>> - to: 'sysw...@xxx.com'
>> send_resolved: false
>> headers:
>> subject: 'Warning: {{ .GroupLabels.alertname }} - Instances: {{
>> range .Alerts }}{{ .Labels.instance }}, {{ end }}'
>>
>> - name: 'email_resolved'
>> email_configs:
>> - to: 'sysw...@xxx.com'
>> send_resolved: true
>> headers:
>> subject: 'Resolved: {{ .GroupLabels.alertname }} - Instances: {{
>> range .Alerts }}{{ .Labels.instance }}, {{ end }}'
>>
>> Kolja Krückmann schrieb am Donnerstag, 8. Juni 2023 um 15:03:14 UTC+2:
>>
>>> [EDIT]
>>> Also I want to split the firing and resolved alerts.
>>> Kolja Krückmann schrieb am Donnerstag, 8. Juni 2023 um 14:48:40 UTC+2:
>>>
>>>> Hi all, I tried it now for two days straight and I just don't get it
>>>> working.
>>>>
>>>> First of all where do I actually find the expressions I can use to
>>>> define the subject under the header? I am still missing a template I can
>>>> use? I don't really have one in my Prom/Alert-files.
>>>> Second of all my current receivers are configured like so:
>>>>
>>>> receivers:
>>>> - name: 'email'
>>>> email_configs:
>>>> - to: 'x...@xxx.com'
>>>> send_resolved: true
>>>> headers:
>>>> Subject: "{{ .Alerts.alertname }} - {{ range .Alerts }} {{
>>>> .Alerts.instance }} {{ end }}"
>>>>
>>>> Like Brian said, I want to iterate in the second half of that
>>>> "statement" (I don't actually know how these {{ .xyz }} are called)
>>>> through
>>>> all instances and the first one should just say which alert is now firing.
>>>> I also tried it as followed:
>>>>
>>>> Subject: "{{ .GroupLabels.alertname }} - {{ range .Alerts }} {{
>>>> .Alerts.instance }} {{ end }}"
>>>> and
>>>> Subject: "{{ .Labels.alertname }} - {{ range .Alerts }} {{
>>>> .Alerts.instance }} {{ end }}"
>>>>
>>>> but nothing is really working.
>>>>
>>>> All I really want is the Subject like so e.x.:
>>>> InstanceDown - SVR-DS01 SVR-DC11
>>>>
>>>> Alertname- Instance1 Instance2 (if it is also possible to
>>>> seperate the instances via comma would be perfect)
>>>> Brian Candler schrieb am Dienstag, 6. Juni 2023 um 13:16:53 UTC+2:
>>>>
>>>>> As its name implies, "CommonLabels" contains only those labels which
>>>>> are common to all alerts in the group. If there are multiple instances,
>>>>> then the "instance" label is not common to all alerts, so won't be in
>>>>> this
>>>>> object.
>>>>>
>>>>> You want to iterate over "Alerts" (or "Alerts.Firing") and look at the
>>>>> Labels.instance within each alert. You will find an example showing how
>>>>> to
>>>>> do that here:
>>>>>
>>>>> https://prometheus.io/docs/alerting/latest/notification_examples/#ranging-over-all-received-alerts
>>>>>
>>>>> Additional references:
>>>>> https
[prometheus-users] Re: tls_config /tls_server_config
Hi Brian, thank you for clarifying the configs! I now got it working by creating my own .pfx via mmc.exe then extracting with openssl the key anf cert file. This is working great! Kind regards Kolja Brian Candler schrieb am Donnerstag, 15. Juni 2023 um 11:47:01 UTC+2: > If you want your website to have a *valid* certificate which web browsers > trust, then you shouldn't use openssl: you should use letsencrypt > <https://letsencrypt.org/>, using a client program such as certbot, > dehydrated, acme.sh. Letsencrypt has plenty of getting started guides, > plus its own forum <https://community.letsencrypt.org/> where you can ask > questions about this. > > This will create two files: a private key, and a certificate chain. You > would then refer to these using tls_server_config > <https://prometheus.io/docs/prometheus/latest/configuration/https/> in a > separate config file that you point to with --web.config.file. > > tls_config is used when prometheus itself is making *outbound* https > connections - for example when scraping an exporter or talking to an API > for service discovery - so isn't relevant here. > > On Thursday, 15 June 2023 at 10:06:39 UTC+1 Kolja Krückmann wrote: > >> [EDIT] >> >> My goal is to connect to prometheus-website (from a client not the host >> itself) via https, because I want to embed the website in a powerpoint. >> Unfortunatly the Add-in from ppt only allows https: websites >> Kolja Krückmann schrieb am Donnerstag, 15. Juni 2023 um 10:58:56 UTC+2: >> >>> Hi all, I want to run prometheus as https. >>> I searched through the whole doc and this google.group. Unfortunatly I >>> still don't understand how I use the certificates and how to exactly get >>> all them files (.key, .ca, .cert) I know so far, that I should use openssl >>> for creating said files but what values are needed, do I need both configs >>> (see subject) for opening the web dashboard of prometheus over https? >>> >>> I just don't find the right instruction for doing this. Maybe I am just >>> having a bad day and can't find anything... >>> >>> Kind regards, >>> Kolja >>> >> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/0e19e485-ac80-413e-b27a-4fef2a674575n%40googlegroups.com.
[prometheus-users] Re: tls_config /tls_server_config
[EDIT] My goal is to connect to prometheus-website (from a client not the host itself) via https, because I want to embed the website in a powerpoint. Unfortunatly the Add-in from ppt only allows https: websites Kolja Krückmann schrieb am Donnerstag, 15. Juni 2023 um 10:58:56 UTC+2: > Hi all, I want to run prometheus as https. > I searched through the whole doc and this google.group. Unfortunatly I > still don't understand how I use the certificates and how to exactly get > all them files (.key, .ca, .cert) I know so far, that I should use openssl > for creating said files but what values are needed, do I need both configs > (see subject) for opening the web dashboard of prometheus over https? > > I just don't find the right instruction for doing this. Maybe I am just > having a bad day and can't find anything... > > Kind regards, > Kolja > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/e3a41a91-45b9-4a94-8d5f-696fabd16b63n%40googlegroups.com.
[prometheus-users] tls_config /tls_server_config
Hi all, I want to run prometheus as https. I searched through the whole doc and this google.group. Unfortunatly I still don't understand how I use the certificates and how to exactly get all them files (.key, .ca, .cert) I know so far, that I should use openssl for creating said files but what values are needed, do I need both configs (see subject) for opening the web dashboard of prometheus over https? I just don't find the right instruction for doing this. Maybe I am just having a bad day and can't find anything... Kind regards, Kolja -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/d706ebd2-c91d-4e83-bc61-eb0169da8062n%40googlegroups.com.
[prometheus-users] Re: change mail template on windows
Got it working like this:
receivers:
- name: 'email'
email_configs:
- to: 'syswa...@xxx.com'
send_resolved: false
headers:
subject: 'Critical: {{ .GroupLabels.alertname }} - Instances: {{
range .Alerts }}{{ .Labels.instance }}, {{ end }}'
- name: 'email_warning'
email_configs:
- to: 'syswa...@xxx.com'
send_resolved: false
headers:
subject: 'Warning: {{ .GroupLabels.alertname }} - Instances: {{ range
.Alerts }}{{ .Labels.instance }}, {{ end }}'
- name: 'email_resolved'
email_configs:
- to: 'syswa...@xxx.com'
send_resolved: true
headers:
subject: 'Resolved: {{ .GroupLabels.alertname }} - Instances: {{
range .Alerts }}{{ .Labels.instance }}, {{ end }}'
Kolja Krückmann schrieb am Donnerstag, 8. Juni 2023 um 15:03:14 UTC+2:
> [EDIT]
> Also I want to split the firing and resolved alerts.
> Kolja Krückmann schrieb am Donnerstag, 8. Juni 2023 um 14:48:40 UTC+2:
>
>> Hi all, I tried it now for two days straight and I just don't get it
>> working.
>>
>> First of all where do I actually find the expressions I can use to define
>> the subject under the header? I am still missing a template I can use? I
>> don't really have one in my Prom/Alert-files.
>> Second of all my current receivers are configured like so:
>>
>> receivers:
>> - name: 'email'
>> email_configs:
>> - to: 'x...@xxx.com'
>> send_resolved: true
>> headers:
>> Subject: "{{ .Alerts.alertname }} - {{ range .Alerts }} {{
>> .Alerts.instance }} {{ end }}"
>>
>> Like Brian said, I want to iterate in the second half of that "statement"
>> (I don't actually know how these {{ .xyz }} are called) through all
>> instances and the first one should just say which alert is now firing.
>> I also tried it as followed:
>>
>> Subject: "{{ .GroupLabels.alertname }} - {{ range .Alerts }} {{
>> .Alerts.instance }} {{ end }}"
>> and
>> Subject: "{{ .Labels.alertname }} - {{ range .Alerts }} {{
>> .Alerts.instance }} {{ end }}"
>>
>> but nothing is really working.
>>
>> All I really want is the Subject like so e.x.:
>> InstanceDown - SVR-DS01 SVR-DC11
>>
>> Alertname- Instance1 Instance2 (if it is also possible to
>> seperate the instances via comma would be perfect)
>> Brian Candler schrieb am Dienstag, 6. Juni 2023 um 13:16:53 UTC+2:
>>
>>> As its name implies, "CommonLabels" contains only those labels which are
>>> common to all alerts in the group. If there are multiple instances, then
>>> the "instance" label is not common to all alerts, so won't be in this
>>> object.
>>>
>>> You want to iterate over "Alerts" (or "Alerts.Firing") and look at the
>>> Labels.instance within each alert. You will find an example showing how to
>>> do that here:
>>>
>>> https://prometheus.io/docs/alerting/latest/notification_examples/#ranging-over-all-received-alerts
>>>
>>> Additional references:
>>> https://prometheus.io/docs/alerting/latest/notifications/#data
>>>
>>> https://prometheus.io/docs/prometheus/latest/configuration/template_reference/
>>>
>>> On Tuesday, 6 June 2023 at 12:03:03 UTC+1 Kolja Krückmann wrote:
>>>
>>>> Currently my alertmanager.yml looks like this:
>>>>
>>>> receivers:
>>>> - name: 'email'
>>>> email_configs:
>>>> - to: 'x...@company.com'
>>>> send_resolved: true
>>>> headers:
>>>> Subject: '{{ .CommonLabels.alertname }} - {{
>>>> .CommonLabels.instance }}'
>>>>
>>>> Unfortunately the subject in the mail now only displays both labels if
>>>> only there is an alert for a single instance (like High CPU Usage on core
>>>> 0,1,2,3 on one instance (see screenshot attatched)) As soon as the alert
>>>> is
>>>> for more then just one instance the instance lable is completly missing...
>>>> Why is it missing? And how can I change it to display all instances
>>>> affected by that alert.
>>>>
>>>> Kind regards.
>>>> Kolja
>>>>
>>>> [image: mail.png]
>>>>
>>>> Kolja Krückmann schrieb am Montag, 5. Juni 2023 um 09:21:22 UTC+2:
>>>>
>>>> Hi y'all
>>>> I'm trying to change the way the mail from my alertmanger looks and
>>>> behaves.
>>>> My goal is to have the subject list the alertname and the targets
>>>> detected by that alert. (I need it like that for future todo's)
>>>> My first problem is, i dont have any templates in my files. Do I need
>>>> to clone one from git? How does the prom know, I have a template?
>>>> The second one would be to change the subject in order for my
>>>> requirements to fit.
>>>>
>>>> Kind regards,
>>>> Kolja
>>>>
>>>>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/ab55c173-a87a-4c10-94ef-8e7bfa9bb311n%40googlegroups.com.
[prometheus-users] Re: change mail template on windows
One Alert is now working like I want it to work. My YAML looks like this:
receivers:
- name: 'email'
email_configs:
- to: 'syswa...@xxx.com'
send_resolved: false
headers:
subject: '{{ .GroupLabels.alertname }} - Instances: {{ range $index,
$value := .GroupLabels.instance }}{{ if $index }}, {{ end }}{{ $value }}{{
end }}'
- name: 'email_warning'
email_configs:
- to: 'syswa...@xxx.com'
send_resolved: false
headers:
subject: 'Warning: {{ .GroupLabels.alertname }} - Instances: {{ range
.Alerts }}{{ .Labels.instance }}, {{ end }}'
- name: 'email_resolved'
email_configs:
- to: 'syswa...@xxx.com'
send_resolved: true
headers:
subject: 'Resolved: {{ .GroupLabels.alertname }} - Instances: {{
range $index, $value := .GroupLabels.instance }}{{ if $index }}, {{ end
}}{{ $value }}{{ end }}'
But my alertmanager is giving me still errors:
ts=2023-06-09T12:24:29.325Z caller=dispatch.go:352 level=error
component=dispatcher msg="Notify for alerts failed" num_alerts=2
err="email/email[0]: notify retry canceled due to unrecoverable error after
1 attempts: execute \"Subject\" header template: template: :1:81: executing
\"\" at <.GroupLabels.instance>: range can't iterate over "
What I do not understand is, that it's working for one rule but not the
others. And the error implies that it can't iterate over the instances. I
don't understand why because for one rule it's working as I said.
Can someone help me fix my issue?
Kind regards,
Kolja
Kolja Krückmann schrieb am Donnerstag, 8. Juni 2023 um 15:03:14 UTC+2:
> [EDIT]
> Also I want to split the firing and resolved alerts.
> Kolja Krückmann schrieb am Donnerstag, 8. Juni 2023 um 14:48:40 UTC+2:
>
>> Hi all, I tried it now for two days straight and I just don't get it
>> working.
>>
>> First of all where do I actually find the expressions I can use to define
>> the subject under the header? I am still missing a template I can use? I
>> don't really have one in my Prom/Alert-files.
>> Second of all my current receivers are configured like so:
>>
>> receivers:
>> - name: 'email'
>> email_configs:
>> - to: 'x...@xxx.com'
>> send_resolved: true
>> headers:
>> Subject: "{{ .Alerts.alertname }} - {{ range .Alerts }} {{
>> .Alerts.instance }} {{ end }}"
>>
>> Like Brian said, I want to iterate in the second half of that "statement"
>> (I don't actually know how these {{ .xyz }} are called) through all
>> instances and the first one should just say which alert is now firing.
>> I also tried it as followed:
>>
>> Subject: "{{ .GroupLabels.alertname }} - {{ range .Alerts }} {{
>> .Alerts.instance }} {{ end }}"
>> and
>> Subject: "{{ .Labels.alertname }} - {{ range .Alerts }} {{
>> .Alerts.instance }} {{ end }}"
>>
>> but nothing is really working.
>>
>> All I really want is the Subject like so e.x.:
>> InstanceDown - SVR-DS01 SVR-DC11
>>
>> Alertname- Instance1 Instance2 (if it is also possible to
>> seperate the instances via comma would be perfect)
>> Brian Candler schrieb am Dienstag, 6. Juni 2023 um 13:16:53 UTC+2:
>>
>>> As its name implies, "CommonLabels" contains only those labels which are
>>> common to all alerts in the group. If there are multiple instances, then
>>> the "instance" label is not common to all alerts, so won't be in this
>>> object.
>>>
>>> You want to iterate over "Alerts" (or "Alerts.Firing") and look at the
>>> Labels.instance within each alert. You will find an example showing how to
>>> do that here:
>>>
>>> https://prometheus.io/docs/alerting/latest/notification_examples/#ranging-over-all-received-alerts
>>>
>>> Additional references:
>>> https://prometheus.io/docs/alerting/latest/notifications/#data
>>>
>>> https://prometheus.io/docs/prometheus/latest/configuration/template_reference/
>>>
>>> On Tuesday, 6 June 2023 at 12:03:03 UTC+1 Kolja Krückmann wrote:
>>>
>>>> Currently my alertmanager.yml looks like this:
>>>>
>>>> receivers:
>>>> - name: 'email'
>>>> email_configs:
>>>> - to: 'x...@company.com'
>>>> send_resolved: true
>>>> headers:
>>>> Subject: '{{ .CommonLabels.alertname }} - {{
>>>> .CommonLabels.instance }}'
>>>>
>>>> Unfortunately the subject in the mail now only displays both labels if
>
[prometheus-users] Re: change mail template on windows
[EDIT]
Also I want to split the firing and resolved alerts.
Kolja Krückmann schrieb am Donnerstag, 8. Juni 2023 um 14:48:40 UTC+2:
> Hi all, I tried it now for two days straight and I just don't get it
> working.
>
> First of all where do I actually find the expressions I can use to define
> the subject under the header? I am still missing a template I can use? I
> don't really have one in my Prom/Alert-files.
> Second of all my current receivers are configured like so:
>
> receivers:
> - name: 'email'
> email_configs:
> - to: 'x...@xxx.com'
> send_resolved: true
> headers:
> Subject: "{{ .Alerts.alertname }} - {{ range .Alerts }} {{
> .Alerts.instance }} {{ end }}"
>
> Like Brian said, I want to iterate in the second half of that "statement"
> (I don't actually know how these {{ .xyz }} are called) through all
> instances and the first one should just say which alert is now firing.
> I also tried it as followed:
>
> Subject: "{{ .GroupLabels.alertname }} - {{ range .Alerts }} {{
> .Alerts.instance }} {{ end }}"
> and
> Subject: "{{ .Labels.alertname }} - {{ range .Alerts }} {{
> .Alerts.instance }} {{ end }}"
>
> but nothing is really working.
>
> All I really want is the Subject like so e.x.:
> InstanceDown - SVR-DS01 SVR-DC11
>
> Alertname- Instance1 Instance2 (if it is also possible to
> seperate the instances via comma would be perfect)
> Brian Candler schrieb am Dienstag, 6. Juni 2023 um 13:16:53 UTC+2:
>
>> As its name implies, "CommonLabels" contains only those labels which are
>> common to all alerts in the group. If there are multiple instances, then
>> the "instance" label is not common to all alerts, so won't be in this
>> object.
>>
>> You want to iterate over "Alerts" (or "Alerts.Firing") and look at the
>> Labels.instance within each alert. You will find an example showing how to
>> do that here:
>>
>> https://prometheus.io/docs/alerting/latest/notification_examples/#ranging-over-all-received-alerts
>>
>> Additional references:
>> https://prometheus.io/docs/alerting/latest/notifications/#data
>>
>> https://prometheus.io/docs/prometheus/latest/configuration/template_reference/
>>
>> On Tuesday, 6 June 2023 at 12:03:03 UTC+1 Kolja Krückmann wrote:
>>
>>> Currently my alertmanager.yml looks like this:
>>>
>>> receivers:
>>> - name: 'email'
>>> email_configs:
>>> - to: 'x...@company.com'
>>> send_resolved: true
>>> headers:
>>> Subject: '{{ .CommonLabels.alertname }} - {{
>>> .CommonLabels.instance }}'
>>>
>>> Unfortunately the subject in the mail now only displays both labels if
>>> only there is an alert for a single instance (like High CPU Usage on core
>>> 0,1,2,3 on one instance (see screenshot attatched)) As soon as the alert is
>>> for more then just one instance the instance lable is completly missing...
>>> Why is it missing? And how can I change it to display all instances
>>> affected by that alert.
>>>
>>> Kind regards.
>>> Kolja
>>>
>>> [image: mail.png]
>>>
>>> Kolja Krückmann schrieb am Montag, 5. Juni 2023 um 09:21:22 UTC+2:
>>>
>>> Hi y'all
>>> I'm trying to change the way the mail from my alertmanger looks and
>>> behaves.
>>> My goal is to have the subject list the alertname and the targets
>>> detected by that alert. (I need it like that for future todo's)
>>> My first problem is, i dont have any templates in my files. Do I need to
>>> clone one from git? How does the prom know, I have a template?
>>> The second one would be to change the subject in order for my
>>> requirements to fit.
>>>
>>> Kind regards,
>>> Kolja
>>>
>>>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/ba1cea79-4c12-4a94-9814-7e739556bd25n%40googlegroups.com.
[prometheus-users] Re: change mail template on windows
Hi all, I tried it now for two days straight and I just don't get it
working.
First of all where do I actually find the expressions I can use to define
the subject under the header? I am still missing a template I can use? I
don't really have one in my Prom/Alert-files.
Second of all my current receivers are configured like so:
receivers:
- name: 'email'
email_configs:
- to: 'x...@xxx.com'
send_resolved: true
headers:
Subject: "{{ .Alerts.alertname }} - {{ range .Alerts }} {{
.Alerts.instance }} {{ end }}"
Like Brian said, I want to iterate in the second half of that "statement"
(I don't actually know how these {{ .xyz }} are called) through all
instances and the first one should just say which alert is now firing.
I also tried it as followed:
Subject: "{{ .GroupLabels.alertname }} - {{ range .Alerts }} {{
.Alerts.instance }} {{ end }}"
and
Subject: "{{ .Labels.alertname }} - {{ range .Alerts }} {{ .Alerts.instance
}} {{ end }}"
but nothing is really working.
All I really want is the Subject like so e.x.:
InstanceDown - SVR-DS01 SVR-DC11
Alertname- Instance1 Instance2 (if it is also possible to
seperate the instances via comma would be perfect)
Brian Candler schrieb am Dienstag, 6. Juni 2023 um 13:16:53 UTC+2:
> As its name implies, "CommonLabels" contains only those labels which are
> common to all alerts in the group. If there are multiple instances, then
> the "instance" label is not common to all alerts, so won't be in this
> object.
>
> You want to iterate over "Alerts" (or "Alerts.Firing") and look at the
> Labels.instance within each alert. You will find an example showing how to
> do that here:
>
> https://prometheus.io/docs/alerting/latest/notification_examples/#ranging-over-all-received-alerts
>
> Additional references:
> https://prometheus.io/docs/alerting/latest/notifications/#data
>
> https://prometheus.io/docs/prometheus/latest/configuration/template_reference/
>
> On Tuesday, 6 June 2023 at 12:03:03 UTC+1 Kolja Krückmann wrote:
>
>> Currently my alertmanager.yml looks like this:
>>
>> receivers:
>> - name: 'email'
>> email_configs:
>> - to: 'x...@company.com'
>> send_resolved: true
>> headers:
>> Subject: '{{ .CommonLabels.alertname }} - {{ .CommonLabels.instance
>> }}'
>>
>> Unfortunately the subject in the mail now only displays both labels if
>> only there is an alert for a single instance (like High CPU Usage on core
>> 0,1,2,3 on one instance (see screenshot attatched)) As soon as the alert is
>> for more then just one instance the instance lable is completly missing...
>> Why is it missing? And how can I change it to display all instances
>> affected by that alert.
>>
>> Kind regards.
>> Kolja
>>
>> [image: mail.png]
>>
>> Kolja Krückmann schrieb am Montag, 5. Juni 2023 um 09:21:22 UTC+2:
>>
>> Hi y'all
>> I'm trying to change the way the mail from my alertmanger looks and
>> behaves.
>> My goal is to have the subject list the alertname and the targets
>> detected by that alert. (I need it like that for future todo's)
>> My first problem is, i dont have any templates in my files. Do I need to
>> clone one from git? How does the prom know, I have a template?
>> The second one would be to change the subject in order for my
>> requirements to fit.
>>
>> Kind regards,
>> Kolja
>>
>>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/0402c9f4-eac8-4e85-b8d1-1500447f0749n%40googlegroups.com.
[prometheus-users] Re: change mail template on windows
Currently my alertmanager.yml looks like this:
receivers:
- name: 'email'
email_configs:
- to: 'x...@company.com'
send_resolved: true
headers:
Subject: '{{ .CommonLabels.alertname }} - {{ .CommonLabels.instance
}}'
Unfortunately the subject in the mail now only displays both labels if only
there is an alert for a single instance (like High CPU Usage on core
0,1,2,3 on one instance (see screenshot attatched)) As soon as the alert is
for more then just one instance the instance lable is completly missing...
Why is it missing? And how can I change it to display all instances
affected by that alert.
Kind regards.
Kolja
[image: mail.png]
Kolja Krückmann schrieb am Montag, 5. Juni 2023 um 09:21:22 UTC+2:
Hi y'all
I'm trying to change the way the mail from my alertmanger looks and behaves.
My goal is to have the subject list the alertname and the targets detected
by that alert. (I need it like that for future todo's)
My first problem is, i dont have any templates in my files. Do I need to
clone one from git? How does the prom know, I have a template?
The second one would be to change the subject in order for my requirements
to fit.
Kind regards,
Kolja
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/f21707a7-b394-44c4-b98b-a6e4a5084c50n%40googlegroups.com.
[prometheus-users] change mail template on windows
Hi y'all I'm trying to change the way the mail from my alertmanger looks and behaves. My goal is to have the subject list the alertname and the targets detected by that alert. (I need it like that for future todo's) My first problem is, i dont have any templates in my files. Do I need to clone one from git? How does the prom know, I have a template? The second one would be to change the subject in order for my requirements to fit. Kind regards, Kolja -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/d70b8b82-9226-4f62-9d03-fd08f868b6d5n%40googlegroups.com.
[prometheus-users] Re: Alerting with result of expression
Thanks alot!
My question now is, that my current expressions only returns the percentage
as value - can I just add another expression for that specific alert?
Currentliy my alerts for diskSpace (one at 10% as warning and one at 5% as
crit):
- alert: LowDiskSpace
expr: ((windows_logical_disk_free_bytes{volume="C:"} /
windows_logical_disk_size_bytes) * 100) <= 10
for: 1m
labels:
severity: warning
annotations:
summary: "10% C-Disk Space"
description: "Das C-Volumen des Hosts {{ $labels.instance }} ist noch
10% frei *({{$value | humanize}})*."
- alert: LowDiskSpace
expr: ((windows_logical_disk_free_bytes{volume="C:"} /
windows_logical_disk_size_bytes) * 100) <= 5
for: 1m
labels:
severity: critical
annotations:
summary: "5% C-Disk Space"
description: "Das C-Volumen des Hosts {{ $labels.instance }} ist noch
5% frei *({{$value | humanize}})*."
I want the values (bold above) to get returned as gigabyte values -> so can
I just add an expression which calculates the free space?
Brian Candler schrieb am Mittwoch, 10. Mai 2023 um 13:36:13 UTC+2:
> The "result" of the expression is available as {{ $value }}, and there are
> functions to convert this into a more human-readable value. See
> https://prometheus.io/docs/prometheus/latest/configuration/template_reference/
>
> Examples:
>
> expr: windows_logical_disk_free_bytes{volume="C:"} /
> windows_logical_disk_size_bytes
> annotations:
> description: "Low free disk space: {{ $value | humanizePercentage }}"
>
> expr: windows_logical_disk_free_bytes{volume="C:"} < 10
> annotations:
> description: "Low free disk space: {{ $value | humanize }}"# or
> humanize1024: depends if you want Gigabytes or Gibibytes.
> https://en.wikipedia.org/wiki/Gigabyte
>
> If you want to do the threshold based on percent, but report the absolute
> value, I would use something like this (untested):
>
> expr: windows_logical_disk_free_bytes and
> (windows_logical_disk_free_bytes{volume="C:"} /
> windows_logical_disk_size_bytes < 0.1)
> annotations:
> description: "Low free disk space: {{ $value | humanize }}"
>
> I believe it's also possible to embed a completely separate query in a
> template (to look up a separate value to include in the annotations), but
> I've never done it, and can't find any examples.
>
> Aside: I find these sort of static alerts annoying. Sometimes a filesystem
> has 8% disk free space and that's a good and normal situation for it to be
> in. Therefore, either you're lost in a sea of unimportant repeating alerts,
> or you're jumping through hoops for setting separate static thresholds per
> filesystem.
> https://groups.google.com/g/prometheus-users/c/wHLxUPtrb-A/m/idIcdJIrBgAJ
>
> Another approach you could consider:
> https://groups.google.com/g/prometheus-users/c/0ncUqLm0LhU/m/mAiwaADXAgAJ
> This looks at how quickly the filesystem is filling up, and tells you how
> long before it expects to be full.
>
> On Wednesday, 10 May 2023 at 09:23:27 UTC+1 Kolja Krückmann wrote:
>
>> Small correction here:
>>
>> I want to have the expression
>> "windows_logical_disk_free_bytes{volume="C:"}/1000/1000/1000" (if this is
>> the actual GB of free disk space (or do I need to device by 1024?)) in my
>> alerting mail. And not as above the percentage.
>>
>> Kolja Krückmann schrieb am Mittwoch, 10. Mai 2023 um 10:20:53 UTC+2:
>>
>>> Hi y'all
>>>
>>> I'm looking for a possibility to add the "result" of an expression to
>>> the alerting description.
>>> My expression is to alert when the c:\ Drive is below 10%. Now I want to
>>> add the actual value of the expression:
>>> ((windows_logical_disk_free_bytes{volume="C:"} /
>>> windows_logical_disk_size_bytes) * 100) <= 10
>>> in the alerting mail. Is this somehow possible so that the free size is
>>> within the mail?
>>>
>>> Kind regards
>>>
>>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/13c4a382-a78c-4492-93cb-1472c690679bn%40googlegroups.com.
[prometheus-users] Re: Alerting with result of expression
Small correction here:
I want to have the expression
"windows_logical_disk_free_bytes{volume="C:"}/1000/1000/1000" (if this is
the actual GB of free disk space (or do I need to device by 1024?)) in my
alerting mail. And not as above the percentage.
Kolja Krückmann schrieb am Mittwoch, 10. Mai 2023 um 10:20:53 UTC+2:
> Hi y'all
>
> I'm looking for a possibility to add the "result" of an expression to the
> alerting description.
> My expression is to alert when the c:\ Drive is below 10%. Now I want to
> add the actual value of the expression:
> ((windows_logical_disk_free_bytes{volume="C:"} /
> windows_logical_disk_size_bytes) * 100) <= 10
> in the alerting mail. Is this somehow possible so that the free size is
> within the mail?
>
> Kind regards
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/6ab7cd37-5dc3-4066-9a17-545351b2afeen%40googlegroups.com.
[prometheus-users] Alerting with result of expression
Hi y'all
I'm looking for a possibility to add the "result" of an expression to the
alerting description.
My expression is to alert when the c:\ Drive is below 10%. Now I want to
add the actual value of the expression:
((windows_logical_disk_free_bytes{volume="C:"} /
windows_logical_disk_size_bytes) * 100) <= 10
in the alerting mail. Is this somehow possible so that the free size is
within the mail?
Kind regards
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/97ef-b266-4c37-ba02-4a9c1d57ed9bn%40googlegroups.com.
Re: [prometheus-users] file_cd_config - label for each target
Thanks for the hint with the wildcard! What I don't yet understand is what you mean by "it is set from the "module" label in the targets file." My target files are built like: - targets: - "name" "ip-adress" - "name" "ip-adress" - "name" "ip-adress" where does the label actually come from? Brian Candler schrieb am Freitag, 28. April 2023 um 14:55:10 UTC+2: > On Friday, 28 April 2023 at 13:13:22 UTC+1 Kolja Krückmann wrote: > > > - job_name: 'Ping (ICMP)' > metrics_path: /probe > params: > module: [icmp_ttl5] > > > Incidentally, you don't need to specify that here, because with the > rewriting rules given, it is set from the "module" label in the targets > file. (However, it will act as a default in case you don't specify it there) > > > > file_sd_configs: > - files: > - C:\Prometheus\targets\hosts.yml > - C:\Prometheus\targets\netcomponents.yml > - C:\Prometheus\targets\vmhost-05.yml > - C:\Prometheus\targets\vmhost-06.yml > - C:\Prometheus\targets\vmhost-08.yml > - C:\Prometheus\targets\vmhost-09.yml > - C:\Prometheus\targets\vmhost-10.yml > - C:\Prometheus\targets\vmopheo-01.yml > - C:\Prometheus\targets\vmopheo-02.yml > - C:\Prometheus\targets\vmopheo-04.yml > - C:\Prometheus\targets\vmopheo-06.yml > - C:\Prometheus\targets\vmopheo-07.yml > - C:\Prometheus\targets\vmopheo-08.yml > > > That's fine but it means you've hard-coded a list of files in your > prometheus.yml. You should be able to simplify it to this (works under > Linux anyway): > > file_sd_configs: > - files: > - C:\Prometheus\targets\*.yml > > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/1b9bc9d6-5eea-48b1-bd5b-defface67f6fn%40googlegroups.com.
Re: [prometheus-users] file_cd_config - label for each target
For context and maybe upcomming errors here is my prom.yml so far:
# my global config
global:
scrape_interval: 1m # Set the scrape interval to every 15 seconds.
Default is every 1 minute.
evaluation_interval: 30s # Evaluate rules every 15 seconds. The default
is every 1 minute.
scrape_timeout: 30s
# scrape_timeout is set to the global default (10s).
# Alertmanager configuration
alerting:
alertmanagers:
- static_configs:
- targets:
- localhost:9093
# Load rules once and periodically evaluate them according to the global
'evaluation_interval'.
rule_files:
- rules.yml
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=` to any timeseries
scraped from this config.
- job_name: 'Windows-Exporter Host-Server'
file_sd_configs:
- files:
- C:\Prometheus\targets\hosts.yml
relabel_configs:
# When __address__ is of the form "name address", extract
# name to "instance" label and address to "__address__"
- source_labels: [__address__]
regex: '(.+) (.+)'
target_label: instance
replacement: '${1}'
- source_labels: [__address__]
regex: '(.+) (.+)'
target_label: __address__
replacement: '${2}'
[...]
- job_name: 'Ping (ICMP)'
metrics_path: /probe
params:
module: [icmp_ttl5]
file_sd_configs:
- files:
- C:\Prometheus\targets\hosts.yml
- C:\Prometheus\targets\netcomponents.yml
- C:\Prometheus\targets\vmhost-05.yml
- C:\Prometheus\targets\vmhost-06.yml
- C:\Prometheus\targets\vmhost-08.yml
- C:\Prometheus\targets\vmhost-09.yml
- C:\Prometheus\targets\vmhost-10.yml
- C:\Prometheus\targets\vmopheo-01.yml
- C:\Prometheus\targets\vmopheo-02.yml
- C:\Prometheus\targets\vmopheo-04.yml
- C:\Prometheus\targets\vmopheo-06.yml
- C:\Prometheus\targets\vmopheo-07.yml
- C:\Prometheus\targets\vmopheo-08.yml
relabel_configs:
- source_labels: [__address__]
regex: '([^ ]+)'# name or address only
target_label: instance
- source_labels: [__address__]
regex: '([^ ]+)'# name or address only
target_label: __param_target
- source_labels: [__address__]
regex: '(.+) (.+)' # name address
target_label: instance
replacement: '${1}'
- source_labels: [__address__]
regex: '(.+) (.+)' # name address
target_label: __param_target
replacement: '${2}'
- source_labels: [module]
target_label: __param_module
- target_label: __address__
replacement: 127.0.0.1:9115 # Blackbox exporter
Kolja Krückmann schrieb am Freitag, 28. April 2023 um 14:10:00 UTC+2:
> Hi Brian, thanks for your quick response.
>
> Actually I just pasted the relabeling you attached and it immediately
> worked :)
>
> Brian Candler schrieb am Freitag, 28. April 2023 um 13:37:33 UTC+2:
>
>> Do you want to show what you've done so far?
>>
>> You can read these:
>>
>>
>> https://nsrc.org/workshops/2022/rwnog/nmm/netmgmt/en/prometheus/ex-blackbox-exporter.html
>>
>> # simple config
>>
>> https://nsrc.org/workshops/2022/rwnog/nmm/netmgmt/en/prometheus/ex-relabeling.html
>>
>> # node_exporter with relabelling
>>
>> If you combine the two, you could end up with something like this:
>>
>> - job_name: blackbox
>> file_sd_configs:
>> - files:
>> - /etc/prometheus/blackbox.d/*.yml
>> metrics_path: /probe
>> relabel_configs:
>> - source_labels: [__address__]
>> regex: '([^ ]+)'# name or address only
>> target_label: instance
>> - source_labels: [__address__]
>> regex: '([^ ]+)'# name or address only
>> target_label: __param_target
>> - source_labels: [__address__]
>> regex: '(.+) (.+)' # name address
>>
>> target_label: instance
>> replacement: '${1}'
>> - source_labels: [__address__]
>> regex: '(.+) (.+)' # name address
>> target_label: __param_target
>> replacement: '${2}'
>> - source_labels: [module]
>> target_label: __param_module
>> - target_label: __address__
>> replacement: 127.0.0.1:9115 # Blackbox exporter
>>
>> Where /etc/prometheus/blackbox.d/ping.yml contains, for example:
>>
>> - labels:
>> module: icmp
>> targets:
>> - google-pri 8.8.8.8
>> - google-sec 8.8.4.4
>>
>> Then you should get metrics like this:
>>
&g
Re: [prometheus-users] file_cd_config - label for each target
Hi Brian, thanks for your quick response.
Actually I just pasted the relabeling you attached and it immediately
worked :)
Brian Candler schrieb am Freitag, 28. April 2023 um 13:37:33 UTC+2:
> Do you want to show what you've done so far?
>
> You can read these:
>
>
> https://nsrc.org/workshops/2022/rwnog/nmm/netmgmt/en/prometheus/ex-blackbox-exporter.html
>
> # simple config
>
> https://nsrc.org/workshops/2022/rwnog/nmm/netmgmt/en/prometheus/ex-relabeling.html
>
> # node_exporter with relabelling
>
> If you combine the two, you could end up with something like this:
>
> - job_name: blackbox
> file_sd_configs:
> - files:
> - /etc/prometheus/blackbox.d/*.yml
> metrics_path: /probe
> relabel_configs:
> - source_labels: [__address__]
> regex: '([^ ]+)'# name or address only
> target_label: instance
> - source_labels: [__address__]
> regex: '([^ ]+)'# name or address only
> target_label: __param_target
> - source_labels: [__address__]
> regex: '(.+) (.+)' # name address
>
> target_label: instance
> replacement: '${1}'
> - source_labels: [__address__]
> regex: '(.+) (.+)' # name address
> target_label: __param_target
> replacement: '${2}'
> - source_labels: [module]
> target_label: __param_module
> - target_label: __address__
> replacement: 127.0.0.1:9115 # Blackbox exporter
>
> Where /etc/prometheus/blackbox.d/ping.yml contains, for example:
>
> - labels:
> module: icmp
> targets:
> - google-pri 8.8.8.8
> - google-sec 8.8.4.4
>
> Then you should get metrics like this:
>
> probe_success{job="blackbox",instance="google-pri",module="icmp"} 1
>
> On Friday, 28 April 2023 at 09:49:21 UTC+1 Kolja Krückmann wrote:
>
>> Hi y'all, now i got my metrics running (Thanks Brian)
>>
>> For the next step I want to use the Blackbox_exporter with the same
>> targets (where the target.yml is build like "- targetname targetIP")
>> I want to have the same labeling as in the question above where each
>> target has a name lable but I'm not quite getting there. Either the
>> Endpoint URL is missing the target IP or the label is missing the name of
>> the endpoint.
>>
>> Kolja Krückmann schrieb am Freitag, 28. April 2023 um 08:38:59 UTC+2:
>>
>>> nevermind - sorry for the question, should have just googled learn
>>> regular expression.
>>> I'm fine for now.
>>> Kolja Krückmann schrieb am Freitag, 28. April 2023 um 08:35:21 UTC+2:
>>>
>>>> Thanks so much!
>>>>
>>>> Can you tell me where I could kinda "learn" regex? I find it very
>>>> difficult to get known to regex because it's nothing im using on daily
>>>> basis ^^
>>>>
>>>>
>>>>
>>>> Brian Candler schrieb am Donnerstag, 27. April 2023 um 17:24:26 UTC+2:
>>>>
>>>>> So to be clear, if you want your targets file to look like this:
>>>>>
>>>>> - targets:
>>>>> - SVR-DS01 172.25.X0.XXX:9182
>>>>> - SRV-DS02 172.21.X1.XXX:9182
>>>>>
>>>>> you can do something like this:
>>>>>
>>>>>relabel_configs:
>>>>> # When __address__ is a single item, set the instance
>>>>> # label to the part without the port
>>>>> - source_labels: [__address__]
>>>>> regex: '([^ ]+):[0-9]+'
>>>>> replacement: '${1}'
>>>>> target_label: instance
>>>>>
>>>>> # When __address__ is space-separated "name address:port",
>>>>> # put the first part in the "instance" label and leave the
>>>>> second part
>>>>> # in "__address__"
>>>>> - source_labels: [__address__]
>>>>> regex: '(.+) (.+)'
>>>>> target_label: instance
>>>>> replacement: '${1}'
>>>>> - source_labels: [__address__]
>>>>> regex: '(.+) (.+)'
>>>>> target_label: __address__
>>>>> replacement: '${2}'
>>>>>
>>>>> If your target entries *always* consist of two items separated by a
>>>>> space, then you don't need the first rule. It's only there in case you
>>>>> have
>>>>> ent
Re: [prometheus-users] file_cd_config - label for each target
Hi y'all, now i got my metrics running (Thanks Brian)
For the next step I want to use the Blackbox_exporter with the same targets
(where the target.yml is build like "- targetname targetIP")
I want to have the same labeling as in the question above where each target
has a name lable but I'm not quite getting there. Either the Endpoint URL
is missing the target IP or the label is missing the name of the endpoint.
Kolja Krückmann schrieb am Freitag, 28. April 2023 um 08:38:59 UTC+2:
> nevermind - sorry for the question, should have just googled learn regular
> expression.
> I'm fine for now.
> Kolja Krückmann schrieb am Freitag, 28. April 2023 um 08:35:21 UTC+2:
>
>> Thanks so much!
>>
>> Can you tell me where I could kinda "learn" regex? I find it very
>> difficult to get known to regex because it's nothing im using on daily
>> basis ^^
>>
>>
>>
>> Brian Candler schrieb am Donnerstag, 27. April 2023 um 17:24:26 UTC+2:
>>
>>> So to be clear, if you want your targets file to look like this:
>>>
>>> - targets:
>>> - SVR-DS01 172.25.X0.XXX:9182
>>> - SRV-DS02 172.21.X1.XXX:9182
>>>
>>> you can do something like this:
>>>
>>>relabel_configs:
>>> # When __address__ is a single item, set the instance
>>> # label to the part without the port
>>> - source_labels: [__address__]
>>> regex: '([^ ]+):[0-9]+'
>>> replacement: '${1}'
>>> target_label: instance
>>>
>>> # When __address__ is space-separated "name address:port",
>>> # put the first part in the "instance" label and leave the second
>>> part
>>> # in "__address__"
>>> - source_labels: [__address__]
>>> regex: '(.+) (.+)'
>>> target_label: instance
>>> replacement: '${1}'
>>> - source_labels: [__address__]
>>> regex: '(.+) (.+)'
>>> target_label: __address__
>>> replacement: '${2}'
>>>
>>> If your target entries *always* consist of two items separated by a
>>> space, then you don't need the first rule. It's only there in case you have
>>> entries in the old format, i.e.
>>>
>>> - targets:
>>> - 172.25.X0.XXX:9182
>>>
>>> (in which case, the instance is set to "172.25.X0.XXX")
>>>
>>> On Thursday, 27 April 2023 at 16:16:38 UTC+1 Brian Candler wrote:
>>>
>>>> Well, there is a good reason to keep the port out of the "instance"
>>>> label: it's very awkward to make a query to join two different different
>>>> metrics collected from two different exporters on the same host, if one
>>>> has
>>>> (say) instance="foo:9100" and another has instance="foo:9104". A second
>>>> reason is that it's nicer in dashboards to see "foo" rather than "foo:9100"
>>>>
>>>> See https://www.robustperception.io/controlling-the-instance-label
>>>>
>>>> IMO the cleanest way to achieve this is to leave the port out of the
>>>> targets file, copy __address__ to instance, and then append the exporter
>>>> port statically to __address__.
>>>>
>>>> However, you could leave the port number in the targets file, and strip
>>>> it out to create the instance label:
>>>>
>>>> - source_labels: [__address__]
>>>> regex: '(.*):[0-9]+'
>>>> target_label: instance
>>>> replacement: '${1}'
>>>>
>>>> This would be useful if the same exporter were running on different
>>>> ports on different hosts (fairly uncommon I think).
>>>>
>>>> Also, you risk a collision of metrics if targets list includes the same
>>>> host twice on two different ports in the same scrape job (which should be
>>>> very unlikely).
>>>>
>>>> On Thursday, 27 April 2023 at 15:14:03 UTC+1 Kolja Krückmann wrote:
>>>>
>>>>> Hi Brian,
>>>>>
>>>>> thank you very much for your response!
>>>>> I just wanted to clarify, if it's better to "regex" the port to the
>>>>> end of the address, or if it is equally fine to just add the port in the
>>>>> target.yml to each target? (That's what im currently doing) Is it just
>>>>> best
>>>>> prac
Re: [prometheus-users] file_cd_config - label for each target
nevermind - sorry for the question, should have just googled learn regular
expression.
I'm fine for now.
Kolja Krückmann schrieb am Freitag, 28. April 2023 um 08:35:21 UTC+2:
> Thanks so much!
>
> Can you tell me where I could kinda "learn" regex? I find it very
> difficult to get known to regex because it's nothing im using on daily
> basis ^^
>
>
>
> Brian Candler schrieb am Donnerstag, 27. April 2023 um 17:24:26 UTC+2:
>
>> So to be clear, if you want your targets file to look like this:
>>
>> - targets:
>> - SVR-DS01 172.25.X0.XXX:9182
>> - SRV-DS02 172.21.X1.XXX:9182
>>
>> you can do something like this:
>>
>>relabel_configs:
>> # When __address__ is a single item, set the instance
>> # label to the part without the port
>> - source_labels: [__address__]
>> regex: '([^ ]+):[0-9]+'
>> replacement: '${1}'
>> target_label: instance
>>
>> # When __address__ is space-separated "name address:port",
>> # put the first part in the "instance" label and leave the second
>> part
>> # in "__address__"
>> - source_labels: [__address__]
>> regex: '(.+) (.+)'
>> target_label: instance
>> replacement: '${1}'
>> - source_labels: [__address__]
>> regex: '(.+) (.+)'
>> target_label: __address__
>> replacement: '${2}'
>>
>> If your target entries *always* consist of two items separated by a
>> space, then you don't need the first rule. It's only there in case you have
>> entries in the old format, i.e.
>>
>> - targets:
>> - 172.25.X0.XXX:9182
>>
>> (in which case, the instance is set to "172.25.X0.XXX")
>>
>> On Thursday, 27 April 2023 at 16:16:38 UTC+1 Brian Candler wrote:
>>
>>> Well, there is a good reason to keep the port out of the "instance"
>>> label: it's very awkward to make a query to join two different different
>>> metrics collected from two different exporters on the same host, if one has
>>> (say) instance="foo:9100" and another has instance="foo:9104". A second
>>> reason is that it's nicer in dashboards to see "foo" rather than "foo:9100"
>>>
>>> See https://www.robustperception.io/controlling-the-instance-label
>>>
>>> IMO the cleanest way to achieve this is to leave the port out of the
>>> targets file, copy __address__ to instance, and then append the exporter
>>> port statically to __address__.
>>>
>>> However, you could leave the port number in the targets file, and strip
>>> it out to create the instance label:
>>>
>>> - source_labels: [__address__]
>>> regex: '(.*):[0-9]+'
>>> target_label: instance
>>> replacement: '${1}'
>>>
>>> This would be useful if the same exporter were running on different
>>> ports on different hosts (fairly uncommon I think).
>>>
>>> Also, you risk a collision of metrics if targets list includes the same
>>> host twice on two different ports in the same scrape job (which should be
>>> very unlikely).
>>>
>>> On Thursday, 27 April 2023 at 15:14:03 UTC+1 Kolja Krückmann wrote:
>>>
>>>> Hi Brian,
>>>>
>>>> thank you very much for your response!
>>>> I just wanted to clarify, if it's better to "regex" the port to the end
>>>> of the address, or if it is equally fine to just add the port in the
>>>> target.yml to each target? (That's what im currently doing) Is it just
>>>> best
>>>> practice with the regex expression or is there actually a technical reason
>>>> behind that?
>>>>
>>>> Kind regards
>>>> Kolja
>>>>
>>>> Brian Candler schrieb am Dienstag, 18. April 2023 um 13:34:53 UTC+2:
>>>>
>>>>> Alternatively, you use rewriting rules.
>>>>>
>>>>> - targets:
>>>>> - SVR-DS01 172.25.X0.XXX
>>>>> - SRV-DS02 172.21.X1.XXX
>>>>> ... etc
>>>>>
>>>>> and the corresponding relabel_configs in your scrape job could be
>>>>> something like this:
>>>>>
>>>>> relabel_configs:
>>>>> # When __address__ consists of just a name or IP address,
>>>>> # copy it to the "instance" label.
Re: [prometheus-users] file_cd_config - label for each target
Thanks so much!
Can you tell me where I could kinda "learn" regex? I find it very difficult
to get known to regex because it's nothing im using on daily basis ^^
Brian Candler schrieb am Donnerstag, 27. April 2023 um 17:24:26 UTC+2:
> So to be clear, if you want your targets file to look like this:
>
> - targets:
> - SVR-DS01 172.25.X0.XXX:9182
> - SRV-DS02 172.21.X1.XXX:9182
>
> you can do something like this:
>
>relabel_configs:
> # When __address__ is a single item, set the instance
> # label to the part without the port
> - source_labels: [__address__]
> regex: '([^ ]+):[0-9]+'
> replacement: '${1}'
> target_label: instance
>
> # When __address__ is space-separated "name address:port",
> # put the first part in the "instance" label and leave the second
> part
> # in "__address__"
> - source_labels: [__address__]
> regex: '(.+) (.+)'
> target_label: instance
> replacement: '${1}'
> - source_labels: [__address__]
> regex: '(.+) (.+)'
> target_label: __address__
> replacement: '${2}'
>
> If your target entries *always* consist of two items separated by a space,
> then you don't need the first rule. It's only there in case you have
> entries in the old format, i.e.
>
> - targets:
> - 172.25.X0.XXX:9182
>
> (in which case, the instance is set to "172.25.X0.XXX")
>
> On Thursday, 27 April 2023 at 16:16:38 UTC+1 Brian Candler wrote:
>
>> Well, there is a good reason to keep the port out of the "instance"
>> label: it's very awkward to make a query to join two different different
>> metrics collected from two different exporters on the same host, if one has
>> (say) instance="foo:9100" and another has instance="foo:9104". A second
>> reason is that it's nicer in dashboards to see "foo" rather than "foo:9100"
>>
>> See https://www.robustperception.io/controlling-the-instance-label
>>
>> IMO the cleanest way to achieve this is to leave the port out of the
>> targets file, copy __address__ to instance, and then append the exporter
>> port statically to __address__.
>>
>> However, you could leave the port number in the targets file, and strip
>> it out to create the instance label:
>>
>> - source_labels: [__address__]
>> regex: '(.*):[0-9]+'
>> target_label: instance
>> replacement: '${1}'
>>
>> This would be useful if the same exporter were running on different ports
>> on different hosts (fairly uncommon I think).
>>
>> Also, you risk a collision of metrics if targets list includes the same
>> host twice on two different ports in the same scrape job (which should be
>> very unlikely).
>>
>> On Thursday, 27 April 2023 at 15:14:03 UTC+1 Kolja Krückmann wrote:
>>
>>> Hi Brian,
>>>
>>> thank you very much for your response!
>>> I just wanted to clarify, if it's better to "regex" the port to the end
>>> of the address, or if it is equally fine to just add the port in the
>>> target.yml to each target? (That's what im currently doing) Is it just best
>>> practice with the regex expression or is there actually a technical reason
>>> behind that?
>>>
>>> Kind regards
>>> Kolja
>>>
>>> Brian Candler schrieb am Dienstag, 18. April 2023 um 13:34:53 UTC+2:
>>>
>>>> Alternatively, you use rewriting rules.
>>>>
>>>> - targets:
>>>> - SVR-DS01 172.25.X0.XXX
>>>> - SRV-DS02 172.21.X1.XXX
>>>> ... etc
>>>>
>>>> and the corresponding relabel_configs in your scrape job could be
>>>> something like this:
>>>>
>>>> relabel_configs:
>>>> # When __address__ consists of just a name or IP address,
>>>> # copy it to the "instance" label. This keeps the port
>>>> # number out of the instance label.
>>>> - source_labels: [__address__]
>>>> regex: '([^ ]+)'
>>>> target_label: instance
>>>>
>>>> # When __address__ is of the form "name address", extract
>>>> # name to "instance" label and address to "__address__"
>>>> - source_labels: [__address__]
>>>> regex: '(.+) (.+)'
>>>> target_label: instance
>>>> replacem
Re: [prometheus-users] file_cd_config - label for each target
Hi Brian,
thank you very much for your response!
I just wanted to clarify, if it's better to "regex" the port to the end of
the address, or if it is equally fine to just add the port in the
target.yml to each target? (That's what im currently doing) Is it just best
practice with the regex expression or is there actually a technical reason
behind that?
Kind regards
Kolja
Brian Candler schrieb am Dienstag, 18. April 2023 um 13:34:53 UTC+2:
> Alternatively, you use rewriting rules.
>
> - targets:
> - SVR-DS01 172.25.X0.XXX
> - SRV-DS02 172.21.X1.XXX
> ... etc
>
> and the corresponding relabel_configs in your scrape job could be
> something like this:
>
> relabel_configs:
> # When __address__ consists of just a name or IP address,
> # copy it to the "instance" label. This keeps the port
> # number out of the instance label.
> - source_labels: [__address__]
> regex: '([^ ]+)'
> target_label: instance
>
> # When __address__ is of the form "name address", extract
> # name to "instance" label and address to "__address__"
> - source_labels: [__address__]
> regex: '(.+) (.+)'
> target_label: instance
> replacement: '${1}'
> - source_labels: [__address__]
> regex: '(.+) (.+)'
> target_label: __address__
> replacement: '${2}'
>
> # Append port number to __address__ so that scrape gets
> # sent to the right port
> - source_labels: [__address__]
> target_label: __address__
> replacement: '${1}:9182'
>
> On Tuesday, 18 April 2023 at 11:14:23 UTC+1 Julius Volz wrote:
>
>> The labels in the SD file can only be provided for each target group (the
>> top-level list item type of the YAML file), so you'd have to do something
>> like this:
>>
>> - targets:
>> - [...first list of targets...]
>> labels:
>> instance: "A"
>> - targets:
>> - [...second list of targets...]
>> labels:
>> instance: "B"
>> - targets:
>> - [...third list of targets...]
>> labels:
>> instance: "C"
>>
>>
>> On Tue, Apr 18, 2023 at 11:20 AM Kolja Krückmann
>> wrote:
>>
>>> Hi y'all I'm trying to get a label for each target in my file_sd_config
>>>
>>> my prom.yml is:
>>> (redacted)
>>>
>>> # my global config
>>> global:
>>> scrape_interval: 1m # Set the scrape interval to every 15 seconds.
>>> Default is every 1 minute.
>>> evaluation_interval: 30s # Evaluate rules every 15 seconds. The
>>> default is every 1 minute.
>>> scrape_timeout: 30s
>>>
>>> scrape_configs:
>>> # The job name is added as a label `job=` to any timeseries
>>> scraped from this config.
>>> - job_name: 'node'
>>> file_sd_configs:
>>> - files:
>>> -
>>> C:/Prometheus/prometheus-2.41.0.windows-amd64/target_cluster_b.yml
>>>
>>> my target_cluster_b.yml:
>>> - targets:
>>> - 172.25.X0.XXX:9182
>>> labels:
>>> instance: "SVR-DS01"
>>> - 172.21.X1.XXX:9182
>>> - 172.25.X2.XXX:9182
>>> - 172.25.X3.XXX:9182
>>>
>>> as seen in the target.yml I want to try to get a label to each target -
>>> to see in the prom dashboard which ip is which targetname
>>>
>>> Thanks in advance.
>>> Kind regards Kolja
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Prometheus Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to prometheus-use...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/prometheus-users/fa4e5c9e-807b-4680-9640-592c8172e568n%40googlegroups.com
>>>
>>> <https://groups.google.com/d/msgid/prometheus-users/fa4e5c9e-807b-4680-9640-592c8172e568n%40googlegroups.com?utm_medium=email_source=footer>
>>> .
>>>
>>
>>
>> --
>> Julius Volz
>> PromLabs - promlabs.com
>>
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/72234998-58e3-4ea5-86ed-880a99dfb0fcn%40googlegroups.com.
[prometheus-users] file_cd_config - label for each target
Hi y'all I'm trying to get a label for each target in my file_sd_config my prom.yml is: (redacted) # my global config global: scrape_interval: 1m # Set the scrape interval to every 15 seconds. Default is every 1 minute. evaluation_interval: 30s # Evaluate rules every 15 seconds. The default is every 1 minute. scrape_timeout: 30s scrape_configs: # The job name is added as a label `job=` to any timeseries scraped from this config. - job_name: 'node' file_sd_configs: - files: - C:/Prometheus/prometheus-2.41.0.windows-amd64/target_cluster_b.yml my target_cluster_b.yml: - targets: - 172.25.X0.XXX:9182 labels: instance: "SVR-DS01" - 172.21.X1.XXX:9182 - 172.25.X2.XXX:9182 - 172.25.X3.XXX:9182 as seen in the target.yml I want to try to get a label to each target - to see in the prom dashboard which ip is which targetname Thanks in advance. Kind regards Kolja -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/fa4e5c9e-807b-4680-9640-592c8172e568n%40googlegroups.com.
[prometheus-users] Re: {\"message\":\"Not Found\"}\n" - mail Alertmanager (and prometheus-msteams)
I am trying to use prometheus-msteams as middleware. there I am using my .tmpl earlyer attached. The error sais that it's missing a summary or text. But my tmpl should have that. any suggestions? Brian Candler schrieb am Dienstag, 7. März 2023 um 11:29:31 UTC+1: > On Tuesday, 7 March 2023 at 08:48:37 UTC Kolja Krückmann wrote: > > I just checked and prom is using the correct yml. I just missed that I > actually changed the alertmanager to localhost:9093 - thats why my error > sais localhost:9093 not 127.0.0.1:9093 > > > > Furthermore, I actually don't know why but I just restartet the prom.exe > and the alertmanager.exe and its working fine. > > localhost:9093 and 127.0.0.1:9093 are different > > The first connects to any address which maps to "localhost" in your hosts > file, and this includes ::1 (IPv6) > > 127.0.0.1 *only* connects to 127.0.0.1 (IPv4) > > > Another question I had was how to configure the webhook to teams > correctly. > > You can't simply point a webhook to Teams. The Alertmanager webhook sends > a fixed format JSON payload, which is not in the format that Teams expects > (as the error says). > > You'll need some middleware to convert it into something that Teams will > understand, for example: > https://github.com/idealista/prom2teams > > Google for "prometheus alertmanager teams" for more info. > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/c6d83517-7683-4faa-9b64-8356e6b619d6n%40googlegroups.com.
[prometheus-users] Re: {\"message\":\"Not Found\"}\n" - mail Alertmanager (and prometheus-msteams)
Thanks for the quick response. I just checked and prom is using the correct yml. I just missed that I actually changed the alertmanager to localhost:9093 - thats why my error sais localhost:9093 not 127.0.0.1:9093 Furthermore, I actually don't know why but I just restartet the prom.exe and the alertmanager.exe and its working fine. Another question I had was how to configure the webhook to teams correctly. The commented block in the end from my alertmanager.yml is now the following: route: receiver: 'email' group_wait: 30s group_interval: 5m repeat_interval: 4h group_by: [cluster, alertname] routes: - receiver: email group_interval: 1m group_wait: 10s repeat_interval: 1m matchers: - severity="critical" - receiver: email group_interval: 1m group_wait: 10s repeat_interval: 1m matchers: - severity="warning" - receiver: teams group_interval: 1m group_wait: 10s repeat_interval: 1m matchers: - severity="critical" receivers: - name: 'email' email_configs: - to: 'm...@company.com' send_resolved: true - name: 'teams' webhook_configs: - url: 'https://company.webhook.office.com/webhookb2/XXX' send_resolved: true error in prom console: ts=2023-03-07T07:42:29.984Z caller=notifier.go:532 level=error component=notifier alertmanager=http://localhost:9093/api/v2/alerts count=25 msg="Error sending alert" err="Post \ "http://localhost:9093/api/v2/alerts\; <http://localhost:9093/api/v2/alerts/>: dial tcp [::1]:9093: connectex: No connection could be made because the target machine actively refused it." and my alertmanager console sais: ts=2023-03-07T08:10:21.942Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=150 err="teams/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 400: https://company.webhook.office.com/webhookb2/XXX: Summary or Text is required." I will attach my tmpl. I copied the prometheus-msteams on my machine and run it via: .\prometheus-msteams-windows-amd64.exe -http-addr "localhost:2000" -teams-incoming-webhook-url "https://company.webhook.office.com/webhookb2/XXX; <https://company.webhook.office.com/webhookb2/XXX> What I already checked: - ports used (running netstat -ao) -> everything looks fine, ports aren't blocked and only the prom services are running on those ports - firewall rules (non blocking) Now i'm kinda stuck because I really want that Teams integration but running prom on Windows Server (without docker) doesn't seem to be that often in use. And one final question in regards of using this google group: When I'm having a completly different question shall I create a new conversation or just ask in an open one like mine here? (e.x. for the relabeling part - because I actually don't know anything about regex and so on) Brian Candler schrieb am Montag, 6. März 2023 um 16:35:12 UTC+1: > On Monday, 6 March 2023 at 12:33:21 UTC Kolja Krückmann wrote: > > # Alertmanager configuration > alerting: > alertmanagers: > - static_configs: > - targets: >- 127.0.0.1:9093 > > > The error message that you've shown suggests otherwise: > > * It seems Prometheus is actually using "localhost:9093", not " > 127.0.0.1:9093", as the alertmanager target > * "localhost" is being resolved to ::1 (the IPv6 loopback address) > * alertmanager is not accepting connections on IPv6 > > What flags are you running alertmanager with? If you've > set --web.listen-address=127.0.0.1:9093 then that would explain it. > > You should double-check what actual config file prometheus is running > with, because I believe it's not actually using 127.0.0.1:9093 to talk to > alertmanager. That is: it may be using a different config file than the > one you're looking at. > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/0bc4c741-f32f-4e08-a6a5-62cf87d0100an%40googlegroups.com.
[prometheus-users] {\"message\":\"Not Found\"}\n" - mail Alertmanager (and prometheus-msteams)
Hi y'all I'm currently trying to get my alerting to work via email. When running my prom.exe I get the following error quite often: ts=2023-03-06T10:32:02.804Z caller=notifier.go:532 level=error component=notifier alertmanager=http://localhost:9093/api/v2/alerts count=30 msg="Error sending alert" err="Post \"http://localhost:9093/api/v2/alerts\": dial tcp [::1]:9093: connectex: No connection could be made because the target machine actively refused it." My prom.yml is: # my global config global: scrape_interval: 1m # Set the scrape interval to every 15 seconds. Default is every 1 minute. evaluation_interval: 30s # Evaluate rules every 30 seconds. The default is every 1 minute. scrape_timeout: 30s # scrape_timeout is set to the global default (10s). # Alertmanager configuration alerting: alertmanagers: - static_configs: - targets: - 127.0.0.1:9093 # Load rules once and periodically evaluate them according to the global 'evaluation_interval'. rule_files: - rules.yml # A scrape configuration containing exactly one endpoint to scrape: # Here it's Prometheus itself. scrape_configs: # The job name is added as a label `job=` to any timeseries scraped from this config. - job_name: 'node' file_sd_configs: - files: - C:/Prometheus/prometheus-2.41.0.windows-amd64/target_cluster_b.yml alertingmanager.yml: global: resolve_timeout: 1m smtp_smarthost: 'smtp.ionos.de:587' smtp_from: 'internal-mail1' smtp_auth_username: ' internal-mail1' smtp_auth_password: 'password' smtp_require_tls: true templates: - 'C:\Prometheus\templates\default-message-card.tmpl' route: receiver: 'email' group_wait: 30s group_interval: 5m repeat_interval: 4h group_by: [cluster, alertname] routes: - receiver: email group_interval: 1m group_wait: 10s repeat_interval: 1m matchers: - severity="critical" - receiver: email group_interval: 1m group_wait: 10s repeat_interval: 1m matchers: - severity="warning" receivers: - name: 'email' email_configs: - to: 'internal-mail2' send_resolved: true #- name: 'alert_channel' # webhook_configs: # - url: 'http://127.0.0.1:2000/alertmanager' #send_resolved: true # # #- name: 'teams' # webhook_configs: #- url: 'my-teams-webhook' # send_resolved: true Can someone help me to correct my fault? If I missed something please let me know :) Kind regards - Kolja -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/3b2102a1-0692-436b-80b6-b56371b410ccn%40googlegroups.com.
[prometheus-users] Blackbox relabel_configs
Hi there, hope y'all doing fine. I'm fairly new to prom and having a bit of trouble with all of the configs. Currently I want to use the blackbox_exporter to ping the endpoints. I just want to know if they are up and healthy or down/off. But I just can't figure out how to relabel correctly in order to get the correct url. the correct one should look like this: http://*1st_client_ip* :9115/probe?module=icmp=*1st_client_ip* my prometheus.yml: # my global config global: scrape_interval: 30s # Set the scrape interval to every 15 seconds. Default is every 1 minute. evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute. scrape_timeout: 15s # scrape_timeout is set to the global default (10s). # Alertmanager configuration alerting: alertmanagers: - static_configs: - targets: - alertmanager:9093 # Load rules once and periodically evaluate them according to the global 'evaluation_interval'. rule_files: - rules.yml [...] static_configs: - job_name: 'blackbox-icmp' metrics_path: /probe params: module: [icmp] static_configs: - targets: - *1st_client_ip*:9115 - *2nd_client_ip*:9182 relabel_configs: - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: instance With this relabel_config I at least have the target and module in the probe-url. But attached at the end is some kind of modulo and I don't really know where it's coming from: http://*1st_client_ip* :9115/probe?module=icmp=*1st_client_ip*%3A9115 and with that all values in the icmp probe is 0. when deleting the modulo manually from url - everything works just fine. Can someone help me to get it right? Kind regards - Kolja -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/8cea7a8b-cc0d-4112-8601-41b735ead7ccn%40googlegroups.com.
[prometheus-users] Blackbox icmp probe relabel_configs
Hi there, hope you're doing well. I'm fairly new to prom and having a bit of trouble with all the configs. I installed the Blackbox exporter for testing on a Windows Client. What I just want to do is a simple ping to the blackbox node and see in my dashboard if the client is up or down. my prometheus.yml: [...] - job_name: 'blackbox-icmp' metrics_path: /probe params: module: [icmp] static_configs: - targets: - 1stclient_ip:9115 - 2ndclient_ip:9115 relabel_configs: - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: instance When doing this i at least get more or less the correct url, like: http://1stclient_ip:9115/probe?module=icmp= 1stclient_ip %3A9115 but the problem with that is the modulo at the end, which is causing the hole probe to give only 0's... so there has to be something wrong with my relabel_config. Completly without i get the error "400 server not reachable" and when clicking the node the html sais "target parameter is missing" Can someone tell me what my fault is? Kind regards - Kolja -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/71c00036-6b35-42bf-89b9-de4bf872c28bn%40googlegroups.com.
