[prometheus-users] How to setup prometheus using ansible

[Anirudh Pasalapudi]

I have a task to setup an environment where I need to have an autoscaling 
group, load balancer and two ec2 instances with prometheus installed on 
them. Along side a single standalone ec2 instance with prom aggregation 
gateway package installed on it. All this has to be set up using ansible. I 
am new to this kind of setup and I really need assistance in writing the 
ansible set up for this architecture. Can any one of you guys please 
provide necessary resources which I can utilize to write automation for 
creating all the above mentioned infrastructure. 

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/8d587607-f13e-464b-9ed0-c4b0ee50152ao%40googlegroups.com.

[prometheus-users] unexspected behaviour for collector.mountstat on kubernetes

[Geerten Schram]

Hi,

I'm getting unexpected results with prometheus-node-exporter on kubernetes. 
I'm using the prometheus operator and I enabled the mounstat collector (I 
can see the right config in the description for the daemonset and for the 
pods). But no statistics are collected. The /proc of the node is mounted in 
the pod on /host/proc , but there is no nfs mounstsat information in the 
mounted thre  (/host/proc). When I take a look at the node level I can see 
the right information. Has anyone has any idea what is going on here? I'm 
kind of stuck...

Regards,

Geerten

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/6be3d9cc-0364-45cf-bbd5-ee8588519cf2o%40googlegroups.com.

[prometheus-users] Re: file_sd_configs common label

[Senthil]

thanks. 

On Saturday, June 6, 2020 at 7:57:51 AM UTC-4, Brian Candler wrote:
>
> Yes, by adding the common labels to each target group:
>
> [
>   {
> "targets": [
>   "localhost:9100"
> ],
> "labels": {
> *  "env": "production",*
>   "job": "slave"
> }
>   },
>   {
> "targets": [
>   "localhost:9200"
> ],
> "labels": {
> *  "env": "production",*
>   "job": "master"
> }
>   }
> ]
>
> (pre-process your JSON if you don't like doing that by hand).  
> Alternatively, if the labels are common to *all* targets in that file, then 
> you can add them using label rewriting in the prometheus job config.
>
> Note though: it is generally not a good idea to override the "job" label.  
> This is one used by prometheus itself to identify the scrape job, and helps 
> to keep metrics unique.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/8098cd2f-3c73-4463-b21a-4f41f5d65db1o%40googlegroups.com.

Re: [prometheus-users] Resuable Template not found

[Brian Candler]

The configuration file that you posted was mangled - partly by your E-mail 
client I guess, and partly by you (there are two single quotes before 
''my.host')

However, if I use *exactly* the following configuration file:

global:
  smtp_smarthost: 'my.host:25'
  smtp_hello: 'my.system'
  smtp_from: 'my.adderss'

route:
  receiver: test-email-receiver
  group_by: [label1, label2]

receivers:
  - name: 'test-email-receiver'
email_configs:
  - to: 'som...@example.org'
text: '{{ template "custom.email" . }}'

templates:
  - '/etc/alertmanager/templates/custom-email-template.tmpl'

then alertmanager starts just fine:

root@prometheus:~# /opt/alertmanager/alertmanager 
--config.file=/tmp/alert.yml
level=info ts=2020-06-06T17:16:00.765Z caller=main.go:231 msg="Starting 
Alertmanager" version="(version=0.20.0, branch=HEAD, 
revision=f74be0400a6243d10bb53812d6fa408ad71ff32d)"
level=info ts=2020-06-06T17:16:00.766Z caller=main.go:232 
build_context="(go=go1.13.5, user=root@00c3106655f8, 
date=20191211-14:13:14)"
level=info ts=2020-06-06T17:16:00.770Z caller=cluster.go:161 
component=cluster msg="setting advertise address explicitly" 
addr=10.12.255.33 port=9094
level=info ts=2020-06-06T17:16:00.777Z caller=cluster.go:623 
component=cluster msg="Waiting for gossip to settle..." interval=2s
level=info ts=2020-06-06T17:16:00.871Z caller=coordinator.go:119 
component=configuration msg="Loading configuration file" file=/tmp/alert.yml
level=info ts=2020-06-06T17:16:00.875Z caller=coordinator.go:131 
component=configuration msg="Completed loading of configuration file" 
file=/tmp/alert.yml
level=info ts=2020-06-06T17:16:00.888Z caller=main.go:497 msg=Listening 
address=:9093
level=info ts=2020-06-06T17:16:02.777Z caller=cluster.go:648 
component=cluster msg="gossip not settled" polls=0 before=0 now=1 
elapsed=2.00028059s
level=info ts=2020-06-06T17:16:10.778Z caller=cluster.go:640 
component=cluster msg="gossip settled; proceeding" elapsed=10.001491515s
q
^C
level=info ts=2020-06-06T17:16:40.881Z caller=main.go:536 msg="Received 
SIGTERM, exiting gracefully..."

root@prometheus:~# /opt/alertmanager/alertmanager --version
alertmanager, version 0.20.0 (branch: HEAD, revision: 
f74be0400a6243d10bb53812d6fa408ad71ff32d)
  build user:   root@00c3106655f8
  build date:   20191211-14:13:14
  go version:   go1.13.5

This is despite the fact that I haven't even created 
/etc/alertmanager/templates/custom-email-template.tmpl

Therefore, I can only surmise that the configuration you're testing with is 
not the one you posted. Maybe you haven't given the correct command-line 
argument to alertmanager to tell it which config file to read, and so it's 
reading a different one than the one you think it is.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/9de8ab3d-7f4e-43e1-be5c-665908e776cdo%40googlegroups.com.

Re: [prometheus-users] Resuable Template not found

[Mark Leone]

Here is my alertmanager version info:

alertmanager, version 0.20.0 (branch: HEAD, revision: f74be04

)
build user: root@00c3106655f8
biold date: 20191211-14:13:14
go version: gol.13.5

Please clarify what you mean when you say "Alertmanager should start with
an unknown template. ". I reference a template in the text field of the
email_configs setting. If I reference the default template (either
explicitly or by not specifying the text property), it works. It finds the
template and sends a templated message as expected.If I reference the
custom template that I defined (custom.email), alertmanager fails to start
and says it can't find the template I referenced. AFAICAT I followed the
example exactly in defining a custom template and pointing to it in the
"templates" config setting. I also exec into the docker image and verify
that the custom template file is where I pointed alertmanager to,
permissions are opened, and it contains the text I expect it to contain.I
did see a report somewhere that you can't override the default templates,
so I gave my template a different name; but still alertmanager is not able
to find it.


On Wed, Jun 3, 2020 at 1:20 AM Julien Pivotto 
wrote:

> Which version is this? Alertmanager should start with an unknown template.
>
> Le mer. 3 juin 2020 à 02:35, Mark Leone  a écrit :
>
>> I still haven't figured out why it's not working. Developers responding
>> to the GitHub issue I opened believe I've got something configured wrong,
>> since no one else has reported this problem. Can someone confirm that
>> they're able to use custom templates per the instructions referenced in the
>> OP? If so, is there something else not mentioned in the instructions that I
>> need to do?
>>
>> On Thu, May 28, 2020, 12:38 AM Mark Leone  wrote:
>>
>>> I'm following this example to define a re-usable template:
>>> https://prometheus.io/blog/2016/03/03/custom-alertmanager-templates/
>>>
>>> and alertmanager is not able to find the template
>>>
>>> This is my alertmanager config file:
>>>
>>> global:
>>> smtp_smarthost: ''my.host:25'
>>> smtp_hello: 'my.system'
>>> smtp_from: 'my.adderss'
>>>
>>> route:
>>> receiver: test-email-receiver
>>> group_by: [label1, label2]
>>>
>>> receivers:
>>>
>>>- name: 'test-email-receiver'
>>>email_configs:
>>>   - to: 'some...@example.org'
>>>   text : '{{ template "custom.email" . }}'
>>>
>>> templates:
>>>
>>>- '/etc/alertmanager/templates/custom-email-template.tmpl'
>>>
>>> This is the contents of /etc/alertmanager/custom-email-template.tmpl:
>>>
>>> {{ define "custom.email" }}Test{{ end }}
>>>
>>> I'm running the alertmanager:latest docker image, and it fails with the
>>> error: template "custom.email" not defined
>>>
>>> The custom template file is written to the local docker volume by a
>>> nomad template stanza in the docker driver config, and mapped to
>>> /etc/alertmanager/templates/custom-email-template.tmpl. I removed the
>>> text entry in email_configs so I can successfully start the container and
>>> connect to it with docker exec. I verified that the custom template file is
>>> where I expect it to be and it has the expected contents. Alertmanager just
>>> refuses to recognize the template. Either the example is wrong, or I'm
>>> missing something.
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Prometheus Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to prometheus-users+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/prometheus-users/f0f16572-6bf7-4085-b9a7-d5948e3b41f3%40googlegroups.com
>>> 
>>> .
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to prometheus-users+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-users/CAFHj5BddEFUdf_dd-3o9gvi65kN-LcYjEn6Ee0P_YieQ%2BHns7Q%40mail.gmail.com
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAFHj5Bfqp5B_-B4%2BZeBC3t7EvFfZXfyMMpz%3D2jQTxUoWduTMpA%40mail.gmail.com.

[prometheus-users] Re: alertmanager: no private IP found

[Saeedeh Moghimi]

It seems this label solve the problem
  --cluster.advertise-address="public-ip:9093"



On Saturday, June 6, 2020 at 4:04:42 PM UTC+4:30, Saeedeh Moghimi wrote:
>
> I made a service for alertmanager, which is like this:
>
>
> Description=Alert Manager
> Wants=network-online.target
> After=network-online.target
>
> [Service]
> Type=simple
> User=alertmanager
> Group=alertmanager
> ExecStart=/usr/local/bin/alertmanager \
>   --config.file=/etc/alertmanager/alertmanager.yml \
>   --storage.path=/data/alertmanager \ 
>   --web.external-url http://public_ip_server:9093
>
> Restart=always
>
> [Install]
> WantedBy=multi-user.target
>  
>
> I get this error when i start the alermanager
> err="couldn't deduce an advertise address: no private IP found, explicit 
> advertise addr not provided"
>
>
> no private IP has been configed on this server, but I set --web.external-url 
> http://public_ip_server:9093 , Why I still get this error?
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/36514088-0580-4af0-81e9-55d08f104f71o%40googlegroups.com.

[prometheus-users] Re: file_sd_configs common label

[Brian Candler]

Yes, by adding the common labels to each target group:

[
  {
"targets": [
  "localhost:9100"
],
"labels": {
*  "env": "production",*
  "job": "slave"
}
  },
  {
"targets": [
  "localhost:9200"
],
"labels": {
*  "env": "production",*
  "job": "master"
}
  }
]

(pre-process your JSON if you don't like doing that by hand).  
Alternatively, if the labels are common to *all* targets in that file, then 
you can add them using label rewriting in the prometheus job config.

Note though: it is generally not a good idea to override the "job" label.  
This is one used by prometheus itself to identify the scrape job, and helps 
to keep metrics unique.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/0a39b0d9-c14d-4098-999e-2c185ed39ee8o%40googlegroups.com.

[prometheus-users] Re: Timestamp of last change of metric

[Brian Candler]

timestamp(my_metric) gives you the time at which the last sample in 
my_metric was recorded.

timestamp(my_metric == 1)in principle ought to give you the last time when 
it was 1, but I think instant queries only look back a few minutes, so you 
might need to use a subquery.

Documentation link: 
https://prometheus.io/docs/prometheus/latest/querying/functions/#timestamp

>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/73c41d43-e355-4db2-88ec-e1838664fa5bo%40googlegroups.com.

[prometheus-users] alertmanager: no private IP found

[Saeedeh Moghimi]

I made a service for alertmanager, which is like this:


Description=Alert Manager
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=alertmanager
Group=alertmanager
ExecStart=/usr/local/bin/alertmanager \
  --config.file=/etc/alertmanager/alertmanager.yml \
  --storage.path=/data/alertmanager \ 
  --web.external-url http://public_ip_server:9093

Restart=always

[Install]
WantedBy=multi-user.target
 

I get this error when i start the alermanager
err="couldn't deduce an advertise address: no private IP found, explicit 
advertise addr not provided"


no private IP has been configed on this server, but I set --web.external-url 
http://public_ip_server:9093 , Why I still get this error?

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/776f1f83-3fb6-4a27-bc28-00a084072d44o%40googlegroups.com.

[prometheus-users] Is there any unique id for an alert sent by alert-manager?

[zichen chuh]

Learned from this link is-the-fingerprint-field-in-alertmanager-unique 

 that 
both fingerprint and generatorURL are not unique.
Wonder whether prometheus would provide a function to generate a GUID that 
can be used in prometheus alert field template.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/2cf87a42-1df0-4b07-a645-d0d7c33dc060o%40googlegroups.com.

Re: [prometheus-users] Re: Freeze dates & release dates

[Ben Kochie]

There is no specific release schedule or cadence for the node_exporter. We
release when there is sufficient need and or bug fixes are ready.

On Fri, Jun 5, 2020 at 2:08 AM Ranganath Sunku 
wrote:

> I meant to ask for Prometheus node_exporter in particular.
>
> On Thursday, June 4, 2020 at 5:03:49 PM UTC-7, Ranganath Sunku wrote:
>>
>> Hello Maintainers,
>>
>>
>> Being new to this community, our team is looking to contribute few
>> collectors to the codebase to be considered for upcoming releases. Going
>> through wiki & milestones, there isnt clarity on release process.
>>
>> Few questions to better help us plan our work:
>>
>>- based on past releases, would it be right to assume 1.1 be in
>>Nov/Dec 2020 while 1.2 be sometime April/May 2021?
>>- Is there a code freeze/feature freeze date that one should be aware
>>of to ensure new feature requests are submitted adequately ahead of time
>>
>> Your feedback will be helpful. Thanks
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/d47d0d82-1d8b-42f2-a158-75f1dfd0dbe6o%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CABbyFmpABy7%2Bmx%3D%2B%3DF%3DqgT9UgPD4%2BBsEeztKv%3DMw_shAWXtOCQ%40mail.gmail.com.

Re: [prometheus-users] Re: Freeze dates & release dates

[Matthias Rampke]

We only have a fixed release schedule for Prometheus itself. Exporter
releases are cut by the maintainers as needed, depending on the changes
merged.

/MR

On Fri, Jun 5, 2020, 02:08 Ranganath Sunku  wrote:

> I meant to ask for Prometheus node_exporter in particular.
>
> On Thursday, June 4, 2020 at 5:03:49 PM UTC-7, Ranganath Sunku wrote:
>>
>> Hello Maintainers,
>>
>>
>> Being new to this community, our team is looking to contribute few
>> collectors to the codebase to be considered for upcoming releases. Going
>> through wiki & milestones, there isnt clarity on release process.
>>
>> Few questions to better help us plan our work:
>>
>>- based on past releases, would it be right to assume 1.1 be in
>>Nov/Dec 2020 while 1.2 be sometime April/May 2021?
>>- Is there a code freeze/feature freeze date that one should be aware
>>of to ensure new feature requests are submitted adequately ahead of time
>>
>> Your feedback will be helpful. Thanks
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/d47d0d82-1d8b-42f2-a158-75f1dfd0dbe6o%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAMV%3D_gY2sD7cLppenBkj286nnFO4pztYs9yAdYLhv80vDKWN1w%40mail.gmail.com.

Re: [prometheus-users] Re: (Alertmanager) Ignore instance label to prevent same alert multiple times

[Matthias Rampke]

Unfortunately you cannot have it both ways: either Alertmanager knows about
separate alert instances that can be silenced separately, or it doesn't.

 I would try to eliminate the need to silence by Prometheus, for example by
making the alert expressions resistant to gaps in the data.

/MR

On Fri, Jun 5, 2020, 05:40 'ping...@hioscar.com' via Prometheus Users <
prometheus-users@googlegroups.com> wrote:

> Thanks for your replies, guys. We have two replicated prometheus instances
> scraping the same metrics and sending the same alerts in parallel to
> alertmanager. We add a label to alerts indicating which prometheus instance
> the alert is fired from, so that if one prometheus instance is going bad we
> can silence alerts from that instance. The pain point is that the alert
> email (grouped) body is bloated with duplicated alert texts from both
> instances with only one label being different.
>
> We would like to keep the label so that we can silence alerts at
> prometheus instance level. So, label replacement or aggregating labels in
> prometheus doesn't seem the right way for us. I think it would work for us
> if alertmanager can be configured to ignore or collapse certain labels in
> email texts, like:
>
> From prometheus instance 1:
> label_A = value1
>
> From prometheus instance 2:
> label_A = value2
>
> In alert email:
> label_A = value1, value2
>
> Regards,
>
> Ping
>
> On Thursday, June 4, 2020 at 4:07:32 PM UTC-4, Christian Hoffmann wrote:
>>
>> Hi,
>>
>> On 6/4/20 7:48 PM, 'pin...@hioscar.com' via Prometheus Users wrote:
>> > We get the same alert multiple times in the same email, because the
>> > monitor label (prometheus instance) being different for our simple
>> > replicated setup. Would be nice to be able to ignore certain labels so
>> > that alert bodies are higher signal.
>>
>> This sounds like it could be done on the Prometheus side using existing
>> (standard) features.
>> Any reason why just aggregating away the unwanted label would not work?
>>
>> E.g.
>>
>> avg without(instance) (some_metric)
>>
>> (Depending on the value, other aggregation functions such as sum, min or
>> max might make more sense)
>>
>> Kind regards,
>> Christian
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/c878c840-06c4-4c92-85ac-6ebb63332dbfo%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAMV%3D_gbGTMg5f-OZjVYha8%2BDy4o1aoOZWKfKdWe%3DAvC%2ByU5d8A%40mail.gmail.com.