[prometheus-users] autoclosable alerts

[Roxana Angheluta]

Hello,

We are using alerta (https://alerta.io/) as the alertmanager receiver.
Is it possible to have only some of the alerts closing themselves based on 
a label?

I have tried something like this:

route:
   receiver: alerta

   routes:
   - receiver: alerta-closable
 match:
   autoclosable: true

receivers:
- name: alerta
  webhook_configs:
  - url: 
- name: alerta-closable
  webhook_configs:
  - url: 
send_resolved: true

It does not seem to work correctly, alerts without the label are 
dissapearring from the alerta dashboard.
Is the approach correct?

Thanks,
Roxana

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/8b6b0477-5e3a-4031-9917-d2843233147an%40googlegroups.com.

[prometheus-users] Using alert manager with external receiver

[Nina Sc]

Hi,

Is there a way to use the alert manager to send an alert to external API,
I mean instead using excage server or slack etc, I will provide an endpoint
like `https://mypullendpoint.host.com` and the alert manager will send the 
alert to this URL ?

Thanks! 
Nina

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/228c0a89-9871-4c50-8835-a65d9c1c87cen%40googlegroups.com.

Re: [prometheus-users] Using alert manager with external receiver

[Wesley Peng]

Hi

Nina Sc wrote:

Is there a way to use the alert manager to send an alert to external API,
I mean instead using excage server or slack etc, I will provide an endpoint
like `https://mypullendpoint.host.com` and the alert manager will send the
alert to this URL ?


Yes you need to define receivers and webhook.

such as:

receivers:
- name: "alerta"
  webhook_configs:
  - url: 'http://localhost:8080/webhooks/prometheus'
send_resolved: true


please refer these:
https://alerta.io/
https://github.com/alerta/prometheus-config


Regards.

--
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/375f41af-8bfb-e0c6-c4ec-38a0c6ca0847%40gmx.ie.

[prometheus-users] Re: Using alert manager with external receiver

[Brian Candler]

Yes - use the webhook 
 
receiver.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/a731b4ab-798c-4519-a97e-8bf671e49bc0o%40googlegroups.com.

[prometheus-users] Regarding Setting High Retension for storage

[Aravind Viswanathan]

Hi All,
I am new to Prometheus and noticed that the data in my Prometheus Server is 
defaulted to 15days and we would like to have the data for more than 1Y.

I have updated my configuration to ,

[Unit]
  Description=Prometheus Monitoring
  Wants=network-online.target
  After=network-online.target

[Service]
  User=prometheus
  Group=prometheus
  Type=simple
  ExecStart=/usr/local/bin/prometheus \
  --config.file /etc/prometheus/prometheus.yml \
  --storage.tsdb.path /var/lib/prometheus/ \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries
  ExecReload=/bin/kill -HUP $MAINPID
  --storage.tsdb.retention.time=1y

[Install]
  WantedBy=multi-user.target
~
~
Then i restarted Prometheus Service in Ubuntu
Navigated to status >> Runtime and Build Information i see
Storage Retention  as 15 Days
Also, navigated to  command line flags and i still see 

storage.tsdb.retention : 0s
storage.tsdb.retention.size : 0s
storage.tsdb.retention.time : 0s.

Is there something that i am missing?

Regards
Aravind Viswanathan. 

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/97c19971-72fa-47f0-a848-40be4b61441an%40googlegroups.com.

Re: [prometheus-users] Implemention advice: Prometheus for mature Windows client/server system

[Stuart Clark]

On 2020-09-07 17:22, Mister Boy wrote:

I'm brand new to Prometheus and my head is spinning after spending
time reading the docs so I hope someone can offer a little advice.

We have a mature client-server system involving dozens of (mostly) C++
modules; workstations and servers all run on Windows and modules
mostly communcate via DCOM. We want to start recording metrics about
real-time state which is new to me.

I understand Prometheus is pull-based and appears to pull only via
HTTP(?) so I am struggling how to reconcile this with our system.
Engineering HTTP endpoints into our modules is realistically unlikely
and some of our modules use obscure tech which doesn't even allow
this.
Our metrics would generally come from:

- in-memory state in a few key modules running on the app-server
- state of the machines themselves e.g. CPU/RAM use
- database data, e.g. many metrics would map nicely to SQL queries
"how many widgets are active?"

My initial thought would be to create new module(s) which exist solely
to expose metrics to Prometheus, essentially a bridge - does that seem
reasonable? I'm open to the answer that perhaps a pull-based system is
simply not optimal and we should look at a different tool? Sorry for
the long post, I hope I've been able to give enough information where
I'm coming from.



The concept of "exporters" will probably help you.

Part of the design of Prometheus is a decentralised focus, so rather 
than having a single source for all metrics for a platform you might 
source different selections of information from all the different 
pieces.


An example of that would be the machine metrics. The standard solution 
to obtaining such information is to install the Node Exporter (or 
Windows Exporter for Windows machines) on all servers/VMs. This then 
gives you basic CPU, network, memory, disk, etc. metrics without any 
development access.


The same is true for databases - there are various exporters for the 
different database types, some of which can additionally support custom 
SQL queries to add more application specific metrics (such as your "how 
many widgets are active?" query).


For other applications the approach depends on what is possible & 
achievable. In the ideal world you would embed one of the many client 
libraries (available for many languages) to directly expose native 
Prometheus format metrics from every application. Sometimes that isn't 
possible due to it being non-modifyable (COTS, developed elsewhere, 
etc.), architectural issues (lack of ability to run long lived HTTP(s) 
endpoints) or political (lack of buy-in from teams/management 
responsible for development effort).


For the applications where full native metrics aren't possible, you 
again have several options. If the application does support another form 
of metrics (JMX, StatsD, Graphite, etc.) then there are exporters 
available to bridge from those systems into Prometheus. The Node 
Exporter also has the textfile collector which can be used to expose 
metrics from an application, without requiring such major code changes.


Finally you can use more standalone exporters. There are some more 
generic ones available for some form of API type calls (e.g. blackbox, 
JSON exporter), but otherwise you can write your own exporter in a 
variety of languages - as long as it is possible to get the data out of 
the application using some mechanism.


--
Stuart Clark

--
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/821dfcb25842e38f130836f27b232591%40Jahingo.com.

[prometheus-users] Need help on Prometheus load testing for capacity planning

[rs vas]

We are planning to attempt to generate the load on the Prometheus instance
so that we can identify the right EC2 instance family to run a specific
load.

We have tried https://github.com/ncabatoff/prombench but without
any success, since this has not been touched since 4 years, not compatible
with the latest Prometheus version.

Our requirement is: we would like to generate some load using a synthetic
exporter and target to expose x number metrics on a y number of targets for
x number of hours to identify the right instance size.

Does one have any suggestions to run a tool from an EC2 instance which can
expose targets? If possible can it also hit Prometheus with some queries...?

Appreciate any inputs on the same.

rs vas

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAPs_AfgE%2BNxkiEP%3DE0Z5pP4yRQdHz9jAt6C3OsXfKGrK%2B9iToA%40mail.gmail.com.

[prometheus-users] Re: Regarding Setting High Retension for storage

[Brian Candler]

Yes, you put the flag in the wrong place.  Add it to the end of the 
ExecStart line, not to the ExecReload line.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/2bc6d8ff-8e41-4266-8139-9238c7798cdco%40googlegroups.com.

[prometheus-users] Re: Need help on Prometheus load testing for capacity planning

[Brian Candler]

I'd be inclined to take the opposite approach: use your production 
exporters, guess the initial instance size, and iterate based on how under- 
or over-utilized it is.   It only takes a minute to switch an instance from 
one type to another in EC2.

As a starting point for your guess, you can use this:
https://www.robustperception.io/how-much-ram-does-prometheus-2-x-need-for-cardinality-and-ingestion

Your real metrics are going to be a better test than any synthetic set of 
metrics.  Remember also that you can scrape the same targets from multiple 
prometheus instances, so you can spin up a second prometheus for testing 
without touching the existing one, if you have one.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/1c6e391b-d06b-40fb-a088-67624b559596o%40googlegroups.com.

Re: [prometheus-users] Re: Simple alive monitor to a website

[Tom Black]

Brian,

Please help with this.

Following your suggestion, I have my rule file defined:

groups:
- name: host-down
  rules:

  # Alert for any instance that is unreachable for >5 minutes.
  - alert: InstanceDown
expr: probe_success == 0
for: 3m
labels:
  severity: page
annotations:
  summary: "Instance {{ $labels.instance }} down"
  description: "{{ $labels.instance }} of job {{ $labels.job }} has
been down for more than 5 minutes."


And I have the alertmanager.yml:

global:
  smtp_smarthost: 'localhost:25'
  smtp_from: 'i...@sample.xyz'

route:
  group_by: ['alertname']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1h
  receiver: team-X-mails

receivers:
- name: 'team-X-mails'
  email_configs:
  - to: 'm...@sample.org'


inhibit_rules:
- source_match:
severity: 'critical'
  target_match:
severity: 'warning'
  equal: ['alertname']


SMTP smarthost is working fine, I have tested it by hand (I know well on
smtp server).

But, I never got an email... there is nothing info in local postfix 's log.

Can you help digging into this?

Thanks in advance.



On Sun, Sep 6, 2020 at 3:28 PM Brian Candler  wrote:

> On Sunday, 6 September 2020 06:16:43 UTC+1, Tom Black wrote:
>>
>> I have used blackbox exporter:
>> https://github.com/prometheus/blackbox_exporter
>>
>> but I just didn't know where to setup alert email when there was server
>> down.
>>
>>
> Blackbox_exporter will give you a metric "probe_success" which returns 0
> or 1.  In prometheus you create an alerting rule on probe_success == 0, and
> you configure alertmanager with the E-mail destination.
> https://prometheus.io/docs/alerting/latest/overview/
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/83b516b1-8bfa-49ad-96aa-51814efaca0bo%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAMn7GYmtiF0vxM%3Dm%2BgtMf9x1JU_YkcJw-_fwefsAVozC24v43w%40mail.gmail.com.

Re: [prometheus-users] Re: Simple alive monitor to a website

[Brian Candler]

Just basic debugging:

1. Use the Prometheus web interface (usually x.x.x.:9090).  It has an 
"Alerts" tab.  It will tell you if the alerts exist, and whether any of 
them are firing.

If not, find out why.  Did you HUP prometheus to re-read the rules after 
changing config?  Does your prometheus.yml reference the rules file?

Try:
promtool check config /etc/prometheus/prometheus.yml

(it will tell you which rules files it read, and how many rules read in 
from each)

If this is all OK, then:

2. Use the Alertmanager web interface (usually x.x.x.x:9093).  It will tell 
you if the alerts are active.

If not, find out why.  (Did you HUP alertmanager to re-read the config?)

Similarly, check the configuration:
amtool check-config /etc/alertmanager/alertmanager.yml

If this is all OK, then:

3. Look at prometheus and alertmanager logs, e.g. if you're running under 
systemd:
journalctl -eu prometheus
journalctl -eu alertmanager 

Increase log verbosity if required.  Run the binaries with "--help" flag to 
see the available options.  Should be something like --log.level=debug

4. Use tcpdump to investigate traffic between prometheus and alertmanager, 
and between alertmanager and postfix.

My guess: you might need to set "smtp_require_tls: false" under global, 
since it defaults to true:
https://prometheus.io/docs/alerting/latest/configuration/#configuration-file

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/e819bea8-85a3-4862-b61a-9022c91df132o%40googlegroups.com.

Re: [prometheus-users] Re: Simple alive monitor to a website

[Tom Black]

Thanks a lot Brian.
It does work right now!
I greatly appreciate your help.

regards.

On Tue, Sep 8, 2020 at 6:56 PM Brian Candler  wrote:

> Just basic debugging:
>
> 1. Use the Prometheus web interface (usually x.x.x.:9090).  It has an
> "Alerts" tab.  It will tell you if the alerts exist, and whether any of
> them are firing.
>
> If not, find out why.  Did you HUP prometheus to re-read the rules after
> changing config?  Does your prometheus.yml reference the rules file?
>
> Try:
> promtool check config /etc/prometheus/prometheus.yml
>
> (it will tell you which rules files it read, and how many rules read in
> from each)
>
> If this is all OK, then:
>
> 2. Use the Alertmanager web interface (usually x.x.x.x:9093).  It will
> tell you if the alerts are active.
>
> If not, find out why.  (Did you HUP alertmanager to re-read the config?)
>
> Similarly, check the configuration:
> amtool check-config /etc/alertmanager/alertmanager.yml
>
> If this is all OK, then:
>
> 3. Look at prometheus and alertmanager logs, e.g. if you're running under
> systemd:
> journalctl -eu prometheus
> journalctl -eu alertmanager
>
> Increase log verbosity if required.  Run the binaries with "--help" flag
> to see the available options.  Should be something like --log.level=debug
>
> 4. Use tcpdump to investigate traffic between prometheus and alertmanager,
> and between alertmanager and postfix.
>
> My guess: you might need to set "smtp_require_tls: false" under global,
> since it defaults to true:
>
> https://prometheus.io/docs/alerting/latest/configuration/#configuration-file
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/e819bea8-85a3-4862-b61a-9022c91df132o%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAMn7GYkVR7r5qsg9hKJNoa7OSazaYUV5i02B272f81V6%3D3PW7w%40mail.gmail.com.

[prometheus-users] Can prometheurs scrape :/deamon/metrics ?

[Rodolphe Ghio]

Hi guys, 
I was wondering if prometheus can scrape a target like this ?
:/deamon/metrics, cause I've tried many times and prometheus 
doesn't want to start anymore.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/8d8bebd9-63d9-483c-8f78-da6293b5d605n%40googlegroups.com.

Re: [prometheus-users] Can prometheurs scrape :/deamon/metrics ?

[Wesley Peng]

I am sure it can. b/c I just finished a monitoring like this form. :)


Rodolphe Ghio wrote:

I was wondering if prometheus can scrape a target like this ?
:/deamon/metrics, cause I've tried many times and prometheus
doesn't want to start anymore.


--
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/65e5ab8b-793e-3e56-0ca5-085adf5b8d74%40gmx.ie.

Re: [prometheus-users] Can prometheurs scrape :/deamon/metrics ?

[Rodolphe Ghio]

Can i please take a look to your scrape config please ?

Le mardi 8 septembre 2020 à 13:29:40 UTC+2, wes...@gmx.ie a écrit :

> I am sure it can. b/c I just finished a monitoring like this form. :)
>
>
> Rodolphe Ghio wrote:
> > I was wondering if prometheus can scrape a target like this ?
> > :/deamon/metrics, cause I've tried many times and prometheus
> > doesn't want to start anymore.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/d9dff6e3-5256-44ac-a1e7-c537df6d8817n%40googlegroups.com.

Re: [prometheus-users] Can prometheurs scrape :/deamon/metrics ?

[Brian Candler]

@Rudolphe: can you show *your* scrape config, plus the error message you 
get when prometheus fails to start?

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/3c02fc0e-5bc3-4805-8420-47cd18da3d35o%40googlegroups.com.

Re: [prometheus-users] Re: Regarding Setting High Retension for storage

[Aravind Viswanathan]

Thanks Brian,

Is this Correct?

[Unit]
  Description=Prometheus Monitoring
  Wants=network-online.target
  After=network-online.target

[Service]
  User=prometheus
  Group=prometheus
  Type=simple
  ExecStart=/usr/local/bin/prometheus \
  --config.file /etc/prometheus/prometheus.yml \
  --storage.tsdb.path /var/lib/prometheus/ \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries
  --storage.tsdb.retention.time=1y
  ExecReload=/bin/kill -HUP $MAINPID


[Install]
  WantedBy=multi-user.target
~
~

On Tue, Sep 8, 2020 at 3:20 PM Brian Candler  wrote:

> Yes, you put the flag in the wrong place.  Add it to the end of the
> ExecStart line, not to the ExecReload line.
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Prometheus Users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/prometheus-users/rj695eUHMQA/unsubscribe
> .
> To unsubscribe from this group and all its topics, send an email to
> prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/2bc6d8ff-8e41-4266-8139-9238c7798cdco%40googlegroups.com
> 
> .
>


-- 
Thanks

Aravind.V

*Mobile*:+91. 8197177674

*Email* : v_6a...@hotmail.com, arwa...@gmail.com

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAK1hgMjRLgtv%2B8wnEpmDXmrANq5nryE4XojpRYQYwPh%3DKL8TWQ%40mail.gmail.com.

Re: [prometheus-users] Re: Regarding Setting High Retension for storage

[Brian Candler]

Not quite.  Since you're splitting the command line over multiple lines, 
you need a trailing space and backslash after "console_libraries" to join 
it to the next line.

>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/7d88121d-b793-4c3a-b840-2243ddf03086o%40googlegroups.com.

[prometheus-users] Modifying labels on default Prometheus metrics

[Ayush Kumar]

Hi Everyone,

I work with .NET Core applications and we use Prometheus along with Grafana 
for dashboarding. I wanted to know if there's a way to add a new Label to 
the default metrics that Prometheus has, for example:
http_requests_duration_seconds_count

This metric has a few default labels like action. I wanted to add one more 
label called callerID to this for my application.

Is it possible to modify the default metrics to include new labels ? If so, 
could you point me towards some documentation on how to do this ?

Thank you
Ayush Kumar

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/6198f160-72a4-4c2a-af65-d4da53c9b9fan%40googlegroups.com.

Re: [prometheus-users] Can prometheurs scrape :/deamon/metrics ?

[Rodolphe Ghio]

I've found how to do this, I've added metrics_path statement into my conf 
file. Thanks for answering 
Le mardi 8 septembre 2020 à 14:01:48 UTC+2, b.ca...@pobox.com a écrit :

> @Rudolphe: can you show *your* scrape config, plus the error message you 
> get when prometheus fails to start?
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/d9bba590-e10b-45a4-b555-e9699370d70fn%40googlegroups.com.

[prometheus-users] Alertmanager inhibitions with webhook

[fiala...@gmail.com]

Hi,

I'm using alertmanager with webhook reciever and stora active alerts in our 
own SQL DB to show active incidents.

I would like to implement inhibition rules. Is there any way how to recieve 
info about inhibited notification via webhook?

Thank you.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/2d01db30-32af-44ea-a44e-40cb2235bd8cn%40googlegroups.com.

[prometheus-users] Monitoring rate of events

[John Dexter]

I'm looking at adopting Prometheus for our system, one thing we want to
monitor are incoming alarms (and alarm clearances).

Metrics of use would be:

   - Total number of active alarms
   - Rate of alarms being raised
   - Rate of alarms being cleared

The second one is confusing me a bit. Prometheus will poll at repeated
intervals, and we might change that interval. I can report the *total*
number of alarm-raises/alarm-clearances since the system started using
Counter metrics but what I am interested in is the rate of change not the
absolute value. I just want to check:

   1. Is rate of change a standard visualization for a metric e.g. a graph
   of alarms-raised-per minute/hour/day?
   2. Our system runs for months at a time, I assume the maximum value for
   a Counter is so astronomically high this is a non issue?
   3. Would it make sense to use alarms raised/cleared as labels on the
   same time-series or should they be two totally separate time-series?

Is this the right way to monitor such a metric?

Thanks.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAGJRanhS1JjQxTPG12ArkZj9_0%2Bxr24puJ300%3DXJkGcJeFANWg%40mail.gmail.com.

Re: [prometheus-users] Modifying labels on default Prometheus metrics

[Stuart Clark]

On 08/09/2020 13:35, Ayush Kumar wrote:

Hi Everyone,

I work with .NET Core applications and we use Prometheus along with 
Grafana for dashboarding. I wanted to know if there's a way to add a 
new Label to the default metrics that Prometheus has, for example:

http_requests_duration_seconds_count

This metric has a few default labels like action. I wanted to add one 
more label called callerID to this for my application.


Is it possible to modify the default metrics to include new labels ? 
If so, could you point me towards some documentation on how to do this ?




What is callerID?

If it is something to do with the end user calling a system then it 
would be strongly advised not to have such a label - things like user 
IDs, usernames or IP addresses have possibly infinite (or generally 
large) cardinality and can quickly cause issues.


However if it is something to do with the service/application itself the 
general approach would be to either:


1. Add such labels via relabeling in your scrape config

2. Not add the label at all, and instead use role metrics as described 
at https://www.robustperception.io/how-to-have-labels-for-machine-roles 
& 
https://www.robustperception.io/exposing-the-software-version-to-prometheus


--
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/e9c6773a-e1d2-f2aa-c2d3-0d9a355b7b7a%40Jahingo.com.

Re: [prometheus-users] Monitoring rate of events

[Stuart Clark]

On 08/09/2020 14:51, John Dexter wrote:
I'm looking at adopting Prometheus for our system, one thing we want 
to monitor are incoming alarms (and alarm clearances).


Metrics of use would be:

  * Total number of active alarms
  * Rate of alarms being raised
  * Rate of alarms being cleared

The second one is confusing me a bit. Prometheus will poll at repeated 
intervals, and we might change that interval. I can report the /total/ 
number of alarm-raises/alarm-clearances since the system started using 
Counter metrics but what I am interested in is the rate of change not 
the absolute value. I just want to check:


 1. Is rate of change a standard visualization for a metric e.g. a
graph of alarms-raised-per minute/hour/day?
 2. Our system runs for months at a time, I assume the maximum value
for a Counter is so astronomically high this is a non issue?
 3. Would it make sense to use alarms raised/cleared as labels on the
same time-series or should they be two totally separate time-series?

Is this the right way to monitor such a metric?

Prometheus / alertmanager expose metrics themselves, so if you scrape 
them you do get access to metrics about alerts.


You can then use the standard rate() function on the counters tracking 
alerts.


For (2) it is true that a counter has a large maximum value, but also 
rate() handles counter resets anyway, so isn't an issue even if a 
counter loops.



If on the other hand you are talking about an external metric relating 
to some sort of alarm then you'd generally want to use counters (and 
then using rate()). With regards to labels on a metric you want them to 
be providing additional granular ways of understanding a metric rather 
than for combining different metrics together. So alarms raised & alarms 
cleared are totally different metrics - one way to think of things would 
be to ask "would summing together all the different label combinations 
within a metric result in something sensible?", so for example a metric 
with labels for status code would make sense, as the sum is just the 
count of all the events, but raised/cleared doesn't really give much 
value to be summed together.




--
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/a7ca6f08-ea44-239e-7363-407419aca0dc%40Jahingo.com.

[prometheus-users] logging

[Andrew Meyer]

Is there a way to send snmp_exporter logs to a different file without 
having to parse it using rsyslog on CentOS or Syslog-NG?

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/0d903b60-fbd8-4c04-ba12-aa9a3edf8e3cn%40googlegroups.com.

Re: [prometheus-users] Re: Need help on Prometheus load testing for capacity planning

[rs vas]

Hey Brian, that link is great to start with, thanks for sharing it. Very
helpful.

If we had to test with some synthetic set of exporters with some
configurable metrics, Do we really have anything we can quickly set-up on
amazon ec2 and get that load tested? Just thought of finding out still... I
am sure many of you must have done it already on this forum...

On Tue, Sep 8, 2020 at 3:01 AM Brian Candler  wrote:

> I'd be inclined to take the opposite approach: use your production
> exporters, guess the initial instance size, and iterate based on how under-
> or over-utilized it is.   It only takes a minute to switch an instance from
> one type to another in EC2.
>
> As a starting point for your guess, you can use this:
>
> https://www.robustperception.io/how-much-ram-does-prometheus-2-x-need-for-cardinality-and-ingestion
>
> Your real metrics are going to be a better test than any synthetic set of
> metrics.  Remember also that you can scrape the same targets from multiple
> prometheus instances, so you can spin up a second prometheus for testing
> without touching the existing one, if you have one.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/1c6e391b-d06b-40fb-a088-67624b559596o%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAPs_Afhmbh9JmY8Rh697P%3DOUaSOrgZijYE6EmEVfkeN2XBn_1w%40mail.gmail.com.

Re: [prometheus-users] how to run prometheus on docker

[babis kagelidis]

my container log is 
"level=error ts=2020-09-08T17:06:51.460Z caller=main.go:283 msg="Error 
loading config (--config.file=/prometheus.yml)" err="read /prometheus.yml: 
is a directory"
level=error ts=2020-09-08T17:06:56.878Z caller=main.go:283 msg="Error 
loading config (--config.file=/prometheus.yml)" err="read /prometheus.yml: 
is a directory"
level=error ts=2020-09-08T17:06:59.519Z caller=main.go:283 msg="Error 
loading config (--config.file=/prometheus.yml)" err="read /prometheus.yml: 
is a directory"
level=error ts=2020-09-08T17:07:05.024Z caller=main.go:283 msg="Error 
loading config (--config.file=/prometheus.yml)" err="read /prometheus.yml: 
is a directory"
level=error ts=2020-09-08T17:07:12.431Z caller=main.go:283 msg="Error 
loading config (--config.file=/prometheus.yml)" err="read /prometheus.yml: 
is a directory"
level=error ts=2020-09-08T17:07:18.013Z caller=main.go:283 msg="Error 
loading config (--config.file=/prometheus.yml)" err="read /prometheus.yml: 
is a directory"
level=error ts=2020-09-08T17:07:40.173Z caller=main.go:283 msg="Error 
loading config (--config.file=/prometheus.yml)" err="read /prometheus.yml: 
is a directory"
level=error ts=2020-09-08T17:07:55.992Z caller=main.go:283 msg="Error 
loading config (--config.file=/prometheus.yml)" err="read /prometheus.yml: 
is a directory"
level=error ts=2020-09-08T17:08:19.886Z caller=main.go:283 msg="Error 
loading config (--config.file=/prometheus.yml)" err="read /prometheus.yml: 
is a directory"
level=error ts=2020-09-08T17:08:48.089Z caller=main.go:283 msg="Error 
loading config (--config.file=/prometheus.yml)" err="read /prometheus.yml: 
is a directory"
level=error ts=2020-09-08T17:09:40.235Z caller=main.go:283 msg="Error 
loading config (--config.file=/prometheus.yml)" err="read /prometheus.yml: 
is a directory"
"
what is going wrong?

Στις Τρίτη, 8 Σεπτεμβρίου 2020 στις 7:31:52 π.μ. UTC+3, ο χρήστης 
kira...@gmail.com έγραψε:

>  I had the same issue where containers are exiting the moment they are 
> created. It was because of permission issue to the directory where 
> Prometheus is writing data to.
>
> On Sunday, September 6, 2020, babis kagelidis  
> wrote:
>
>> Hello , i try to run prometheus on docker with this command :
>>
>> sudo docker run -d --name prometheus-server -p 9090:9090 \
>> -v /home/mccdev/prometheus/prometheus/prometheus.yml:/prometheus.yml \
>> prom/prometheus \
>> --config.file=/prometheus.yml 
>>
>> but container is always exited immediatelly and i have no access to 
>> localhost:9090.
>>
>> What can i do to set container to 'up'?
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to prometheus-users+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/prometheus-users/dbe8d871-d3c4-43ae-b6bc-d90104f5d84cn%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/003870a7-8905-4f6b-adfa-ef471436e059n%40googlegroups.com.

Re: [prometheus-users] any best practice on using limited le's for a given histogram

[Bjoern Rabenstein]

On 02.09.20 00:38, rs vas wrote:
> 
>   • any good number we can cross when defining buckets for example not to
> define more than 10 le's.

It all really depends on your total cardinality. It's fine to create a
histogram with loads of buckets if that's only exposed on three
targets and has no further labels at all.

In your case, where you have many hosts _and_ partitioning by a bunch
of other labels with some significant cardinality, too, you really
have to be careful with the number of buckets.

A common pattern for something like HTTP request metrics is to have a
counter with many labels (like method, path, status code, ...) and
then a histogram for the request duration with no further labels (or
at least only a few with low cardinality). In that way, you cannot
calculate latency per status code and such, but it might be a good
compromise.

In different news, I'm working on ways to allow high-res histograms in
the future, see
https://grafana.com/blog/2020/08/24/kubecon-cloudnativecon-eu-recap-better-histograms-for-prometheus/
for a bunch of links to talks etc.

-- 
Björn Rabenstein
[PGP-ID] 0x851C3DA17D748D03
[email] bjo...@rabenste.in

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/20200908205643.GA2356%40jahnn.

[prometheus-users] Is it possible to monitor Azure AD Connect healthy in windows server 2019?

[Khanh Nguyen Duc]

Can we monitoring Azure AD connect healthy status  by using prometheus + 
Grafana ?


-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/1cc76812-644b-494f-875c-de7b725687c4n%40googlegroups.com.

Re: [prometheus-users] Is it possible to monitor Azure AD Connect healthy in windows server 2019?

[Hieu Nguyen]

I think you should try windows-exporter.

On Wed, Sep 9, 2020 at 8:09 AM Khanh Nguyen Duc 
wrote:

>
> Can we monitoring Azure AD connect healthy status  by using prometheus +
> Grafana ?
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/1cc76812-644b-494f-875c-de7b725687c4n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CANv4vgTzZF_MUCd4LtpLzyqabH1j1vvk4p1Aur2xCzacpL6x2w%40mail.gmail.com.

Re: [prometheus-users] Re: Regarding Setting High Retension for storage

[Aravind Viswanathan]

Hi Brian,

Thanks for the help.

OK After i did that and restarting the services I see the following
under Command-Line Flags,

storage.tsdb.retention  : 0s
storage.tsdb.retention.size : 0B
storage.tsdb.retention.time: 1y

Is that correct or do I need to make any other configuration changes?
Regards
Aravind Viswanathan.


On Tue, Sep 8, 2020 at 5:57 PM Brian Candler  wrote:

> Not quite.  Since you're splitting the command line over multiple lines,
> you need a trailing space and backslash after "console_libraries" to join
> it to the next line.
>
>> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Prometheus Users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/prometheus-users/rj695eUHMQA/unsubscribe
> .
> To unsubscribe from this group and all its topics, send an email to
> prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/7d88121d-b793-4c3a-b840-2243ddf03086o%40googlegroups.com
> 
> .
>


-- 
Thanks

Aravind.V

*Mobile*:+91. 8197177674

*Email* : v_6a...@hotmail.com, arwa...@gmail.com

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAK1hgMhDDHHa9dCfKEQZOgPgswXbEnLTLa2SSu3bsY%3D03HWPdQ%40mail.gmail.com.