[prometheus-users] Re: Prometheus Operator and auto discovery of AlertManagers

Answering my own mail:) Autodiscovery of AlertManagers is not supported in the Prom custom resource. It requires a custom configuration. https://github.com/coreos/prometheus-operator/blob/master/Documentation/custom-configuration.md On Tuesday, March 31, 2020 at 3:54:10 PM UTC+2, Richard Moe wro

Re: [prometheus-users] SNMP_GENERATOR

Oh ! did you build the project? make sure to cd to the folder containing the binary or add it to you $PATH On Tue, Mar 31, 2020 at 7:45 PM yesmine kessentini < yesminekessentin...@gmail.com> wrote: > but when writing this command : which generator > the response is : > /usr/bin/which: no generato

[prometheus-users] Inhibits within before/after

I have a fairly simple setup with a handful of alerts from a specific host. If that host is down, I want to suppress these alerts. I'm able to do that with a straightforward inhibit. - source_match: alertname: 'HostMissing' equal: ['instance'] The problem comes when the host comes back. The

[prometheus-users] Re: Getting status of services in CentOS 6.

Hey Yagyansh, Sorry to hijack your post but I am wondering if you made this work in Cent OS, if you could help me with my issue with monitoring Systemd Services. Basically I need to add custom services to the --collector.systemd.unit-whitelist="(apache2|ssh|rsyslog|nginx).service" but not sure

Re: [prometheus-users] how can I monitor custom systemd services?

Hey Christian, Thanks for getting back to me. Yes I just ran this and looks active and running: root@aws-mtl-dev3:~# systemctl list-units |grep nuxt nuxt_site-a.service loaded active running Nuxt Vue JS Service nuxt_site-b.service

Re: [prometheus-users] probe_ssl_earliest_cert_expiry showing wrong expiry date

Hi Amjad, blackbox_exporter's probe_ssl_earliest_cert_expiry outputs exactly what the name says -- the *earliest* cert expiry, i.e. when this certificate will become invalid as seen from a user/browser/client validating this cert. This is not necessarily identical to the value of the end of validi

Re: [prometheus-users] Getting status of services in CentOS 6.

Hi, On 3/31/20 12:22 PM, Yagyansh S. Kumar wrote: > Hi. We have systemd collector in Prometheus that enables us to get the > metrics - status of services from systemd. This works well in CentOS 7. > But CentOS 6 does not have systemd, hence, the collector keeps throwing > error. Is there any alter

Re: [prometheus-users] how can I monitor custom systemd services?

Hi, On 3/31/20 10:56 PM, Joey Jojo wrote: > I have a node exporter setup and installed in a Linux Ubuntu server and > everything works fine. I've had to setup a few different custom SystemD > services located in /etc/systemd/system/ and I'd like to know how I can > whitelist them into the node_ex

[prometheus-users] how can I monitor custom systemd services?

Hello, I have a node exporter setup and installed in a Linux Ubuntu server and everything works fine. I've had to setup a few different custom SystemD services located in /etc/systemd/system/ and I'd like to know how I can whitelist them into the node_exporter.service which is also located in

[prometheus-users] Re: Dynamically naming histogram_quantile expressions in console graph

Defining name: ['p50', 'p90', ...] works. On Tuesday, March 31, 2020 at 12:38:55 PM UTC-6, Brad Walker wrote: > > Hello, > > I'm creating a console featuring a latency profile graph. I want this > graph to display p50, p90, etc. for a histogram. The names of the plot > lines for the several hist

[prometheus-users] Dynamically naming histogram_quantile expressions in console graph

Hello, I'm creating a console featuring a latency profile graph. I want this graph to display p50, p90, etc. for a histogram. The names of the plot lines for the several histogram_quantile PromQL expressions are 'undefined{}'. Is there something that can be supplied to the PromConsole.Graph's n

Re: [prometheus-users] Too many scrape configs on prometheus config file

Just replace your static_sd section with a file_sd section. Before: job_name: node static_configs: - targets: - foo:9100 - bar:9100 - baz:9100 After: job_name: node file_sd_configs: - files: - /etc/prometheus/targets.d/node_targets.yml

[prometheus-users] Re: AlertManager is not sending alert emails

First: change "receiver: default" to "receiver: team-emaillist". This will check whether the SMTP part is working properly, as all alerts should be sent out. Then start adding some matching routes. The route you have given will match only alerts which have both alertname: KubeNodeNotReady *an

Re: [prometheus-users] SNMP_GENERATOR

but when writing this command : which generator the response is : /usr/bin/which: no generator in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/go/bin) Le mar. 31 mars 2020 à 18:42, yesmine kessentini < yesminekessentin...@gmail.com> a écrit : > thanks , > > but i have

Re: [prometheus-users] SNMP_GENERATOR

thanks , but i have already this package , /usr/lib/libnetsnmp.so.35 /usr/lib/libnetsnmp.so.30 /usr/lib/libnetsnmp.so.30.0.3 /usr/lib64/libnetsnmp.so.31 /usr/lib64/libnetsnmp.so.31.0.2 /usr/lib64/libnetsnmp.so /usr/local/lib/libnetsnmp.so.35.0.0 /usr/local/lib/libnetsnmp.so.35 /usr/local/lib/libne

Re: [prometheus-users] SNMP_GENERATOR

This is a stackoverflow question ╭─sayf@degla ~ ╰─$ sudo apt-file search libnetsnmp.so libsnmp-dev: /usr/lib/x86_64-linux-gnu/libnetsnmp.so libsnmp30: /usr/lib/x86_64-linux-gnu/libnetsnmp.so.30 libsnmp30: /usr/lib/x86_64-linux-gnu/libnetsnmp.so.30.0.3 You need this package On Tue, Mar 31, 2020

Re: [prometheus-users] Too many scrape configs on prometheus config file

Hi, Thanks. I have used file_sd for my black box exporter but how to use this for prometheus config yml.can you point to some resource? Thank Eswar -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop re

Re: [prometheus-users] Too many scrape configs on prometheus config file

On 31 Mar 16:54, Stuart Clark wrote: > The general suggestion is to use a configuration management system such as > Ansible to create the configuration file. Equally you can use the different > service discovery mechanisms to simplify your configuration too, including > the file_sd to pull targe

Re: [prometheus-users] Too many scrape configs on prometheus config file

The general suggestion is to use a configuration management system such as Ansible to create the configuration file. Equally you can use the different service discovery mechanisms to simplify your configuration too, including the file_sd to pull targets into separate files (which themselves migh

Re: [prometheus-users] Re: Data retention policy

No that sounds fairly normal. One thing to note is that those timestamps are not the times the methods were called. They are when Prometheus scraped your application. So if you scrape once a minute the actual call could have been at any point during that minute. Equally if there are multiple cal

[prometheus-users] Alertmanager race condition with inhibited rules

Hi! I think this is mostly a configuration issue so i'm posting this here before github to see if someone can help me. I have a prometheus server configured in kubernetes with 2 alertmanagers in HA. (1 prometheus server and 2 AlertManagers). Alertmanager Configuration:

[prometheus-users] AlertManager is not sending alert emails

Hi, My alertmanager email config (below) is not working, could you share a sample(redacted) that works for you. global*:* smtp_smarthost*: **mail.* *abcxyz.com* smtp_from*: alertmana...@abcxyz.com* resolve_timeout*: 5m* route*:* group_wait*: 30s*

[prometheus-users] Alert emails not being sent

Hi, I have the following alert manager config setup, but it's not giving the expected results like sending the alert emails. Is there something missing here? openshift_cluster_monitoring_operator_alertmanager_config*: |+* global*:* smtp_smarthost*: **mail.* *abc.com

Re: [prometheus-users] Too many scrape configs on prometheus config file

Typically people use a configuration management software. For example Ansible is pretty nice. https://github.com/cloudalchemy/ansible-prometheus On Tue, Mar 31, 2020 at 4:59 PM Ishvar B wrote: > Hi, > > I have a prometheus set up which has too many (roughly 50 static configs) > and this number

[prometheus-users] Too many scrape configs on prometheus config file

Hi, I have a prometheus set up which has too many (roughly 50 static configs) and this number would increase as we have lot of vms still be scraped for prometheus. The file is becoming huge , especially when some applications require basic auth. is there any best practice to maintain the config

Re: [prometheus-users] Re: How to solve "Two hour in-memory prometheus data during upgrade/failover"

Another option is to configure Prometheus instances to replicate data to remote storage via remote_write . Prometheus replicates data to the configured remote storage systems as soon as the data is scraped, so i

Re: [prometheus-users] Re: Data retention policy

In our code, we are using a counter to count the accesses to the various methods of the API. We have one counter per method. We do not store the timestamp. But when we ask Prometheus with a "query_range" (see request below), it returns the list of all the methods that have been accessed. curl

[prometheus-users] Tabular representation of %Deviation

Hey, The requirement I have is to have a table with columns as below I'm using Grafana for visualisation with a 'Table' panel Metric Name Avg value of M1 b/w t1 and t2 Avg value of M1 b/w t3 and t4 %Increase/Decrease ( A2-A1/A1) M1A1

[prometheus-users] SNMP_GENERATOR

HI , i have this error while writing ./generator generate ./generator: error while loading shared libraries: libnetsnmp.so.35: cannot open shared object file: No such file or directory Can you help me in finding where is the issue? -- You received this message because you are subscribed to

[prometheus-users] Re: Couldn't able to set up the JMX exporter to scrape the JVM metrics.

Exception in thread "main" java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Delegat

[prometheus-users] Prometheus Operator and auto discovery of AlertManagers

Hi! I am struggling a bit to figure out how to correctly configure autodiscovery of AlertManagers in a Kubernetes cluster. Anyone have a fully working example Prometheus custom resource of this? Just putting example config from Prometheus doc doesn't seem to work as the resource expects a list

[prometheus-users] probe_ssl_earliest_cert_expiry showing wrong expiry date

I checked the cert chain and nothing is expiring. Host operating system: output of uname -a Linux blackbox-deployment-fbd566f88-l6b7d 4.14.165-133.209.amzn2.x86_64 #1 SMP Sun Feb 9 00:21:30 UTC 2020 x86_64 GNU/Linux blackbox_exporter versi

Re: [prometheus-users] Re: Data retention policy

How are you storing the timestamp? Is that in a label or a metric value as the last call to the API? In general these are sounding like you are trying to store events within Prometheus rather than metrics. Normally you'd not have a timestamp but a counter of the number of calls to the API. On

Re: [prometheus-users] Re: Data retention policy

Le lundi 30 mars 2020 16:37:11 UTC+2, Brian Candler a écrit : > > On Monday, 30 March 2020 09:34:01 UTC+1, REMI DRUILHE wrote: >> >> In our context, Prometheus is storing system metrics and business >> metrics, especially the number of accesses to the methods of our API. >> >>> > That presu

[prometheus-users] Getting status of services in CentOS 6.

Hi. We have systemd collector in Prometheus that enables us to get the metrics - status of services from systemd. This works well in CentOS 7. But CentOS 6 does not have systemd, hence, the collector keeps throwing error. Is there any alternative for this for CentOS 6? Also, in my 6 VMs the URL h

[prometheus-users] Re: Too stupid for quantile_over_time

quantile_over_time(0.5, foo[t]) returns some value X, where 50% of the values in foo over the time range are below X and 50% are above X (i.e. "the median value"). Think of it as: sort all the values from low to high, and pick the middle one. quantile_over_time(0.95, foo[t]) returns some value

[prometheus-users] Re: Prometheus exporters for scraping ELK metrics

just checking if anyone has any input? On Sun, Mar 29, 2020, 8:47 PM rs vas wrote: > Hello, I was looking for some exporters where it can easily scrape ELK > metrics into Prometheus. > > I found below exporters for the same: > Elasticsearch: https://github.com/justwatchcom/elasticsearch_exporter

Re: [prometheus-users] IOT architecture with prometheus; mqtt, aws, pushgateway

hi A R, I haven't set it up myself yet, but was thinking about the same use case. What I've found is that there are mqtt/prome gateways - that work kind of like pushgateway: https://github.com/inuits/mqttgateway https://github.com/Svedrin/mqtt-pushgateway the general idea is that the exporter su