Re: [prometheus-users] Send resolved without slack

2021-05-13 Thread Sebastian Glock
Ok so this one should look like this: - alert: CPU load is more than 80%! expr: 100 - (1 -avg(irate(windows_cpu_time_total{mode="user"}[10m])) by (instance)) * 100 >= 40 for: 10s labels: severity: "[High]" annotations: summary: "CPU load is more than 80%!" descr

[prometheus-users] Rule msg: "Error on ingesting results from rule evaluation with different value but same timestamp"

2021-05-13 Thread Umut Cokbilir
Hi All, What is the reason for getting the errors I mentioned below? I'm not seeing any mistake in my group rules. If possible, can you check the configuration I shared below? Thanks Error samples: level=warn ts=2021-05-14T01:52:59.075Z caller=manager.go:654 component="rule manager" group=rul

Re: [prometheus-users] How-To debug prometheus_rule_evaluation_failures_total? Prometheus is failing rule evaluations

2021-05-13 Thread 'ping...@hioscar.com' via Prometheus Users
We are facing the issue where rules fail sporadically time to time. Are these errors logged somewhere if they cannot be found on UI? Thanks On Saturday, May 1, 2021 at 11:01:49 AM UTC-4 matt...@prometheus.io wrote: > That looks good, I think the issue is which target(s) you discover for > these

[prometheus-users] Need help in querying multiple metrics at the same time

2021-05-13 Thread gokul krish
I want to run 100s of prometheus range vector queries with different metric names as well as different labels and value. I want to fetch the data for last one hour for all queries, at once For example: calls_per_min{account="se", service="serv1"}[1h] error_per_min{account="me", host="serv2"}[

[prometheus-users] Using externalLabels with federation

2021-05-13 Thread Joe Conrady
Hello, We have 4 stand-alone prometheus instances spread across different data-centers. We have one of them federating the other 3, with externalLabels set to "dc:". This setup is mostly for the ease of graphing, querying and alerting. For example, if I want to see the different latencies to

[prometheus-users] Configuring prometheus to scrape only one pod of a service

2021-05-13 Thread anurag
Hi All, We have a kubernetes service which runs on multiple pods and is scraped by prometheus at the default metrics endpoint /service/metrics. In addition to existing metrics, we need to have a custom metrics which is generated by querying the DB, we would like this custom metrics be scraped

Re: [prometheus-users] prometheus with future data points

2021-05-13 Thread Johny
Thats odd. I confirmed remote read is returning future data points to Prometheus. But I can't see the data points in Prom UI. The series is truncated to 'now'. On Monday, May 10, 2021 at 7:44:20 AM UTC-4 Julien Pivotto wrote: > On 07 May 09:42, Johny wrote: > > Hi, > > Could you please look int

Re: [prometheus-users] How to add label for alert sent to specified Alertmanager

2021-05-13 Thread Morven Cao
thanks Brian, for the information. I will check if setting the cluster label on all alerts is acceptable to our team. -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an

Re: [prometheus-users] How to add label for alert sent to specified Alertmanager

2021-05-13 Thread Brian Brazil
On Thu, 13 May 2021 at 10:24, Morven Cao wrote: > Thanks Brian, > Let me try to explain. > > I setup one prometheus with two Alertmanagers in my k8s cluster, one of > the two Alertmanagers is in-cluster and configured by dynamic discovery, > while another Alertmanager is out of the cluster and it

[prometheus-users] is there a tool to generate sounds of metrics automatically

2021-05-13 Thread nina guo
hi may I ask if there is a tool for metrics generation on Prometheus? -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegr

Re: [prometheus-users] How to add label for alert sent to specified Alertmanager

2021-05-13 Thread Morven Cao
Thanks Brian, Let me try to explain. I setup one prometheus with two Alertmanagers in my k8s cluster, one of the two Alertmanagers is in-cluster and configured by dynamic discovery, while another Alertmanager is out of the cluster and it is statically configured via remote URL. What I'm try to

[prometheus-users] Issue with jmx export to monitor the jboss

2021-05-13 Thread Janarthanan Jaganathan
-- Forwarded message - From: Mail Delivery Subsystem Date: Wed, 12 May 2021, 5:22 pm Subject: Delivery Status Notification (Failure) To: [image: Error Icon] Address not found Your message wasn't delivered to *prometheus-u...@googlegroups.com* because the address couldn't be foun

Re: [prometheus-users] How to add label for alert sent to specified Alertmanager

2021-05-13 Thread Brian Brazil
On Thu, 13 May 2021 at 08:29, Morven Cao wrote: > I have a case in which one prometheus configured with two Alertmanagers(A > and B). > I want to add additional label for alert send to Alertmanager A, but not > for alert sent to Alertmanager B. > Can you explain more about what you're trying to

[prometheus-users] How to add label for alert sent to specified Alertmanager

2021-05-13 Thread Morven Cao
I have a case in which one prometheus configured with two Alertmanagers(A and B). I want to add additional label for alert send to Alertmanager A, but not for alert sent to Alertmanager B. I know I can add *external_labels* to prometheus configuration, but the label wil be added to alerts for b

[prometheus-users] Absent in prometheus config.

2021-05-13 Thread Sebastian Glock
Hi, I have a problem with absent hours in prometheus: hour(v=vector(time()) instant-vector) returns the hour of the day for each of the given times in UTC. Returned values are from 0 to 23. minute(v=vector(time()) instant-vector) returns the minute of the hour for each of the given times in UT