[prometheus-users] Alerting on 2 consecutive failures - daily Jenkins job - default_jenkins_builds_last_build_result

Hi We have an automatic Jenkins job that runs early every morning. If the job is failed, i.e. red, it is not a big problem. But if it is red 2 times in a row, an alert has to be triggered and a human has to look at it. Here what I have done: --- XXX-Job-Status (1 active) name: XXX-Job-Stat

Re: [prometheus-users] Re: amtool: error: [GET /alerts][400] getAlertsBadRequest matcher value contains unescaped double quote: instance=~".+rem-prod.+" team="OMNR2"

On 06.06.22 19:11, Brian Candler wrote: I suggest you try simply removing the double quotes from your amtool query: Thanks. That solved the issue. kind regards Evelyn -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from th

[prometheus-users] amtool: error: [GET /alerts][400] getAlertsBadRequest matcher value contains unescaped double quote: instance=~".+rem-prod.+" team="OMNR2"

Hi I get this error with "amtool" amtool: error: [GET /alerts][400] getAlertsBadRequest matcher value contains unescaped double quote: instance=~".+rem-prod.+" team="OMNR2" This the relevant snippet QUERY_STRING='instance=~".+rem-prod.+" team="OMNR2"' COUNTER=0 until [[ -z $ALERTS ]] do

Re: [prometheus-users] context deadline exceeded

On 09.04.22 19:16, BHARATH KUMAR wrote: Hi all, Hi Kumar I am installing node exporter on many servers and I a monitoring those servers using prometheus. Initially i didn't face any kind of errors. but on few servers I am facing the error "Get "http://some_ip:9100/metrics": context deadli

[prometheus-users] Jenkins job runs once a day: if it is on failed 2 times in a row, we want an alarm

Hi I am using https://plugins.jenkins.io/prometheus. This plugin unfortunately has no documentation at all and I don't know what the metrics mean. I have a Jenkins job that runs automatically and daily. If it fails 2 times in a row, I want an alarm. Are there more elegant ways to monitor J

[prometheus-users] Jenkins job runs once a day: if it is on failed 2 times in a row, we want an alarm

Hi I am using https://plugins.jenkins.io/prometheus. This plugin unfortunately has no documentation at all and I don't know what the metrics mean. I have a Jenkins job that runs automatically and daily. If it fails 2 times in a row, I want an alarm. Are there more elegant ways to monitor J

Re: [prometheus-users] Best practice for storing hundreds of alert rules

On 22.10.21 09:31, jthunder wrote: Hi guys, We use Prometheus to monitor K8s clusters and applications on them. We have create 500-800 alert rules to be applied to Prometheus. But we don't find any doc about how to arrange these rules. Why you need 800 alert rules? Maybe write the rules more

Re: [prometheus-users] Prometheus is now available on AWS. Has anyone from the project been involved?

Hi Julien thank you, that's very good if you are satisfied with AWS. Quite a few projects and small companies complain that AWS exploits them and gives nothing back. So that is not the case with Prometheus. kind regards Evelyn -- You received this message because you are subscribed to the Go

[prometheus-users] Prometheus is now available on AWS. Has anyone from the project been involved?

Hi Has there been any contact between AWS and the Prometheus project? Have any developers from Prometheus been involved? How is the collaboration? Amazon Managed Service for Prometheus Is Now Generally Available with Alert Manager and Ruler https://archive.is/aSY0w kind regards Evelyn --

Re: [prometheus-users] Moderation for new subscribers?

On 03.10.21 18:06, Ben Kochie wrote: Maybe we can get Google to fix their filters for this. If you will ask Google for support, you will get an answer from a useless bot. kind regards Evelyn -- You received this message because you are subscribed to the Google Groups "Prometheus Users" gro

Re: [prometheus-users] Moderation for new subscribers?

On 03.10.21 11:59, sayf eddine Hammemi wrote: +1 also +1 from me Meanwhile I just set up a gmail filter to automatically delete them. Please don't use Gmail, see this list https://www.privacyguides.org/providers/email/ for better alternatives. kind regards Evelyn -- You received this me

[prometheus-users] label "aksv0a7h237av_58_98_enq_TX_row_lock_contention_" not accepted by Prom (or Pushgateway)

Hi We use Telegate (is it now called Prometheus Pushgateway?) to send output from cronjob. This the relevant part of shell script: cat

Re: [prometheus-users] [ANN] PromQL Prettier

On 18.09.21 17:42, Jiacai Liu wrote: Have trouble in reading PromQL written by your colleagues or SRE team? promql-prettier to the rescue! - https://github.com/jiacai2050/promql-prettier Any issues or feedbacks are welcomed. Thanks. Hi Very nice! Thanks I would send out the mails also to

Re: [prometheus-users] Monitoring comparision

On 17.09.21 05:49, KrantZ wrote: Can someone has a better idea if AppDynamics can provide better monitoring metrics than Prometheus for an application which is developed in Spring boot App. Can Prometheus has advantage over AppD related to a Spring boot metrics. I have an opinion but just wante

[prometheus-users] query Prometheus /api/v1/query and decide based on output the actions

Hi We would like to start a job within K8s. We start this job based on the value of a metric. The Prometheus server is outside of K8s. Is this procedure correct? Using curl to query the HTTP API of Prometheus and then, based on output start a job or not? Are there other possibilities? Is t

Re: [prometheus-users] Re: mute_time_intervals in Alertmanager

On 10.09.21 09:36, Brian Candler wrote: I already saw the YAML that you posted (with bad formatting, but I suspect that's a problem with google groups). Hi Brian yes Google bug: https://www.mail-archive.com/prometheus-users@googlegroups.com/msg08744.html kind regards Evelyn -- You received

Re: [prometheus-users] Re: mute_time_intervals in Alertmanager

On 09.09.21 15:26, Brian Candler wrote: But to answer your original question, you *can* match on year in mute_time_intervals if you really want, and the documentation shows how: https://prometheus.io/docs/alerting/latest/configuration/#time_interval Hi Brian please example YML. Thank you. ki

Re: [prometheus-users] Re: mute_time_intervals in Alertmanager

On 09.09.21 15:23, Brian Candler wrote: For both planned and ad-hoc maintenance, you would be better off creating "Silences" in alertmanager, via its own web interface or via one of the alertmanager frontends like karma or alerta.io Hi Brian our team decided we wish to have this info in Git.

[prometheus-users] mute_time_intervals in Alertmanager

Hi mute_time_intervals: - name: Release time_intervals: - months: ["august"] days_of_month: ["25", "26", "27"] times: - start_time: "20:00" end_time: "23:00" - months: ["september"] days_of_month: ["25", "26", "27"] times:

[prometheus-users] enable Ehcache metrics for Prom?

Hi How to get metrics from https://www.ehcache.org (Java)? > Ehcache is an open source, standards-based cache that boosts performance, offloads your database, and simplifies scalability. It's the most widely-used Java-based cache because it's robust, proven, full-featured, and integrates with

Re: [prometheus-users] Re: wildcards not working in alertmanager.yml

On 30.08.21 13:35, Brian Candler wrote: "match" matches exact strings.  You need "match_re" if you want to match regular expressions. thank you from docs: # DEPRECATED: Use matchers below. # A set of regex-matchers an alert has to fulfill to match the node. match_re: I need this? # A list o

[prometheus-users] wildcards not working in alertmanager.yml

Hi routes: - match: channel: iii receiver: yyy # mute_time_intervals: #- Maintenance - match: cluster: devtest receiver: "null" - match: cluster: poc receiver: "null" (...)

[prometheus-users] "global" excludes in rules

Hi I just copy those alerts https://awesome-prometheus-alerts.grep.to/rules#kubernetes I not wish to modify the individual alert (query). I wish to exclude globally some namespaces and pods. Is it possible to exclude some stuff, but not modify the individual query I got from upstream? kin

Re: [prometheus-users] Re: mute_time_interval examples (for maintenance window)

On 20.08.21 01:29, Benjamin Ridley wrote: I hope that helps! Thank you very much. Exactly what I am looking for. Only 1 small problem I not know how-to solve: start_time: '22:00' end_time: '02:00' (next day) have a great day kind regards Evelyn -- You received this message because you are

[prometheus-users] mute_time_interval examples (for maintenance window)

Hi I need some mute_time_interval examples. Beginning of the year we define every month 3x maintenance window. The dates different for every month. The time the same. I wish that in those 3x maintenance window every month no alerts going out. Can you provide me examples how-to implement that

[prometheus-users] Are there news/updates about "Prometheus ecosystem certification"

Hi Are there any follup/news/updates? https://www.mail-archive.com/prometheus-developers@googlegroups.com/msg00803.html thank you and a happy day kind regards Evelyn -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from thi

Re: [prometheus-users] Re: add an alias for label

On 09.08.21 19:19, Brian Candler wrote: Labels are single valued. ok. Thanks. kind regards Evelyn -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometh

[prometheus-users] add an alias for label

Hi I add labels to each target. This way i can filter the targets with labels in the alerts. Now i want to add two values for a label (something like an alias). for a query i should be able to filter by both values. No matter if i take the orginal label or the alias, the result should be the

Re: [prometheus-users] Prometheus basic auth support

On 02.08.21 12:26, Draven Zuo wrote: Can we set a username and password for prometheus server instead of introducing a reverse proxy? Does this answer your question? https://prometheus.io/docs/guides/basic-auth/ Prometheus supports basic authentication (aka "basic auth") for connections to t

[prometheus-users] GitHub Copilot helps editing prometheus.yml?

Hi Did anybody try the hyped GitHub Copilot? Sorry I have no GitHub account, can't test. Does it help editing prometheus.yml? I mean not only writing code, but also manifests. best regards Evelyn -- You received this message because you are subscribed to the Google Groups "Prometheus Users

Re: [prometheus-users] http://prometheus.example.com/prom/config

On 09.06.21 22:04, Bjoern Rabenstein wrote: On 07.06.21 15:27, 'Evelyn Pereira Souza' via Prometheus Users wrote: I edit of my prometheus.yml and is no longer formatted so nicely. Can I copy from prometheus.example.com/prom/config to the prometheus.yml and nothing is lost?

[prometheus-users] http://prometheus.example.com/prom/config

Hi I edit of my prometheus.yml and is no longer formatted so nicely. Can I copy from prometheus.example.com/prom/config to the prometheus.yml and nothing is lost? Say it the identical config as in my file? This webUI endpoint pretty print the config? kind regards Evelyn -- You received this

[prometheus-users] Getting data out from Prometheus (2021)

Hi I found groups.google.com/g/prometheus-developers/c/vAiOnEmbTlg and wonder if there is an update for 2021. My use case is that other department wish the values of queries in an Excel (for visualisation). query: quantile_over_time(0.5, probe_duration_seconds{team="zzz",env="yyy",job="xx

[prometheus-users] Error on ingesting out-of-order with k8s Federation

Hi My log file is full with messages like this level=warn ts=2021-05-14T05:23:49.382Z caller=scrape.go:1372 component="scrape manager" scrape_pool=federate target="https://prometheus.qna-prod-in.k8s.example.org:9090/federate?match%5B%5D=%7Bjob%3D%22ambassador-admin%22%7D&match%5B%5D=%7Bjob%3D%

Re: [prometheus-users] [ANN] pint - Prometheus rule linter

Hi thanks. How to install that? Is there a brew formula? There is nothing about it in README.md and also no releases. kind regards Evelyn -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving e

Re: [prometheus-users] How-To debug prometheus_rule_evaluation_failures_total? Prometheus is failing rule evaluations

On 23.04.21 20:35, Matthias Rampke wrote: It seems like you are federating through an ingress or load balancer that balances over multiple Prometheus server replicas. Either federate from each separately, or make sure that you only get responses from one consistently. As an alternative to

Re: [prometheus-users] How-To debug prometheus_rule_evaluation_failures_total? Prometheus is failing rule evaluations

On 22.04.21 20:20, Matthias Rampke wrote: Your best starting point is the rules page of the Prometheus UI (:9090/rules). It will show the error. You can also evaluate the rule expression yourself, using the UI, or maybe using PromLens to help debug expression issues. /MR :9090/rules show th

[prometheus-users] How-To debug prometheus_rule_evaluation_failures_total? Prometheus is failing rule evaluations

Hi We have those alerts constantly: {group="local", instance="localhost:9090", job="prometheus", rule_group="/etc/config/prometheus-rules.yml;node.rules"} 10 {group="local", instance="localhost:9090", job="prometheus", rule_group="/etc/config/prometheus-rules.yml;prometheus"} 10 Source: ht

[prometheus-users] no docs for github.com/jenkinsci/prometheus-plugin

Hi We have some jobs on a shared jenkins (operated by other team). We use https://github.com/jenkinsci/prometheus-plugin. There is sadly no docs. I checked the metrics by greping for our jobname, but don't know what they mean curl -s -u $user:$pw https://jenkins.example.com/prometheus -L |

Re: [prometheus-users] SunOS 5.10 architecture: sparc Access monitoring

On 14.04.21 09:41, Shuai Shao wrote: Machine parameters that need to be connected to monitoring Release: 5.10 Kernel architecture: sun4u Application architecture: sparc Hardware provider: Sun_Microsystems Domain: Kernel version: SunOS 5.10 Generic_144488-17 --

Re: [prometheus-users] How to connect a simple REST API to Prometheus

On 06.04.21 09:58, Shutima Potivorakun wrote: Hello all, I am new to Prometheus. I need to learn how to configure Prometheus, therefore I setup Prometheus and Grafana locally using docker-compose. Then I wrote a simple REST API that display some text on the browser. Now, I would like to conne

Re: [prometheus-users] fix yamllint errors specific for Prometheus.yml

On 24.03.21 14:37, David Leadbeater wrote: I’ve not used it but https://github.com/jumanjihouse/pre-commit-hook-yamlfmt looks like it could be useful for you. Thank you. This tool did very well the job and the result was compatible with

Re: [prometheus-users] fix yamllint errors specific for Prometheus.yml

On 23.03.21 14:45, Matt Zagrabelny wrote: $ cat ~/.config/yamllint/config rules:     line-length: disable I'm guessing there may be config options for the other warnings, too. -m Hi Thanks. But I wish not to exclude the error, I wish to automatically fix the errors with a tool. kind regar

[prometheus-users] fix yamllint errors specific for Prometheus.yml

Hi promtool check config doesn't complain, but yamllint prints several issues: https://cryptpad.fr/pad/#/2/pad/view/icFsx9ZE8U5LHYWyMAvIUbZxsvnpFL34eL4hI0wVfFg/embed/ warning missing document start "---" (document-start) wrong indentation: expected 2 but found 0 (indentation) line too long

Re: [prometheus-users] Enterprises using Prometheus

On 20.03.21 14:45, Stuart Clark wrote: Personally I try to keep things more separated, so using different storage for each team, which makes maintenance easier and allows each team to control their own configuration, at the cost of a bit more complexity/infrastructure. We do the same here. Bu

Re: [prometheus-users] Support Forums for Prometheus

On 17.03.21 17:12, Ben Kochie wrote: Which is done, you can now test out https://discuss.prometheus.io . Thanks. I did register but never received the verification mail. Can you please provide some details how you host it? AWS? Container? Deployment manifests pu

Re: [prometheus-users] Support Forums for Prometheus

On 16.03.21 17:18, Greg Beyer wrote: Second, Google groups participation, requires a Google account and the privacy implications that entails. I fully agree on the privacy issues with Google and other Big Tech, but there is no Google account required. See https://webapps.stackexchange.com/a/

Re: [prometheus-users] Support Forums for Prometheus

On 16.03.21 17:22, Julien Pivotto wrote: Hello Greg, We are evaluating multiple solutions at the moment, which include discourse and github discussions. We have not taken any decision yet in any direction, but those discussions are happening. We hope to bring that as a topic soon in one of our

[prometheus-users] monitor NFS mounts (despite NFS is in collector.filesystem.ignored-fs-types)

Hi Some team (outside of our control) installed node_exporter and excluded NFS mounts /usr/bin/node_exporter --collector.textfile.directory=/var/lib/node_exporter/textfile_collector --collector.filesystem.ignored-fs-types=^(autofs|binfmt_misc|bpf|cgroup2?|cifs|configfs|debugfs|devpts|devtmpf

Re: [prometheus-users] What is a sample? Why different number of scrape_samples_scraped for node_exporter VMs?

On 09.03.21 13:43, Stuart Clark wrote: How much does the value differ between the VMs? I did a vimdiff of /metrics endpoint and it's because there is a different number of CPUs in the VMs thanks for help kind regards Evelyn -- You received this message because you are subscribed to the Goo

Re: [prometheus-users] What is a sample? Why different number of scrape_samples_scraped for node_exporter VMs?

On 09.03.21 13:32, Stuart Clark wrote: Different node exporter settings, different number of filesystems, network interfaces, etc.? Hi The node_exporter settings are exactly the same (Puppet rollout). Only nr. of filesystems differ, but only 1-2 more fs. kind regards Evelyn -- You received

[prometheus-users] What is a sample? Why different number of scrape_samples_scraped for node_exporter VMs?

Hi What is a sample exactly? Why different number of scrape_samples_scraped for node_exporter VMs? kind regards Evelyn -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send

Re: [prometheus-users] Show scrape errors in prometheus metrics

On 06.03.21 11:45, Ben Kochie wrote: Yes, this is what the `up` metric provides. There's also `scrape_duration_seconds` that provides the time it took to perform the scrape. This makes it easier to see timeouts Hi a few additions from https://www.omerlh.info/2019/03/04/keeping-prometheus-in-s

Re: [prometheus-users] Blackbox return probe_success 0, but the website is up

On 05.03.21 10:27, Mohamed Boukhlif wrote: Hi everyone, I'm getting probe_success 0 but my site is up, any help please. Here is my config : (...) *prometheus.yml*   - job_name: 'blackbox'     metrics_path: /probe     params:       module: [http_2xx]     static_configs:       - targets:

[prometheus-users] Tru64 exporter (Digital UNIX)

Hi Is there a Tru64 exporter? It's Alpha processor. I coudn't find one on GitHub.. Unfortunately, we still have some Tru64 machines running. The specialists for this are retired. kind regards Evelyn -- You received this message because you are subscribed to the Google Groups "Prometheus U

[prometheus-users] How do I test blackbox exporter setup for Tomcat apps (running on Kubernetes/Istio)?

Hi Soon we will migrate from classic VMs to containers (k8s). In the first step, I integrated all Tomcat apps into the blackbox exporter. Later I will deploy Prometheus Operator. Blackbox config - parse with Regex the body - check for non 200er - POST a document and see http return code -

[prometheus-users] Monitoring Systems and Services with Prometheus (LFS241)

Hi Anyone took LFS241 from Linux Foundation? Did you like it? Not cheap $299. best regards Evelyn -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to promethe

[prometheus-users] What is Rules "Evaluation Time" and "Last Evaluation"? Do I need to invest time into query tuning?

Hi What is Rules "Evaluation Time" and "Last Evaluation"? Dashboard: https://example.org/prom/new/rules Most is some value like "0.xxxms" which seems fast But this node_systemd_unit_state{instance=~".+yyy.+",name=~"bbb.+",state="failed"} == 1 is 3.635ms and probe_ssl_earliest_cert_expiry{m

[prometheus-users] plain text mails in alertmanager

Hi This is my current alertmanager config: - name: 'mail_example.org' email_configs: - to: 't...@example.org' headers: subject: "Example: {{ .GroupLabels.alertname }} is {{ .Status }}" send_resolved: true It's sending out HTML formated mails. I wish to send out plain text mai

[prometheus-users] promtool check config prometheus.yml equivalent for blackbox_exporter

Hi I really like # promtool check config prometheus.yml because it detects errors in syntax, etc Is there an equivalent for blackbox_exporter? Checking the config without deploying the changes? regards Evelyn -- You received this message because you are subscribed to the Google Groups "Pr

Re: [prometheus-users] Re: Google SRE book "The Four Golden Signals" with blackbox_exporter

On 05.11.20 18:47, Brian Candler wrote: What you care about is the proportion of those real user requests which give errors, and the latency of those real user requests.  These measurements are *much* more useful than synthetic measurements from blackbox exporter. Thank you for explanation. I

[prometheus-users] Google SRE book "The Four Golden Signals" with blackbox_exporter

Hi https://landing.google.com/sre/sre-book/chapters/monitoring-distributed-systems/ scroll to "The Four Golden Signals" Latency -> my solution: avg_over_time(probe_duration_seconds{"}[15m]) > zz' Traffic -> not possible with blackbox Errors -> my solution: avg_over_time(probe_success{yyy}[20m

[prometheus-users] still running wmi_exporter v0.7.0 from Mar 13, 2019 in production - what are the risks?

Hi We have a few hundred legacy windows servers where we manually installed wmi_exporter v0.7.0. https://github.com/prometheus-community/windows_exporter/releases/tag/v0.7.0 ¨ The application running on windows is also legacy. Now the project has been renamed and has several breaking changes

[prometheus-users] Prometheus ecosystem certification

Hi Prometheus is also under the roof of CNCF. For Kubernetes there are several prestigious certifications. Nice is that there are separate courses for admins and developers. Is something like this planned for Prometheus? Has something like this already been discussed? I would be very happy