Re: [prometheus-users] Re: (Alertmanager) Ignore instance label to prevent same alert multiple times

Unfortunately you cannot have it both ways: either Alertmanager knows about separate alert instances that can be silenced separately, or it doesn't. I would try to eliminate the need to silence by Prometheus, for example by making the alert expressions resistant to gaps in the data. /MR On Fri,

[prometheus-users] Count file upload sizes in app

I'm new to Prometheus and I'm using it in conjunction with Grafana and Django. Simple metrics are easy enough to make without knowing much. Now I'm having trouble with collecting data about files being uploaded. In my application the user can upload files up to 50MB so it would be nice if I coul

Re: [prometheus-users] Thanos/Prometheus Query

On Fri, Jun 5, 2020 at 5:01 PM Bartłomiej Płotka wrote: > I would say it's best to go to #thanos Slack (CNCF Slack workspace) > with those questions. We have more than > 1k users there willing to answer some of those! > > TL;DR > 1. Why you need that? Why not putt

[prometheus-users] kubernetes_sd_configs for nodes

Hi everyone, I am running node exporter as a pods in k8s and exposing the metrics on port 11100 for each node. When I try to use kubernetes_sd_configs for nodes, it returns the node IP + kublet port (10250). ``` prometheus.yml - job_name: gke-nodes kubernetes_sd_configs: - role: n

Re: [prometheus-users] Thanos/Prometheus Query

I would say it's best to go to #thanos Slack (CNCF Slack workspace) with those questions. We have more than 1k users there willing to answer some of those! TL;DR 1. Why you need that? Why not putting Alertmanager to the central location instead? Alertmanager is to

Re: [prometheus-users] Re: Prometheus / Grafana - How to best display counters in a Singlestat panel?

Grafana introduced $__range for Prometheus. See https://stackoverflow.com/a/54776593/1244605 On Friday, January 12, 2018 at 7:34:11 PM UTC+2, Ben Kochie wrote: > > I don't remember the exact timing, but the $__interval values, and > behavior, in Grafana were improved in the last couple release

Re: [prometheus-users] Ways to Mitigate on huge label value pair during high cardinalities

Hi Aliaksandr, We will potentially consider Victoria Metrics in the future, Between how does increasing scrape interval can reduce RAM usage and any other resource level optimization can are also would be off great help. Regards Dinesh On Fri, 5 Jun, 2020, 6:09 pm Aliaksandr Valialkin, wrote:

Re: [prometheus-users] Ways to Mitigate on huge label value pair during high cardinalities

On Fri, Jun 5, 2020 at 2:55 PM Dinesh N wrote: > Hi Aliaksandr, > > Thanks for the valuable insights.. > > I shall take a look at bomb-squad in the meanwhile do you foresee any > generic options optimizations using metrics_relabel_configs or by using > sample_limit can help to reduce the cardinal

[prometheus-users] Thanos/Prometheus Query

Hi Team, First Let me give you the background of the prometheus architecture in our environment in which we have Thanos at central location and Prometheus on local sites which will be sending data to Thanos Querier. Pls advice/suggest on the below queries 1) Is it possible that the alert gene

Re: [prometheus-users] Ways to Mitigate on huge label value pair during high cardinalities

Hi Aliaksandr, Thanks for the valuable insights.. I shall take a look at bomb-squad in the meanwhile do you foresee any generic options optimizations using metrics_relabel_configs or by using sample_limit can help to reduce the cardinalities. Time series - Currently we have close to 8 million t

Re: [prometheus-users] jmx_exporter's MBean-level fetching well-intentioned but preventing required telemetry [feature request discussion]

On Fri, 5 Jun 2020 at 12:31, Cameron Kerr wrote: > > > On Friday, 5 June 2020 22:31:13 UTC+12, Brian Brazil wrote: >> >> On Fri, 5 Jun 2020 at 11:13, Cameron Kerr wrote: >> >>> Hi all, I've spent a few days coming up to speed on understanding >>> jmx_exporter, and I think I'm in a pretty good pl

Re: [prometheus-users] Ways to Mitigate on huge label value pair during high cardinalities

High cardinality labels may be monitored at /api/v1/status/tsdb page. See https://prometheus.io/docs/prometheus/latest/querying/api/#tsdb-stats for details. Note that this page became available starting from Prometheus v2.14 . Take a look also at https://github.com/open-fresh/bomb-squad project, w

Re: [prometheus-users] jmx_exporter's MBean-level fetching well-intentioned but preventing required telemetry [feature request discussion]

On Friday, 5 June 2020 22:31:13 UTC+12, Brian Brazil wrote: > > On Fri, 5 Jun 2020 at 11:13, Cameron Kerr > wrote: > >> Hi all, I've spent a few days coming up to speed on understanding >> jmx_exporter, and I think I'm in a pretty good place now, understanding how >> MBeans, JMX, and RMI work.

Re: [prometheus-users] Prometheus dashboard loads however in target section promtheus isn't UP.

Thanks a lot !! I completely forgot that the system was rebooted yesterday and I lost track of firewall. After disabling firewall and flushing iptables it works now. On Friday, June 5, 2020 at 4:00:26 PM UTC+5:30, Christian Hoffmann wrote: > > Hi, > > On 6/5/20 11:45 AM, Isabel Noronha wrote: >

Re: [prometheus-users] jmx_exporter's MBean-level fetching well-intentioned but preventing required telemetry [feature request discussion]

On Fri, 5 Jun 2020 at 11:13, Cameron Kerr wrote: > Hi all, I've spent a few days coming up to speed on understanding > jmx_exporter, and I think I'm in a pretty good place now, understanding how > MBeans, JMX, and RMI work. > > I've so far deployed jmx_exporter in two ways: > > * as a Java Agent

Re: [prometheus-users] Prometheus dashboard loads however in target section promtheus isn't UP.

Hi, On 6/5/20 11:45 AM, Isabel Noronha wrote: > I have been working on prometheus for the past 3 months. > This is the first time where I have come across such a problem where > prometheus is running inside a container. > However, when I check the dashboard it says"context deadline > exceeded".It

Re: [prometheus-users] Scrape more than one value at once, per second?

If you implement the API yourself it would be better to instrument that; that way you have real tests based on how clients are actually using the API and not synthetic tests based on someones belief in how the API will be used. Or if you're logging this activity (eg. to Elasticsearch or some ot

[prometheus-users] jmx_exporter's MBean-level fetching well-intentioned but preventing required telemetry [feature request discussion]

Hi all, I've spent a few days coming up to speed on understanding jmx_exporter, and I think I'm in a pretty good place now, understanding how MBeans, JMX, and RMI work. I've so far deployed jmx_exporter in two ways: * as a Java Agent on Tomcat 6 (on RHEL6) for a small application that includes

[prometheus-users] Prometheus dashboard loads however in target section promtheus isn't UP.

Hey, I have been working on prometheus for the past 3 months. This is the first time where I have come across such a problem where prometheus is running inside a container. However, when I check the dashboard it says"context deadline exceeded".It is not even able to discover node_exporter. Can

[prometheus-users] Timestamp of last change of metric

Hi fellows, I want to get the timestamp or the time passed since a certain metric has been a certain value, like 1 in this case: I have an enum value (can make it a Gauge too, if needed) and want to see, when it has been at a certain state the last time, hence getting the timestamp of it. The

Re: [prometheus-users] Is it safe to grab metrics via JMX in production?

On Fri, 5 Jun 2020 at 01:19, Cameron Kerr wrote: > > > On Friday, 5 June 2020 05:20:55 UTC+12, Harald Koch wrote: >> >> On Thu, Jun 4, 2020, at 08:10, Cameron Kerr wrote: >> >> You should be sure to use the whitelist and blacklist features; browse >> the available mbean objects using jconsole or

Re: [prometheus-users] Alertmanager: how to split alerts in email notifications

Thank you so much Anya for help. Thanks, Sunil Sagar On Fri, Jun 5, 2020 at 4:25 PM anya...@gmail.com wrote: > Hi, Sunil! > Sure! Catch: > global: > resolve_timeout: 5m > > # The smarthost and SMTP sender used for mail notifications. > smtp_smarthost: 'smtphost:25' > smtp_from: 'z...@zzz

Re: [prometheus-users] Alertmanager: how to split alerts in email notifications

Hi, Sunil! Sure! Catch: global: resolve_timeout: 5m # The smarthost and SMTP sender used for mail notifications. smtp_smarthost: 'smtphost:25' smtp_from: 'z...@zzz.com' smtp_auth_username: 'z...@zzz.com' smtp_auth_password: '' smtp_require_tls: false # The directory from which n

Re: [prometheus-users] How to generate a GUID or current time for my annotations in Prometheus alerting rule templates

Hi Julius, Let me explain my use case. For notices that alert-manger sent by receivers, I'm going to write a webhook to store them and show them on a website. Let's say notice-A got sent by a mail-receiver and a wechat-receiver to a user. What I wanna do is to enable the user to be able to cli

Re: [prometheus-users] Alertmanager: how to split alerts in email notifications

Hi Anya, Can you share the snippet, I am also facing the same issue . Thanks On Fri, Jun 5, 2020 at 4:02 PM anya...@gmail.com wrote: > Thank you a lot, Christian! > It's a kind of magic, your ['...']. I tried to comment it or stay empty [] > and it doesn't work. But ['...'] works perfectly! > T

Re: [prometheus-users] Alertmanager: how to split alerts in email notifications

Thank you a lot, Christian! It's a kind of magic, your ['...']. I tried to comment it or stay empty [] and it doesn't work. But ['...'] works perfectly! Thanks all of you, dear colleagues! пятница, 5 июня 2020 г. в 10:51:51 UTC+3, Christian Hoffmann: > Hi, > > On 6/5/20 9:23 AM, anya...@gmail.c

Re: [prometheus-users] Alertmanager: how to split alerts in email notifications

Thank you, Stuart! I tried to change 'group_by' and comment it, but it doesn't work for me: all alerts are gathering in one email. I need to separate it, but can't. пятница, 5 июня 2020 г. в 10:42:59 UTC+3, Stuart Clark: > On 05/06/2020 08:23, anya...@gmail.com wrote: > > Hi, Christian! > Than

Re: [prometheus-users] How to generate a GUID or current time for my annotations in Prometheus alerting rule templates

Hi, On 6/5/20 9:48 AM, zichen chuh wrote: > I still wanna generate a GUID. After going through these documents, > couldn't find any. > Wonder u have a clue. Haven't tried it, but you could check if you can access the alert's fingerprint and if this is what you want. This seems to be possible in

Re: [prometheus-users] Alertmanager: how to split alerts in email notifications

Hi, On 6/5/20 9:23 AM, anya...@gmail.com wrote: > Hi, Christian! > Thank you for your answer, but could you a little bit explain your > point? I don't understand it. I used 'group_by' in my config. If I > understood you correctly, you offered to comment the 'group_by' block, > don't you? You set

Re: [prometheus-users] How to generate a GUID or current time for my annotations in Prometheus alerting rule templates

hi Christian, I still wanna generate a GUID. After going through these documents, couldn't find any. Wonder u have a clue. 在 2020年5月21日星期四 UTC+8上午1:48:52，Christian Hoffmann写道： > > Hi, > > On 5/20/20 12:26 PM, zichen chuh wrote: > > {{ with query "time()" }}{{ . | first | value }}{{ end}} > >

Re: [prometheus-users] Alertmanager: how to split alerts in email notifications

On 05/06/2020 08:23, anya...@gmail.com wrote: Hi, Christian! Thank you for your answer, but could you a little bit explain your point? I don't understand it. I used 'group_by' in my config. If I understood you correctly, you offered to comment the 'group_by' block, don't you? The suggestion

Re: [prometheus-users] Alertmanager: how to split alerts in email notifications

Hi, Christian! Thank you for your answer, but could you a little bit explain your point? I don't understand it. I used 'group_by' in my config. If I understood you correctly, you offered to comment the 'group_by' block, don't you? пятница, 5 июня 2020 г. в 09:25:39 UTC+3, Christian Hoffmann: >