Re: [prometheus-users] Memory usage grows from 4GB to eventually 10GB memory usage

2020-12-18 Thread Julien Pivotto
On 18 Dec 16:40, Brett Larson wrote: > Here is a link to the "snapshot" of the dashboard. > https://snapshot.raintank.io/dashboard/snapshot/crxdjU7fhzAhl0x0KWiH1ZHGZXKyhqmF Thanks, however this does not seem to show any memory issue. There might be an issue with your configuration, where you co

Re: [prometheus-users] Memory usage grows from 4GB to eventually 10GB memory usage

2020-12-18 Thread Brett Larson
Here is a link to the "snapshot" of the dashboard. https://snapshot.raintank.io/dashboard/snapshot/crxdjU7fhzAhl0x0KWiH1ZHGZXKyhqmF On Friday, December 18, 2020 at 6:09:13 PM UTC-6 Julien Pivotto wrote: > On 18 Dec 16:03, Brett Larson wrote: > > Hello, > > I am using a Prometheus 2.21 server and

Re: [prometheus-users] Memory usage grows from 4GB to eventually 10GB memory usage

2020-12-18 Thread Julien Pivotto
On 18 Dec 16:03, Brett Larson wrote: > Hello, > I am using a Prometheus 2.21 server and I am seeing that the memory is > growing at an untenable rate. I will start the pod at 4GB and eventually it > will move to 10GB after a few days and go into a crashloop back off state. > > The pod is configu

[prometheus-users] Memory usage grows from 4GB to eventually 10GB memory usage

2020-12-18 Thread Brett Larson
Hello, I am using a Prometheus 2.21 server and I am seeing that the memory is growing at an untenable rate. I will start the pod at 4GB and eventually it will move to 10GB after a few days and go into a crashloop back off state. The pod is configured to only keep around 8 hours of data, and this

[prometheus-users] Re: --storage.tsdb.retention.time prometheus syntax

2020-12-18 Thread Brett Larson
>From a comment in the code it looks like.. Units Supported: y, w, d, h, m, s, ms. Here is a link https://github.com/prometheus/prometheus/blob/4eca4dffb89d9da927f1e7884efe85cbd6dc30f3/cmd/prometheus/main.go#L207 On Friday, December 11, 2020 at 9:26:06 AM UTC-6 rodolp...@gmail.com wrote: > > Hi

Re: [prometheus-users] Best practice for Prometheus + GCP Cloud Run

2020-12-18 Thread Rohit Ramkumar
I always knew Stackdriver existed but didn't think to incorporate it here. Thanks! This is helpful. On Fri, Dec 18, 2020 at 11:28 AM Stuart Clark wrote: > On 18/12/2020 14:50, Rohit Ramkumar wrote: > > Hi, > > > > I'm running a service in Cloud Run (https://cloud.google.com/run) and > > wonder

[prometheus-users] Request for edit

2020-12-18 Thread Richard Hartmann
Dear all, let's try something new. We have two largely unwritten design documents and would love to get your input. Global suggestions & comments are enabled, you can simply add what you want. Prometheus Agent: https://docs.google.com/document/d/1wWrTMWn8UgxgSeao8n1TABPm6pnKIGJmEDdbbWV7lRQ Repr

Re: [prometheus-users] Best practice for Prometheus + GCP Cloud Run

2020-12-18 Thread Stuart Clark
On 18/12/2020 14:50, Rohit Ramkumar wrote: Hi, I'm running a service in Cloud Run (https://cloud.google.com/run) and wondering what the best practice is here for setting up Prometheus. Specifically, I'm wondering how to handle the case when there are multiple container instances running behin

Re: [prometheus-users] Preserve labels during Aggregation

2020-12-18 Thread Raghu Udiyar
The use case is to alert on the disk usage deviation from the mean, within a cluster of hosts. The alert should indicate which host deviates from the median disk usage. So this could be written as: metric format: `disk_usage_pct{cluster=foo, instance=foo01}` `max(disk_usage_pct) by (cluster)

[prometheus-users] Best practice for Prometheus + GCP Cloud Run

2020-12-18 Thread Rohit Ramkumar
Hi, I'm running a service in Cloud Run (https://cloud.google.com/run) and wondering what the best practice is here for setting up Prometheus. Specifically, I'm wondering how to handle the case when there are multiple container instances running behind a single Cloud Run API endpoint. If there

Re: [prometheus-users] non-regex label mapping

2020-12-18 Thread Ben Kochie
Yes, it looks like fcFeModuleName is a TEXTUAL-CONVENTION for the Worldwide Name. Which isn't exactly a friendly name. To be honest, I don't think there's going to be a lot of interest in changing the regexp matching behavior. You could write a proposal to prometheus-develop...@googlegroups.com, b

Re: [prometheus-users] Pushing custom metrics to avoid AKS load balancer

2020-12-18 Thread Ben Kochie
If the metrics are unique to each instance behind the load balancer, the best practice is to have Prometheus scrape the pods directly. On Fri, Dec 18, 2020 at 11:16 AM Marcin Burakiewicz wrote: > Hi. > We have got .NET core application deployed on AKS which exposes an > endpoint with some custom

Re: [prometheus-users] non-regex label mapping

2020-12-18 Thread Michael Thomas
Here's my generator.yml: # QLogic Fiberchannel switches fiberchannel: walk: - fcFxPortID - fcFxPortC3InFrames - fcFxPortC3OutFrames - fcFxPortC3Discards - fcFxPortLinkResetIns - fcFxPortLinkResetOuts lookups: - source_indexes: [fcFeModuleIndex]

Re: [prometheus-users] non-regex label mapping

2020-12-18 Thread Ben Kochie
Are you sure this isn't an issue with OctetString vs DIsplayString? Can you share your generator.yml? On Fri, Dec 18, 2020 at 1:58 PM Michael Thomas wrote: > I need to perform a large number (~120) of metric label remappings in my > prometheus config. Right now I'm doing it with 120 entries tha

[prometheus-users] non-regex label mapping

2020-12-18 Thread Michael Thomas
I need to perform a large number (~120) of metric label remappings in my prometheus config. Right now I'm doing it with 120 entries that look like these: - source_labels: [fcFeModuleName,fcFxPortIndex] target_label: portName regex: 0x10C0DD18FE13;1 replacement:

[prometheus-users] Run jmx_exporter on Windows10

2020-12-18 Thread Manu Chadha
Hi I am following this tutorial to run jmx_exporter` with cassandra https://www.robustperception.io/monitoring-cassandra-with-prometheus I have found that instead of changing JVM_OPTS in cassandra-env.sh file, I should change bin/cassandra.bat to make javaagent work (I tried adding JVM_OPTS in

Re: [prometheus-users] Configure blackbox as daemon set to check multiple external services avaliability

2020-12-18 Thread Skaven
Thanks, Brian. пятница, 18 декабря 2020 г. в 13:54:06 UTC+3, Brian Brazil: > On Fri, 18 Dec 2020 at 10:50, Skaven wrote: > >> Hello, folks. >> >> The story. >> In my org we have several k8s clusters and quite unreliable security >> department, who has control over firewall and have a habbit of

Re: [prometheus-users] Configure blackbox as daemon set to check multiple external services avaliability

2020-12-18 Thread Brian Brazil
On Fri, 18 Dec 2020 at 10:50, Skaven wrote: > Hello, folks. > > The story. > In my org we have several k8s clusters and quite unreliable security > department, who has control over firewall and have a habbit of corrupting > the rules on said firewall. > The confusion is immense. The issue is that

Re: [prometheus-users] Scraping Kubernetes metrics from standalone Prometheus Instance

2020-12-18 Thread Sakthi Raam
Thanks Stuart, Will check it out and get back to you Regards Sakthi On Fri, 18 Dec 2020 at 4:20 PM, Stuart Clark wrote: > On 18/12/2020 06:03, Sakthi Raam wrote: > > Hi All, > > > > We have restrictions to deploy Prometheus inside Kubernetes > > environment due to shared infrastructure but we s

[prometheus-users] Configure blackbox as daemon set to check multiple external services avaliability

2020-12-18 Thread Skaven
Hello, folks. The story. In my org we have several k8s clusters and quite unreliable security department, who has control over firewall and have a habbit of corrupting the rules on said firewall. The confusion is immense. The issue is that at any point in time *one or several nodes can lose ac

Re: [prometheus-users] How to design a well target labels scheme?

2020-12-18 Thread Ben Kochie
Locality information is good to have, but you don't want to go too wild with it. For example, if you have a large network, where services are running in multiple networks/regions/zones/etc, you will likely have or want multiple Prometheus servers. For example, we have Prometheus servers in each c

Re: [prometheus-users] How to design a well target labels scheme?

2020-12-18 Thread Allenzh li
Understaood. Thank you very much. 在2020年12月18日星期五 UTC+8 下午4:16:44 写道: > On 18/12/2020 02:54, Allenzh li wrote: > > Hi > > > > I use prometheus to monitor service. > > > > At first, it was simple. I use four labels(team, product, service, > > instance) to identify a service, the metric like > >

[prometheus-users] Pushing custom metrics to avoid AKS load balancer

2020-12-18 Thread Marcin Burakiewicz
Hi. We have got .NET core application deployed on AKS which exposes an endpoint with some custom metrics in Prometheus format. This application is deployed on two hosts which are behind a load balancer. The problem with load balancer is that whenever Prometheus pulls the metrics from the endpoin

Re: [prometheus-users] Scraping Kubernetes metrics from standalone Prometheus Instance

2020-12-18 Thread Stuart Clark
On 18/12/2020 06:03, Sakthi Raam wrote: Hi All, We have restrictions to deploy Prometheus inside Kubernetes environment due to shared infrastructure but we still need to scrape all the node/services/endpoints/pod level metrics using service discovery approach. When I checked almost all the po

Re: [prometheus-users] How to design a well target labels scheme?

2020-12-18 Thread Stuart Clark
On 18/12/2020 02:54, Allenzh li wrote: Hi I use prometheus to monitor service. At first, it was simple. I use four labels(team, product, service, instance) to identify a service, the metric like ``` metric_name{team="t1", product="p1", instance="i1", service="s1", other business label} ```

[prometheus-users] Java library for average time metrics for POJO methods

2020-12-18 Thread Manish G
Hi All, I have a POJO class, and I want to generate average time metrics for it's methods. Is there any library available for this which enables me to just put some annotation on my methods and average time metrics for that method gets created, which can then be pushed to prometheus. I basically