Re: [prometheus-users] Uptime when no up metric is available

2021-01-15 Thread Tom Gregory
Thanks Stuart. I switched to relabel_config and now all good. On Thu, 14 Jan 2021, 21:05 Stuart Clark, wrote: > On 13/01/2021 20:31, Tom Gregory wrote: > > Hi Stuart. The metrics are coming from different scrapes. Each service is > a different endpoint. > > I'm doing the relabelling using *metri

[prometheus-users] Re: Prometheus relabeling (drop action)

2021-01-15 Thread Sam Aram
solve the problem with: - source_labels: [__name__, l2] regex: 'm2;str' action: drop On Friday, January 15, 2021 at 8:06:58 AM UTC-8 Sam Aram wrote: > I use a prometheus federation to gather data from several prometheus > server, I have some metrics like m1, m2 and m3, and ti

[prometheus-users] A query on rate

2021-01-15 Thread Manish G
Hi All, rate function handles restarts if metrics is of counter type, as documentation tells. But how is it of help in something like kubernetes environment. For eg, we have application running on several pods and metrics scraped by prometheus has pod id as one of the label. So if a pod goes away

Re: [prometheus-users] Pushgateway - any way to save/restore metrics

2021-01-15 Thread Peter Toft
The default is 5 minutes which is super for me. But I do not see anything happening On Friday, January 15, 2021 at 3:23:17 PM UTC+1 Julien Pivotto wrote: > I think that this is flushed on disk at regular interval: > --persistence.interval > > On 15 Jan 06:08, Peter Toft wrote: > > Thanx for th

Re: [prometheus-users] Alertmanager yml - invalid leading UTF-8 octet

2021-01-15 Thread Mohan
Resolved by removing bullet • in the code and launched the container. It works now. On Wed, 13 Jan 2021, 10:13 pm Mohan, wrote: > I m trying in centos8.2 system is it something wrong in my yml file? I > can't find the cause. So any help will be appreciated. > > On Mon, 11 Jan 2021, 4:52 pm Mohan

[prometheus-users] Prometheus relabeling (drop action)

2021-01-15 Thread Sam Aram
I use a prometheus federation to gather data from several prometheus server, I have some metrics like m1, m2 and m3, and time series on each of them have labels like l1,l2,l3, how can I drop all time series of m2 with l2="str" ? I now how to drop unnecessary metrics, or unnecessary time series

[prometheus-users] Enriching Alert Metadata - Labels vs Annotations

2021-01-15 Thread Brian Gottfried
We're forwarding our alerts out to an external system that requires some additional metadata and I'm trying to understand the effect of providing this metadata as annotations vs labels on each alert. Context: We're utilizing both custom alerts and the default alerts that come with Openshift. Th

Re: [prometheus-users] Pushgateway - any way to save/restore metrics

2021-01-15 Thread Julien Pivotto
I think that this is flushed on disk at regular interval: --persistence.interval On 15 Jan 06:08, Peter Toft wrote: > Thanx for the comment Julien. > I try exactly this with > pushgateway: > container_name: pushgateway > image: prom/pushgateway > ports: > - 9091:9091 > volu

Re: [prometheus-users] Does Prometheus optimise storage for non-changing metrics?

2021-01-15 Thread Ben Kochie
Yes, Prometheus uses "Double Delta" and other compression techniques to reduce these to a few bits per sample. There's a good video about this: https://youtu.be/b_pEevMAC3I On Fri, Jan 15, 2021 at 3:03 PM John Dexter wrote: > I have a counter metric with a label which has quite a few possible >

Re: [prometheus-users] Pushgateway - any way to save/restore metrics

2021-01-15 Thread Peter Toft
Thanx for the comment Julien. I try exactly this with pushgateway: container_name: pushgateway image: prom/pushgateway ports: - 9091:9091 volumes: - ./pushgateway.txt:/var/log/pushgateway.txt command: - '--web.enable-admin-api' - '--persistence.file=/va

[prometheus-users] Does Prometheus optimise storage for non-changing metrics?

2021-01-15 Thread John Dexter
I have a counter metric with a label which has quite a few possible values. However in reality many of these labeled metrics change very rarely and some of them might never change depending on the system. Does Prometheus intelligently handle storage in such cases? i.e. only record changes to values

[prometheus-users] Reloading DNS IPs in prometheus

2021-01-15 Thread eswar yaga
Hi, My configuration uses a DNS like this *- job_name: 'app-dev'* *metrics_path: '/federate'* *params:* *'match[]':* * - "{__name__ =~ '^.+'}"* *static_configs:* * - targets: ['app-dev-prom.localdns.myaddress.com']* But When the DNS points to a new IP of APP server, DNS looks sticky and still

[prometheus-users] PromQL step parameter

2021-01-15 Thread kiran
Hello all 1. In PromQL range query what step value is used if I don’t give step parameter? 2. Is there a way to determine for the given start and end dates, what is the step value I can give so it’s with the limit where query can handle? E.g I used a query for last 30 days and I gave 1 min as step

Re: [prometheus-users] Pushgateway - any way to save/restore metrics

2021-01-15 Thread Julien Pivotto
You will need to use a docker volume to save the file across restarts. On 15 Jan 03:36, Peter Toft wrote: > Thanx Stuart > I use docker where I just now added - > '--persistence.file=/var/log/pushgateway.txt' > It does not seem to give persistence. > > version: '3.3' > services: > pushga

Re: [prometheus-users] Pushgateway - any way to save/restore metrics

2021-01-15 Thread Peter Toft
Thanx Stuart I use docker where I just now added - '--persistence.file=/var/log/pushgateway.txt' It does not seem to give persistence. version: '3.3' services: pushgateway: container_name: pushgateway image: prom/pushgateway ports: - 9091:9091 volumes: - ./push

Re: [prometheus-users] Prometheus with remote_write VM dies with OOM: remote_write settings problem?

2021-01-15 Thread Aliaksandr Valialkin
Try increasing `capacity` to 3x max_samples_per_send, i.e. to 2 for your case according to https://prometheus.io/docs/practices/remote_write/ . Prometheus may require up to 30% more memory after enabling remote_write according to production measurements. Make sure that your Prometheus instance

Re: [prometheus-users] Pushgateway - any way to save/restore metrics

2021-01-15 Thread Stuart Clark
The push gateway supports a persistence file via the --persistence.file parameter to allow cached metrics to survive restarts. On 15 January 2021 09:46:18 GMT, Peter Toft wrote: >Hi guys > >I use a pushgateway -> prometheus -> grafana system to display >simulation >metrics, and when I reboot t

[prometheus-users] Pushgateway - any way to save/restore metrics

2021-01-15 Thread Peter Toft
Hi guys I use a pushgateway -> prometheus -> grafana system to display simulation metrics, and when I reboot the server I loose all the metrics I have in the pushgateway. They appear when I have rerun my external test server pushing metrics. If I know that there is a reboot of the pushgateway

[prometheus-users] Prometheus with remote_write VM dies with OOM: remote_write settings problem?

2021-01-15 Thread Olga Chukanova
Hello! I have prometheus like monitoring system in kubernetes, and I trying to set up remote_write to victoria metrics. But I have one tragic problem - my prometheus dies by OOM. I’ve tested two versions of Prometheus (v2.11.0 and v.2.23.0) and had same problem on both. My average value of rat

Re: [prometheus-users] Password Protection for my Prometheus Exporter

2021-01-15 Thread Dhanush Eswar
Yes, I am using prometheus_client.I wish to add the basic authentication to my endpoint. Best, Dhanush On Wednesday, January 13, 2021 at 2:38:09 PM UTC+5:30 sayf.eddi...@gmail.com wrote: > Hello, You are using the prometheus python client? because the feature is > supported out of the box >

[prometheus-users] Alerting in Prometheus federated environment.

2021-01-15 Thread 'Dennis Murtic' via Prometheus Users
Hi all, Currently we have different environments where Prometheus and Alertmanager are used. These environments are also protected by certain network settings, so we need to use PushProx. To collect data from all environments, we use a "global" Prometheus server that federates with the environment