[prometheus-users] Re: Cannot see metrics values after loading through create-blocks-from

2022-02-27 Thread Laurent Dumont
Digging a bit more and this seems to be a local issue on my end. Spun a grafana + prometheus on my laptop, used the same data and I see it in Prometheus. On Sun, Feb 27, 2022 at 12:43 PM Laurent Dumont wrote: > Forgot to add the runtime info! > > Runtime Information > Start time

[prometheus-users] Re: Cannot see metrics values after loading through create-blocks-from

2022-02-27 Thread Laurent Dumont
Build Information Version 2.33.4 Revision 83032011a5d3e6102624fe58241a374a7201fee8 Branch HEAD BuildUser root@d13bf69e7be8 BuildDate 20220222-16:51:28 GoVersion go1.17.7 On Sun, Feb 27, 2022 at 12:40 PM Laurent Dumont wrote: > Hey folks, > > Trying to load some past data into Prometh

[prometheus-users] Cannot see metrics values after loading through create-blocks-from

2022-02-27 Thread Laurent Dumont
Hey folks, Trying to load some past data into Prometheus using create-blocks-from. - The data appears to be proper Openmetrics formatted. - No errors from Prometheus - I do see the metric in Prometheus, but there doesnt appear to be any data points. - Tried restarting the server to

[prometheus-users] Filter query based on result from first query.

2021-08-06 Thread Laurent Dumont
Hi! Trying to wrap my brain around the group_left / group_right in PromQL. I have the following metrics (removed some labels) potato{short_hostname="qasite1-compute001",site="qasite1",hardware="hardwaretype1"} 1 potato{short_hostname="qasite1-compute001",site="qasite1",hardware="hardwaretype2"}

Re: [prometheus-users] False Positive Alerts for node CPU Usage for one node

2021-07-09 Thread Laurent Dumont
I don't know how GCP calculates their CPU metrics, but node_cpu_seconds_total looks to contain statistics for user/kernel/interrupt etc spaces. Maybe you can make a separate graph based on each of those and see if one is much higher ( https://www.robustperception.io/understanding-machine-cpu-usage)

Re: [prometheus-users] Read collected data in Elasticsearch

2021-06-26 Thread Laurent Dumont
I dont think that will work out of the box. Prometheus doesn't understand Elasticsearch by itself. You could use this https://github.com/braedon/prometheus-es-exporter to export the data inside ES to a Prometheus standard format. You can then scrape that target from your Prometheus server and aler

Re: [prometheus-users] Prometheus.service not starting [HELP]

2021-05-04 Thread Laurent Dumont
Salut! First thing would be to look at the actual service logs of the Prometheus service. Something like : journalctl -u prometheus.service That should tell you what is failing to start and forcing systemd to kill the service. On Tue, May 4, 2021 at 10:38 AM Support IT Adept wrote: > Hi, >

Re: [prometheus-users] Re: Prometheus

2021-03-23 Thread Laurent Dumont
This is a general help ML with no expectations of support. You will need to provide more details regarding how you are getting metrics for CPU usage and if it's available on a per-core basis. On Tue, Mar 23, 2021 at 4:16 PM Vamshi Bhargav wrote: > Any update on this. > > On Tuesday, March 23, 2

Re: [prometheus-users] Prometheus container does not run

2021-03-23 Thread Laurent Dumont
What docker-compose command are you using? You need to use "docker-compose up" to actually build + start the container. On Tue, Mar 23, 2021 at 8:30 PM Anauê Curi wrote: > Hi everyone, > > I'm trying to run prometheus via a container in docker, but after run it > the container status moves to cr

Re: [prometheus-users] Typical usecase for using the "amtool alert add" command

2020-12-16 Thread Laurent Dumont
I don't think your use case is something that AM or Prometheus is looking to solve. The way I see it : - Prometheus has metrics and alarm patterns. - It triggers an alarm and sends it to AM. - AM receives the alarm and does some basic routing based on labels. - Once the Prometheus pat

Re: [prometheus-users] Re: Allow local time in Prometheus log timestamps using TZ env

2020-12-16 Thread Laurent Dumont
This is usually something solved outside of Prometheus. Prometheus container logs --> Fluentd --> ElasticSearch --> Kibana display You can then use Kibana to change the display of the time to your local timezone. A pretty common design decision is to log everything in UTC and let other systems pa

Re: [prometheus-users] How do I set the maintenance cycle

2020-12-15 Thread Laurent Dumont
If you are coming from a Zabbix world, there is no equivalent maintenance mechanic in Prometheus. Most of the service assurance/service mapping is not part of the Prometheus stack. Silences are only active between X and Y and will expire after Y. They cannot repeat so you'll need some kind of exte

Re: [prometheus-users] pressure testing tools

2020-12-14 Thread Laurent Dumont
It depends on what you want to test Actual Prometheus performance - https://blog.freshtracks.io/load-testing-prometheus-metric-ingestion-5b878711711c - https://github.com/thanos-io/thanosbench I'm not sure just testing a GET on prom-ip:9090/ is going to tell you much. Maybe a bunch of

Re: [prometheus-users] Correctly using metric_relabel_configs

2020-11-19 Thread Laurent Dumont
Thanks! I missed that part. I'll tinker around with it. On Thu, Nov 19, 2020 at 5:55 PM Christian Hoffmann < m...@hoffmann-christian.info> wrote: > Hi, > > On 11/19/20 11:32 PM, Laurent Dumont wrote: > > collectd_openstack_nova_gauge{exported_instance=" > s

[prometheus-users] Correctly using metric_relabel_configs

2020-11-19 Thread Laurent Dumont
Hi! I was trying to use metric_relabel_configs in order to change the label content (sourcing from another label) of a metric. The original metric looks like this: collectd_openstack_nova_gauge{exported_instance="site1-director.potato.com ",instance="123.123.123.123:9103",job="collectors",openst

Re: [prometheus-users] The possibility of multiple push gateways

2020-11-16 Thread Laurent Dumont
I don't think there would be any issues with multiple jobs with one target OR one job with multiple targets. Is the data being exposed by each PG different? On Sun, Nov 15, 2020 at 9:57 PM Andy Pan wrote: > Is there a right path forward for Prometheus to collect metrics from > multiple push gat

Re: [prometheus-users] Guaranteeing that last metric of a pod is scraped

2020-11-12 Thread Laurent Dumont
You can add some HTTP auth in front of the PG (not directly though PG but with nginx/apache). But yes, as long as the pod needs to access PG directly, I guess it means that the customer running code inside the pod will be able to talk to PG. On Thu, Nov 12, 2020 at 10:19 AM Rafael Paulovic wrote:

Re: [prometheus-users] Guaranteeing that last metric of a pod is scraped

2020-11-12 Thread Laurent Dumont
I'm not sure if I fully understand the flow of metrics, but you can use a PushGateway as a "central" scrape target. In your case, since your application pods seem to be short-lived, a per pod scrape architecture might not be great. You can use the prometheus_client library to push application metri

Re: [prometheus-users] blackbox exporter returns Gateway Time-out when running the URL manually

2020-11-12 Thread Laurent Dumont
If the blackbox exporter itself doesn't seem to respond/load properly, I would troubleshoot that first. You can look at the pod logs. On Thu, Nov 12, 2020 at 2:36 AM nesa...@gmail.com wrote: > I have a blackbox exporter deployed on a K8S cluster, The problem is, some > URLs (which are using the

Re: [prometheus-users] prometheus server in crash loop when volume is full

2020-11-10 Thread Laurent Dumont
ance > > On Mon, Nov 9, 2020 at 10:06 Laurent Dumont > wrote: > >> Your retention is 16 days and you filled the data volume in 13? >> >> I don't Prometheus will alarm for it's data partition by default. >> >> On Mon, Nov 9, 2020, 12:27 PM Laurent De

Re: [prometheus-users] Re: promql.(*Engine).execEvalStmt very high cpu

2020-11-10 Thread Laurent Dumont
How many nodes do you have as targets? Are all of the 2000 rules the same? With a large number of nodes and a rule that is returning a large number of metrics, it might be a lot to process. On Tue, Nov 10, 2020 at 1:18 AM 万锐 wrote: > On Tuesday, November 10, 2020 at 2:16:02 PM UTC+8 万锐 wrote: >

Re: [prometheus-users] prometheus server in crash loop when volume is full

2020-11-09 Thread Laurent Dumont
Your retention is 16 days and you filled the data volume in 13? I don't Prometheus will alarm for it's data partition by default. On Mon, Nov 9, 2020, 12:27 PM Laurent Demailly wrote: > *What did you do?* > Setup using community helm chart (8Gi data volume, 15 days retention) > > *What did you

Re: [prometheus-users] suspending a VM that is running prometheus?

2020-11-08 Thread Laurent Dumont
; Thanks for your reply! > > I agree, but the the state of the art continues to be that this ... > doesn't work very well. Even RedHat's KVM+QEMU documentation recommends > that the guests run NTP. > > I'm still curious what the source of the issue is. Laurent Dumon

Re: [prometheus-users] suspending a VM that is running prometheus?

2020-11-04 Thread Laurent Dumont
When you resume the VM, does NTP resync? I guess that since you just pause the VM, once it unpause, the time will be incorrect and that would explain the out-of-order message. The scaped host will be in the future from the perspective of the Prometheus server. On Wed, Nov 4, 2020 at 8:46 PM Harald

Re: [prometheus-users] aggregating metrics across remote systems with spotty connectivity?

2020-08-22 Thread Laurent Dumont
I wonder if you could use a Push Gateway to which all the spotty hosts would write to. They could write locally the metrics to a file and keep trying to push the metrics once they have the connectivity to the central Prometheus. On Sat, Aug 22, 2020 at 1:43 PM Brian Candler wrote: > Another opti

Re: [prometheus-users] Prometheus generate status metric (Value: 0, 1, 2, 3)

2020-08-14 Thread Laurent Dumont
You could use a templated tile. You have a variable that represents every system monitored by Prometheus and the same tile is replicated for every value of that variable. On Fri, Aug 14, 2020, 6:22 PM Thomas Berger wrote: > For Info: > > I use a Dashboard in Grafana with Tiles. > Each Tile stand

Re: [prometheus-users] Prometheus generate status metric (Value: 0, 1, 2, 3)

2020-08-14 Thread Laurent Dumont
I don't think Prometheus can do much in that case. It's end goal is to expose a metric. With Alert rules, you can tie some level of business logic. What we do is standardize our exporting values, where 1 = OK and 0 = NOT OK for custom data. You can then have alert rules that will alarm based on th

Re: [prometheus-users] Openstack exporter Query

2020-08-14 Thread Laurent Dumont
I think you need to look at the source code for the Openstack exporter :) On Fri, Aug 14, 2020 at 7:49 AM Saurabh Jain wrote: > Hi Everyone ... > > > I need your advice related to openstack_neutron_up metric from openstack > exporter.The value which we are getting for this metric is 0 .What does

Re: [prometheus-users] Silenced alerts and web.hook receiver

2020-08-10 Thread Laurent Dumont
That something that I investigated as well. So far, it seems a silence will mute an alarm at the root level on the AM side. There is no notifications that a silence was created either. In your case, you could have the reverse where your monitoring system could create the silences locally and push

[prometheus-users] [AlertManager] Silence an alert but still send it for a specific route.

2020-07-23 Thread Laurent Dumont
Hi! We have a pretty specific use case where we have three different recipients for Alerts. -Recipient1 -Recipient2 -Recipient3 Right now, if I create a silence, the Alarms matching that silence will be ignored for all three Recipients. I was wondering if there was a way to create a silence b