Re: [prometheus-users] Prometheus reports Out of memory while loading segment

2021-06-01 Thread Stuart Clark
On 02/06/2021 00:01, Naveen Badam wrote: Thanks Stuart. That would be the obvious thing to do. Is there any way we can manage the memory segment size limits? I believe below err represents the issue loading specific segment or it exhaust after loading x number of segment since it exceeded the

Re: [prometheus-users] Re: level=error ts=2021-05-28T09:23:52.326Z caller=main.go:894 err="opening storage failed: lock DB directory: resource temporarily unavailable"

2021-06-01 Thread nina guo
Thank you very much Stuart. On Tuesday, June 1, 2021 at 6:14:45 PM UTC+8 Stuart Clark wrote: > On 31/05/2021 04:31, nina guo wrote: > > May I ask why multiple Prometheus cannot write to the same directory? > > Different instances of Prometheus are not aware of each other and so > would overwrite

[prometheus-users] Blackbox prober - count bad events for latency SLI

2021-06-01 Thread George Brighton
Hi folks, I've written a blackbox_exporter-like process which exposes a probe_duration_seconds gauge. It is scraped every 15s. Given a latency threshold, I'd like to create recording rules to count the number of bad events and number of total events over the last 1m to derive a latency SLI. Ob

Re: [prometheus-users] Prometheus reports Out of memory while loading segment

2021-06-01 Thread Naveen Badam
Thanks Stuart. That would be the obvious thing to do. Is there any way we can manage the memory segment size limits? I believe below err represents the issue loading specific segment or it exhaust after loading x number of segment since it exceeded the memory limit? On Wed, 2 Jun 2021, 00:15 S

Re: [prometheus-users] Support for HTTPS in JMX exporter

2021-06-01 Thread Daniel Lacher
Apparently, you can't do that without a reverse proxy in front of your java node: https://github.com/prometheus/jmx_exporter/issues/442 Connecting with SSL doesn't mean protecting the /metrics endpoint in this case: https://groups.google.com/g/prometheus-developers/c/ltI-gB0seVs On Tuesday, Febr

Re: [prometheus-users] Prometheus reports Out of memory while loading segment

2021-06-01 Thread Stuart Clark
On 2021-06-01 17:09, Naveen Badam wrote: Hi Team, I found prometheus started reporting "fatal error: runtime: out of memory" recently. the only change i did was, we have added few bunch of new scrappers to prometheus, probably that is causing issues loading the segment data? below is the error

[prometheus-users] Prometheus reports Out of memory while loading segment

2021-06-01 Thread Naveen Badam
Hi Team, I found prometheus started reporting "fatal error: runtime: out of memory" recently. the only change i did was, we have added few bunch of new scrappers to prometheus, probably that is causing issues loading the segment data? below is the error am getting while prometheus startup: Appr

[prometheus-users] Getting data out from Prometheus (2021)

2021-06-01 Thread 'Evelyn Pereira Souza' via Prometheus Users
Hi I found groups.google.com/g/prometheus-developers/c/vAiOnEmbTlg and wonder if there is an update for 2021. My use case is that other department wish the values of queries in an Excel (for visualisation). query: quantile_over_time(0.5, probe_duration_seconds{team="zzz",env="yyy",job="xx

Re: [prometheus-users] Re: level=error ts=2021-05-28T09:23:52.326Z caller=main.go:894 err="opening storage failed: lock DB directory: resource temporarily unavailable"

2021-06-01 Thread Stuart Clark
On 31/05/2021 04:31, nina guo wrote: May I ask why multiple Prometheus cannot write to the same directory? Different instances of Prometheus are not aware of each other and so would overwrite & corrupt the files. Prometheus is also not supported with shared file systems such as NFS (due to