[prometheus-users] Storing future timestamps in Prometheus

2020-07-13 Thread Hari Krishna Reddy Golamari
HI, My use case is to Pull data (lets say CPU utilisation) from prometheus using httpAPI, Do a 4 hour forecast using Some ML models like deepAR or LSTM, push the forecasted values to prometheus Trigger an alarm when the actual data deviates from forecast BLOCKER : cant able to push more than 2

[prometheus-users] Dont resolve alerts when exporter is down

2020-07-13 Thread Marek Fiala
Hi, I have my own custom exporter with metrics of our network devices. Sometimes this exporter is down for short time and it causes resolve all alerts and then start again. So all sms/email notifications are sent again. How can I solve this issue? I don't want to resolve alerts (and send notif

[prometheus-users] Re: Is Prometheus/Grafana works on Sun-Solaris Operating System too

2020-07-13 Thread Malaka Fernando
Hello Alexander Golikov; Is there any way you could share the Solaris sparc exporters you build. Kind regards Mal On Monday, December 23, 2019 at 5:51:08 PM UTC+8, Alexander Golikov wrote: > > Hello, > I also have to monitor Oracle SPARC Solaris Servers. > Now they are monitored by my own na

[prometheus-users] Re: Multiple blackbox exporters as one job

2020-07-13 Thread Roman
Wow, that's complicated. Looks legit, I'll give it a try, thanks! On Monday, July 13, 2020 at 4:44:14 PM UTC+3, Brian Candler wrote: > > Perhaps I confused you by putting both example.com and example.net, but > it was to show that you that your multiple exporters could poll multiple > hosts. > >

[prometheus-users] [snmp-exporter] empty lookup values 2 electric boogaloo

2020-07-13 Thread Elliott Saille
I have been working on standing up prometheus to monitor our environment and one of the big sticking points has been getting the checkpoint firewalls working correctly. I was reading https://groups.google.com/g/prometheus-users/c/NDyO8qRlXoM?pli=1 and saw the suggested work around as well as t

[prometheus-users] Re: node_ntp_stratum metric is not exposed by node_exporter to verify NTP time synchronization

2020-07-13 Thread Jayaprakash Rangaswamy
Thanks Brain Candler. It works. On Monday, July 13, 2020 at 1:09:11 PM UTC-4, Brian Candler wrote: > > You did notice that these are in the "disabled by default" section of > node_exporter, and you have to enable them with a CLI flag? > (--collector.ntp) > > $ curl -Ss 127.0.0.1:9100/metrics |

[prometheus-users] Aggregations on high cardinality metrics

2020-07-13 Thread Dimitrije M
I have a metric with a large cardinality so when I attempt to do something like this histogram_quantile(0.99, sum(rate(http_request_duration_bucket[15d])) by (le , slo, job, namespace)) I get the error `*query processing would load too many samples into memory in query execution*` What is the

Re: [prometheus-users] Sent resolved or inactive status (alertmanager)

2020-07-13 Thread Brian Candler
It goes under each receiver separately, e.g. email_configs: - to: em...@gmail.com from: em...@gmail.com send_resolved: true -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receivin

[prometheus-users] Re: node_ntp_stratum metric is not exposed by node_exporter to verify NTP time synchronization

2020-07-13 Thread Brian Candler
You did notice that these are in the "disabled by default" section of node_exporter, and you have to enable them with a CLI flag? (--collector.ntp) $ curl -Ss 127.0.0.1:9100/metrics | egrep '^(node_ntp_|# (HELP|TYPE) node_ntp_)' # HELP node_ntp_leap NTPD leap second indicator, 2 bits. # TYPE no

Re: [prometheus-users] Sent resolved or inactive status (alertmanager)

2020-07-13 Thread Dmitry
No, I did not set send_resolved: true. I am trying to add this line to my configuration file, but there is no result. But if I look at the alertmanager status page, I will see the line send_resolved: false in my configuration. global: slack_api_url: 'https://hooks.' route:

[prometheus-users] Re: Prometheus docker container using Terraform

2020-07-13 Thread 'Mr C0ns1st3nt' via Prometheus Users
I wrote an article which may help you and hopefully explains my journey to a working Prometheus: medium.com/@ellishoile/prometheus-with-alert-manager-hosted-in-fargate-1e18e6dbb34f On Tuesday, 15 October 2019 15:55:30 UTC+1, Andrew Hooker wrote: > > I have the standard prometheus container worki

[prometheus-users] JMX Exporter and ContainerID

2020-07-13 Thread 'Mr C0ns1st3nt' via Prometheus Users
Afternoon, Is anyone running JMX exporter inside a container and if so have you managed to grab the ContainerID for use in Prometheus / Grafana? -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiv

[prometheus-users] node_ntp_stratum metric is not exposed by node_exporter to verify NTP time synchronization

2020-07-13 Thread Jayaprakash Rangaswamy
I'm looking for a metric to verify NTP time synchronization with remote NTP servers located within the organization through node-exporter & Prometheus. In the below documentation, I see *node_ntp_stratum* metric is good enough to identify the synchronization against the stratum value. https://g

[prometheus-users] repeat interval on alertmanager

2020-07-13 Thread ebe...@kobo.com
Hi, I have alertmanager configured to send alerts on my test systems to a slack channel. Now test systems are not immediately attended but alertmanager keeps sending alerts to slack after each repeat_interval expires. is there a way to stop these alerts from repeating again. I want them only on

Re: [prometheus-users] Re: Storage Retention 15d for Prometheus

2020-07-13 Thread Christian Hoffmann
On 7/13/20 3:58 PM, Bhupendra kumar wrote: > Yes pls check. The last line looks like the restart was not successful. This might mean that Prometheus is still running with an older cmdline. Can you check the process list? ps aux | grep prometheus or something? I suspect it might still say 15d. If

[prometheus-users] Re: Alert manager GUI does not open using IPV6 address

2020-07-13 Thread Brian Candler
Alternatively, it could be that you haven't told alertmanager the URL to generate links with, using the command line option: --web.external-url=http://[x:x:x::x]:32109 Some of the links it generates may use that URL. You'll be able to tell by looking in your browser console logs, to see if it

[prometheus-users] Re: Alert manager GUI does not open using IPV6 address

2020-07-13 Thread Brian Candler
Then this probably isn't an alertmanager problem, it's a kubernetes problem. Are you running dual-stack Kubernetes? This is an alpha feature (which means it's behind a feature gate): https://kubernetes.io/docs/concepts/services-networking/dual-stack/ I haven't tried it. What I did find in a no

Re: [prometheus-users] Re: Storage Retention 15d for Prometheus

2020-07-13 Thread Julien Pivotto
On 13 Jul 06:34, Bhupendra kumar wrote: > Yes Mmmh can you use --log.level=debug and look for the line: level=debug ts=2020-07-13T13:46:04.886Z caller=main.go:702 msg="TSDB options" MinBlockDuration=2h MaxBlockDuration=36h MaxBytes=0B NoLockfile=false RetentionDuration=15d WALSegmentSize=0B Al

[prometheus-users] Re: Multiple blackbox exporters as one job

2020-07-13 Thread Brian Candler
Perhaps I confused you by putting both example.com and example.net, but it was to show that you that your multiple exporters could poll multiple hosts. There are two jobs: one does replacement: blackbox-eu:9449 and one does replacement: blackbox-us: 9449. If you only have two exporters then th

Re: [prometheus-users] Re: Storage Retention 15d for Prometheus

2020-07-13 Thread Julien Pivotto
Can we get the full log? Thanks On 13 Jul 06:18, Bhupendra kumar wrote: > Yes, I restart the Prometheus service still showing me 15d. Please check > the snapshot. > > On Monday, 13 July 2020 14:31:09 UTC+5:30, Bhupendra kumar wrote: > > > > Hi All, > > > > I want to know how to change Storage

Re: [prometheus-users] Sent resolved or inactive status (alertmanager)

2020-07-13 Thread Christian Hoffmann
Hi, On 7/13/20 1:27 PM, Dmitry wrote: > Hello! > I have standard rule for prometheus alertmanager: >   rules: >   - alert: Instance_down > expr: up == 0 > for: 1m > # Labels - additional labels to be attached to the alert > labels: >   severity: 'critical' > annotations: >

[prometheus-users] Submitting a Lightning Talk video for PromCon Online

2020-07-13 Thread Brian Brazil
Hi, PromCon which starts tomorrow is virtual, so on-the-day lightning talk signup is going to work a bit differently than in previous years. You can submit a pre-recorded lighting talk video of no more than 5 minutes at https://docs.google.com/forms/d/e/1FAIpQLSfmhPBjKWAVkGnLJWv1-YWfPw8k4t_F3zN_I

[prometheus-users] Multi/Remote Cluster Monitoring

2020-07-13 Thread Nithin Thomas
Hi, I would like to know how I can use prometheus to monitor remotely, ie a management Cluster where I have the server and a client cluster where I have the node exporters and kube-state metrics pod. This I hope should be done through service discovery and defining the endpoints. How do I achiev

[prometheus-users] Sent resolved or inactive status (alertmanager)

2020-07-13 Thread Dmitry
Hello! I have standard rule for prometheus alertmanager: rules: - alert: Instance_down expr: up == 0 for: 1m # Labels - additional labels to be attached to the alert labels: severity: 'critical' annotations: title: 'Instance {{ $labels.instance }} down' des

Re: [prometheus-users] Storage Retention 15d for Prometheus

2020-07-13 Thread Julius Volz
Hi, The flag "--storage.tsdb.retention=20d" should definitely work (although "--storage.tsdb.retention.time=20d" is preferred nowadays). If the /status page still says "15d", are you sure you restarted the right Prometheus server? Long startup times come from write-ahead-log (WAL) replay. There h

Re: [prometheus-users] When trying to start node_exporter I get ""TLS is disabled and it cannot be enabled on the fly." http2=false"

2020-07-13 Thread Julien Pivotto
On 13 Jul 03:33, Rod wrote: > Thanks > > I moved it to a RHEL 7 server and it is working fine now. > > I am guessing that there some issues with my RHEL 6 image. I see no issue here. What do you mean? It can not be enabled *on the fly*. It can still be enabled however. > > On Monday, Jul

Re: [prometheus-users] When trying to start node_exporter I get ""TLS is disabled and it cannot be enabled on the fly." http2=false"

2020-07-13 Thread Rod
> > Thanks Julien I think that there an issue with the RHEL6 image... I installed on a RHEL7 image and it is working fine. --Rod > Hello, > > First, it is not required to run the node exporter as root. > > Then, it means that you did not enable TLS, which is the default: > https://gith

Re: [prometheus-users] When trying to start node_exporter I get ""TLS is disabled and it cannot be enabled on the fly." http2=false"

2020-07-13 Thread Rod
Thanks I moved it to a RHEL 7 server and it is working fine now. I am guessing that there some issues with my RHEL 6 image. On Monday, July 13, 2020 at 3:13:06 AM UTC-4, Julien Pivotto wrote: > > On 12 Jul 23:41, Rod wrote: > > ]# ./node_exporter > > level=info ts=2020-07-13T06:20:23.671Z

[prometheus-users] Storage Retention 15d for Prometheus

2020-07-13 Thread Bhupendra kumar
Hi All, I want to know how to change Storage Retention in prometheus, And I am already tried with this command ( prometheus --storage.tsdb.retention=20d ) but still not working. So please give me a solution for this so that prometheus working fine, And also when i need to restart prometheus ser

Re: [prometheus-users] Best way to scrape metrics from a JSON log files

2020-07-13 Thread sayf eddine Hammemi
As said Prometheus is just a fancy wrapper around a TSDB (time-series database). Loki, being developed by Grafana has a sleek integration. You can jump from metric to log file and vice versa inside grafana. Advice you to check it out On Sun, Jul 12, 2020 at 6:39 PM Stuart Clark wrote: > On 12/07

Re: [prometheus-users] When trying to start node_exporter I get ""TLS is disabled and it cannot be enabled on the fly." http2=false"

2020-07-13 Thread Julien Pivotto
On 12 Jul 23:41, Rod wrote: > ]# ./node_exporter > level=info ts=2020-07-13T06:20:23.671Z caller=node_exporter.go:177 > msg="Starting node_exporter" version="(version=1.0.1, branch=HEAD, > revision=3715be6ae899f2a9b9dbfd9c39f3e09a7bd4559f)" > level=info ts=2020-07-13T06:20:23.671Z caller=node_exp