[prometheus-users] Re: [Alertmanager] Configuration option group_wait does not work

2020-02-20 Thread Андрій К
Hello, seems like the reason is silenced alert with severity: disaster Does muted alerts also count into group_wait? Regards, Andrii четверг, 20 февраля 2020 г., 14:35:59 UTC+2 пользователь Андрій К написал: > > > *Hello,* > > > *What did you do?* > Send test alert from prometheus to

Re: [prometheus-users] Re: Data getting corrpted - missing meta.json

2020-02-20 Thread Guru SD
Thanks for confirming, kind of thought this was the problem. Can you pls help with which file system to use? We are running Ubuntu on Azure VMs. Is it ext4? On Thursday, 20 February 2020 18:36:45 UTC+5:30, Stuart Clark wrote: > > If you have more than one copy of Prometheus running against the

Re: [prometheus-users] Re: How do you deal with rarely updated values

2020-02-20 Thread Brian Candler
I think a gauge is the right thing here: it represents the amount of time taken to rebuild the last configuration. You can just scrape this periodically (say every minute), and easily see if it goes up or down over time. Of course, most of the time you'll be scraping the same value - but

[prometheus-users] Is there a way to hide labels in received notifications?

2020-02-20 Thread Shlomi
Hi, Currently my Prometheus+Alertmanager deployment is notifying me on Alerts via emails. The thing is, within each email i receive a lot of unneeded information, especially under the 'Labels' section. Is there a way to hide those label values which I'm not interested in? Or maybe it is

[prometheus-users] Re: jmx exporter does not allow cassandra to start

2020-02-20 Thread Maxim Efimov
You probably have a broken config.yaml file for the agent. Try to go through these steps https://www.robustperception.io/monitoring-cassandra-with-prometheus with your version of agent and config file. Start cassandra in the foreground. пятница, 29 декабря 2017 г., 9:34:00 UTC+3 пользователь

[prometheus-users] output of promql query as input to other query

2020-02-20 Thread sunil sagar
Hi , With topic name I am able to find the servers associated with that topic . Now I want to query cpu and memory for those server . I have cpu and metrics for those servers . But how to form query . Details : There are 10 servers say server1 .. server10 . Topic is assocated with only

Re: [prometheus-users] Re: Data getting corrpted - missing meta.json

2020-02-20 Thread Stuart Clark
If you have more than one copy of Prometheus running against the same data directory you will cause corruption. They need to be using separate directories and the use of NFS is generally not recommended. On 20 February 2020 12:40:59 GMT, Guru SD wrote: >Yes we have two Prometheus servers

[prometheus-users] Re: Data getting corrpted - missing meta.json

2020-02-20 Thread Guru SD
Yes we have two Prometheus servers accessing the same storage backend. Not sure if it is via NFS. They have configured to use the same storage.tsdb.path on one of the two servers. The servers are Azure VMs running on Ubuntu. Using the latest version of Prometheus. On Wednesday, 19 February

[prometheus-users] [Alertmanager] Configuration option group_wait does not work

2020-02-20 Thread Андрій К
*Hello,* *What did you do?* Send test alert from prometheus to alertmanager *What did you expect to see?* Alertmanager should send alert after 10 minutes after received it https://prometheus.io/docs/alerting/configuration/#route How long to initially wait to send a notification for a group

Re: [prometheus-users] AWS cross region Node Exporter monitoring issue

2020-02-20 Thread Brian Candler
Grafana can query multiple prometheus data sources: https://www.robustperception.io/switching-between-prometheus-servers-in-grafana-using-data-source-variables That requires the user to select the required data source from a drop-down. Alternatively, can also put promxy in front of the

Re: [prometheus-users] AWS cross region Node Exporter monitoring issue

2020-02-20 Thread Stuart Clark
On 2020-02-20 10:48, Ollie Sutton wrote: I use Grafana to visualise the data collected by Prom. If i have three separate instances of Prom, can i link two instances to my one main instance or still have grafana pull from only one Prom data source? Grafana can have multiple Prometheus

Re: [prometheus-users] AWS cross region Node Exporter monitoring issue

2020-02-20 Thread Ollie Sutton
I use Grafana to visualise the data collected by Prom. If i have three separate instances of Prom, can i link two instances to my one main instance or still have grafana pull from only one Prom data source? After posting this topic, i since researched VPC peering which seems to be what i

Re: [prometheus-users] AWS cross region Node Exporter monitoring issue

2020-02-20 Thread Stuart Clark
That sounds like an AWS thing. However it is generally advisable to run Prometheus servers within each failure domain and not scrape things across domains. So for this I'd suggest a Prometheus server in each region. On 20 February 2020 10:18:00 GMT, Ollie Sutton wrote: >I am having this issue

Re: [prometheus-users] Raise an alert before system goes down.

2020-02-20 Thread Stuart Clark
As long as there are some metrics which give a good indication that things are about to break then yes. For example you can alert on 100% disk usage, but you can also alert on disk usage projected to reach 100% within the next hour On 20 February 2020 10:26:05 GMT, Isabel Noronha wrote: >I

[prometheus-users] Raise an alert before system goes down.

2020-02-20 Thread Isabel Noronha
I have configured alerts to raise after the system goes down. However, is there a way where I can get an alert before the instance goes down in Prometheus? -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and

[prometheus-users] AWS cross region Node Exporter monitoring issue

2020-02-20 Thread Ollie Sutton
I am having this issue where my service discovery on Prometheus can see all my EC2 instances from 3 different regions (including the region it is in) but it can't view the metrics. Node exporter is installed on these instances, i can access their public ip from my machine and go to the

Re: [prometheus-users] Prometheus-services

2020-02-20 Thread Stuart Clark
Maybe you could use the blackbox exporter to monitor the service as well as response times? On 20 February 2020 07:34:27 GMT, Srinivasa Reddy B wrote: >Hi am srinivasa am installed prometheus in my EKS cluster using Helm >charts >currently its running fine But my Project manager asking to do