Re: [prometheus-users] Increasing scrape_interval causing missing data point for metrics

Thanks Stuart. I'll need to think about if it's doable for my case to run node_exporter on each ec2 instances. I am in an infra team, doing that will have lots of impact which I need to evaluate. But thanks for your suggestions. One more questions regarding cloudwatch exporter: for my case, ano

Re: [prometheus-users] Increasing scrape_interval causing missing data point for metrics

On 22/03/2021 23:30, chuanjia xing wrote: I have one more question for node_exporter: say if I want to get ec2 instance cpu metrics for _/lots/_ of clusters, do I need to run node_exporter on every node in all clusters? From the doc of node_exporter, it looks like one exporter will only collect

Re: [prometheus-users] Increasing scrape_interval causing missing data point for metrics

I have one more question for node_exporter: say if I want to get ec2 instance cpu metrics for *lots* of clusters, do I need to run node_exporter on every node in all clusters? From the doc of node_exporter, it looks like one exporter will only collect metrics for the node it's running on, which

Re: [prometheus-users] Increasing scrape_interval causing missing data point for metrics

Thanks Stuart. I didn't know node exporter can also collect metrics at instance level. If it can get per instance level cpu metrics and faster than cloudwatch exporter, then that should satisfy my requirements. I'll take a look at node exporter then. On Monday, March 22, 2021 at 4:03:48 PM UTC-

Re: [prometheus-users] Increasing scrape_interval causing missing data point for metrics

On 22/03/2021 22:53, chuanjia xing wrote: Thanks. The reason I am using cloudwatch exporter is because I want to get cpuutilization metrics per cluster / service, not on the node level. I haven't used node_exporter before, not sure if I can get cpuutilization metrics for per cluster / service?

Re: [prometheus-users] Increasing scrape_interval causing missing data point for metrics

On 22/03/2021 22:53, chuanjia xing wrote: Thanks. The reason I am using cloudwatch exporter is because I want to get cpuutilization metrics per cluster / service, not on the node level. I haven't used node_exporter before, not sure if I can get cpuutilization metrics for per cluster / service?

Re: [prometheus-users] Increasing scrape_interval causing missing data point for metrics

Thanks. The reason I am using cloudwatch exporter is because I want to get cpuutilization metrics per cluster / service, not on the node level. I haven't used node_exporter before, not sure if I can get cpuutilization metrics for per cluster / service? On Monday, March 22, 2021 at 3:35:24 PM UT

Re: [prometheus-users] Increasing scrape_interval causing missing data point for metrics

You should gather CPU utilization from the node_exporter, not cloudwatch. This is much more scaleable and won't run into these problems. On Mon, Mar 22, 2021 at 11:22 PM chuanjia xing wrote: > Thanks for your quick response Stuart! > The reason I increase the scrape_interval to be longer than 2

Re: [prometheus-users] Increasing scrape_interval causing missing data point for metrics

Thanks for your quick response Stuart! The reason I increase the scrape_interval to be longer than 2 mins is that I have several regions in aws to query for ec2 cpuutilization metrics, and for the Exporter, some region it took ~3mins to return the cloudwatch matrics. Let's say if it took 3mins,

Re: [prometheus-users] Increasing scrape_interval causing missing data point for metrics

On 22/03/2021 21:48, chuanjia xing wrote: Hi there,       I recently hit an missing data point issue using prometheus. Want to get some help here. Thanks. *Issue:* Increasing scrape_interval in prometheus resulted in missing data points. *My scenario:* I am using prometheus CloudWatch Exp