Re: [prometheus-users] Target Server - Which Prometheus Server Is Scraping
Stuart, I am not sure I understand the log files question. I am not aware of any log files related to the scrape itself. We do have log files related to the exporters running on the server but they do not capture the scrapes. I am trying to get details of what is going on on the target server itself, not so concerned about what the Prometheus server has log wise. If you will add some more detail around the question, I will be glad to answer. On Tuesday, February 15, 2022 at 4:41:41 PM UTC-6 kekr...@gmail.com wrote: > I got the Grafana date problem solved and have the scrape time history for > the metric created - the info I needed. > > Thank you Stuart and Brian for the assistance. > > Kevin > > On Tuesday, February 15, 2022 at 4:32:02 PM UTC-6 Stuart Clark wrote: > >> On 15/02/2022 22:29, kekr...@gmail.com wrote: >> > I am not seeing the frequency is more often than I expect. I am being >> > told a log file is being created by the scrapes in a temp directory >> > every minute. I am saying it is not Prometheus. So now i have to >> > prove it is not Prometheus. >> > >> > As an alternate solution, I am trying to use the Prometheus timestamp >> > function on the metric being created by the scrape in Grafana to get >> > the time history of the metric as proof. The thought being the time >> > difference between the metric history is 3 minutes. But I am having >> > trouble getting the value of the timestamp function to act as an epoch >> > date.If I use the value returned in a web epoch translator, it >> > translate to the correct date. If I multiple the value by 1000, as >> > you do every epoch date in Grafana, it actually multiplies the value >> > rather than putting it in human readable date format. >> I'm not clear if you are getting logs from these requests or not? I'd >> expect any request logs to include the path being requested, time & >> source IP. What do you see? >> >> -- >> Stuart Clark >> >> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/4fdc7c08-bbbf-40bc-a26c-c791337155c2n%40googlegroups.com.
Re: [prometheus-users] Target Server - Which Prometheus Server Is Scraping
I got the Grafana date problem solved and have the scrape time history for the metric created - the info I needed. Thank you Stuart and Brian for the assistance. Kevin On Tuesday, February 15, 2022 at 4:32:02 PM UTC-6 Stuart Clark wrote: > On 15/02/2022 22:29, kekr...@gmail.com wrote: > > I am not seeing the frequency is more often than I expect. I am being > > told a log file is being created by the scrapes in a temp directory > > every minute. I am saying it is not Prometheus. So now i have to > > prove it is not Prometheus. > > > > As an alternate solution, I am trying to use the Prometheus timestamp > > function on the metric being created by the scrape in Grafana to get > > the time history of the metric as proof. The thought being the time > > difference between the metric history is 3 minutes. But I am having > > trouble getting the value of the timestamp function to act as an epoch > > date.If I use the value returned in a web epoch translator, it > > translate to the correct date. If I multiple the value by 1000, as > > you do every epoch date in Grafana, it actually multiplies the value > > rather than putting it in human readable date format. > I'm not clear if you are getting logs from these requests or not? I'd > expect any request logs to include the path being requested, time & > source IP. What do you see? > > -- > Stuart Clark > > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/6017e329-45a9-4b0a-bfa5-02de4a74713fn%40googlegroups.com.
Re: [prometheus-users] Target Server - Which Prometheus Server Is Scraping
On 15/02/2022 22:29, kekr...@gmail.com wrote: I am not seeing the frequency is more often than I expect. I am being told a log file is being created by the scrapes in a temp directory every minute. I am saying it is not Prometheus. So now i have to prove it is not Prometheus. As an alternate solution, I am trying to use the Prometheus timestamp function on the metric being created by the scrape in Grafana to get the time history of the metric as proof. The thought being the time difference between the metric history is 3 minutes. But I am having trouble getting the value of the timestamp function to act as an epoch date. If I use the value returned in a web epoch translator, it translate to the correct date. If I multiple the value by 1000, as you do every epoch date in Grafana, it actually multiplies the value rather than putting it in human readable date format. I'm not clear if you are getting logs from these requests or not? I'd expect any request logs to include the path being requested, time & source IP. What do you see? -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/e8c647e2-e295-9c59-01a7-9321b34d9962%40Jahingo.com.
Re: [prometheus-users] Target Server - Which Prometheus Server Is Scraping
I am not seeing the frequency is more often than I expect. I am being told a log file is being created by the scrapes in a temp directory every minute. I am saying it is not Prometheus. So now i have to prove it is not Prometheus. As an alternate solution, I am trying to use the Prometheus timestamp function on the metric being created by the scrape in Grafana to get the time history of the metric as proof. The thought being the time difference between the metric history is 3 minutes. But I am having trouble getting the value of the timestamp function to act as an epoch date.If I use the value returned in a web epoch translator, it translate to the correct date. If I multiple the value by 1000, as you do every epoch date in Grafana, it actually multiplies the value rather than putting it in human readable date format. Kevin On Tuesday, February 15, 2022 at 3:47:10 PM UTC-6 Stuart Clark wrote: > On 15/02/2022 18:41, kekr...@gmail.com wrote: > > My end goal is to prove monitoring is not running every minute on the > > server. My word saying it is not, there's no way, the job is not > > configured to run every minute, is not good enough. > > > > There is a possibility that the two Prometheus servers are scraping at > > the same time but there is no way the scrapes are happening every > > minute. The scrape interval is 3m with a scrape time out of 2m45s. > > So are you seeing more frequent requests than you expect? How are you > telling this? Do you have request logs & what do they say/record? > > -- > Stuart Clark > > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/9188e918-6750-43c3-80f7-7570803cad5fn%40googlegroups.com.
Re: [prometheus-users] Target Server - Which Prometheus Server Is Scraping
On 15/02/2022 18:41, kekr...@gmail.com wrote: My end goal is to prove monitoring is not running every minute on the server. My word saying it is not, there's no way, the job is not configured to run every minute, is not good enough. There is a possibility that the two Prometheus servers are scraping at the same time but there is no way the scrapes are happening every minute. The scrape interval is 3m with a scrape time out of 2m45s. So are you seeing more frequent requests than you expect? How are you telling this? Do you have request logs & what do they say/record? -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/bf39f5ed-4df9-b663-7000-94d1c58cab1d%40Jahingo.com.
Re: [prometheus-users] Target Server - Which Prometheus Server Is Scraping
My end goal is to prove monitoring is not running every minute on the server. My word saying it is not, there's no way, the job is not configured to run every minute, is not good enough. There is a possibility that the two Prometheus servers are scraping at the same time but there is no way the scrapes are happening every minute. The scrape interval is 3m with a scrape time out of 2m45s. On Tuesday, February 15, 2022 at 1:47:37 AM UTC-6 Brian Candler wrote: > You can add whatever query params you like in the scrape job, e.g. you > could add ?from=foo or ?from=bar as part of the URL being scraped. > > - job_name: blah > metrics_path: /metrics > params: > from: [ foo ] > ... > > However, I wonder how you intend to use this information. The data which > is scraped *should* be independent of who is scraping it, and it should be > possible to do additional scrapes without affecting the data (e.g. hitting > the exporter with "curl" to test it shouldn't alter the data for anyone > else). > > Therefore, I wonder if there's a better way to achieve what you're trying > to achieve. For example, if you are keeping a counter of "how many widgets > processed in the last minute", and resetting it to zero on each scrape, > then you should not be doing this; you should be keeping a counter which > just keeps incrementing. It's up to the consumer of the data to look at > data and work out the number of widgets processed per minute, or per hour > or whatever. Having the data in this format is much more useful anyway. > > If you can describe what it is you're doing, and why it matters where the > scrape is coming from, we may be able to give some alternative suggestions. > > On Tuesday, 15 February 2022 at 07:15:11 UTC Stuart Clark wrote: > >> On 15/02/2022 04:32, kekr...@gmail.com wrote: >> > >> > If you have multiple Prometheus servers using an identical target >> > list, is there a way on the target server to tell which Prometheus >> > server is scraping at the time the scrap occurs. >> > >> > For example, Prom_server_a and Prom_server_b scrape >> > target_server_123. On target_server_123 is there something on this >> > server that says Prom_server_a is scraping right now. Prom_server_b is >> > scraping right now? >> > >> The source IP address in any logs? >> >> -- >> Stuart Clark >> >> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/4136dffc-56f4-480c-ab54-0358b86864f2n%40googlegroups.com.
[prometheus-users] Recording Rules are not working
We have configured some recording rules in promethues master server to get different metric usage. Till last week those rules were working but now some of them are not working. If we try to execute its expression we are getting data but if we try to execute with rule name it is showing “Empty Query Result”. If possible, could anyone please help here. *Not Working Rules*: record:Disk_Usage_Node expr:sum by(cluster, kubernetes_io_hostname) (rate(container_fs_usage_bytes{container_name!="POD"}[5m])) / 1.073741824e+09 * 100 labels: Disk_Usage_Node: Disk_Usage_Node OK -2m 18s ago1.821ms record:CPU_Usage_Node expr:sum by(cluster, kubernetes_io_hostname) (rate(container_cpu_usage_seconds_total{container_name!="POD"}[5m])) / sum by(cluster, kubernetes_io_hostname) (machine_cpu_cores) * 100 labels: CPU_Usage_Node: CPU_Usage_Node OK -2m 18s ago2.731ms record:Memory_Usage_Node expr:sum by(cluster, kubernetes_io_hostname) (rate(container_memory_usage_bytes{container_name!="POD"}[5m])) / 1.073741824e+09 * 100 labels: Memory_Usage_Node: Memory_Usage_Node *Working Rules*: record:Current_Cluster_CPU_Usage expr:sum by(cluster, kubernetes_io_hostname, node_role_kubernetes_io_worker) (rate(container_cpu_usage_seconds_total{id="/"}[1h])) labels: Current_Cluster_CPU_Usage: Current_Cluster_CPU_Usage OK -2m 18s ago0.779ms record:Current_Cluster_Memory_Usage expr:sum by(cluster, kubernetes_io_hostname, node_role_kubernetes_io_worker) (container_memory_working_set_bytes{id="/"}) / 1.073741824e+09 labels: Current_Cluster_Memory_Usage: Current_Cluster_Memory_Usage OK -2m 18s ago0.426ms record:Current_Cluster_Disk_Usage expr:sum by(cluster, kubernetes_io_hostname, node_role_kubernetes_io_worker) (container_fs_usage_bytes{device=~"^/dev/.*$",id="/"}) / 1.073741824e+09 labels: Current_Cluster_Disk_Usage: Current_Cluster_Disk_Usage Thanks and Regards, Priyank -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/88394502-a6b2-4ea7-82b0-0d4daacf4a40n%40googlegroups.com.