Re: [prometheus-users] Target Server - Which Prometheus Server Is Scraping
Thanks Ben. That is a much easier way to do it than my round about way. Will keep that in mind next time. Stuart, I see what you are getting at now. We do not parse through the application log files. We wrote our exporters that can take in a list of scripts that generate metrics, translate into Prometheus format and get written to the DB. On Wednesday, February 16, 2022 at 3:35:09 AM UTC-6 Stuart Clark wrote: > On 16/02/2022 01:11, kekr...@gmail.com wrote: > > Stuart, I am not sure I understand the log files question. I am not > > aware of any log files related to the scrape itself. We do have log > > files related to the exporters running on the server but they do not > > capture the scrapes. I am trying to get details of what is going on > > on the target server itself, not so concerned about what the > > Prometheus server has log wise. > > > I was talking about logs from the application being scraped. > > -- > Stuart Clark > > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/c541e63e-ab92-43c6-b11c-4df83ca2318en%40googlegroups.com.
Re: [prometheus-users] Target Server - Which Prometheus Server Is Scraping
On 16/02/2022 01:11, kekr...@gmail.com wrote: Stuart, I am not sure I understand the log files question. I am not aware of any log files related to the scrape itself. We do have log files related to the exporters running on the server but they do not capture the scrapes. I am trying to get details of what is going on on the target server itself, not so concerned about what the Prometheus server has log wise. I was talking about logs from the application being scraped. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/bbc21e3c-3d86-d3d1-4573-871f019201a2%40Jahingo.com.
Re: [prometheus-users] Target Server - Which Prometheus Server Is Scraping
If you just need to know the scrape timestamps, you can query each
Prometheus instance like this:
up{instance="foo"}[1h]
This will return the timetsamps for each scrape over a given time range.
On Wed, Feb 16, 2022 at 2:11 AM kekr...@gmail.com
wrote:
> Stuart, I am not sure I understand the log files question. I am not aware
> of any log files related to the scrape itself. We do have log files
> related to the exporters running on the server but they do not capture the
> scrapes. I am trying to get details of what is going on on the target
> server itself, not so concerned about what the Prometheus server has log
> wise.
>
> If you will add some more detail around the question, I will be glad to
> answer.
>
> On Tuesday, February 15, 2022 at 4:41:41 PM UTC-6 kekr...@gmail.com wrote:
>
>> I got the Grafana date problem solved and have the scrape time history
>> for the metric created - the info I needed.
>>
>> Thank you Stuart and Brian for the assistance.
>>
>> Kevin
>>
>> On Tuesday, February 15, 2022 at 4:32:02 PM UTC-6 Stuart Clark wrote:
>>
>>> On 15/02/2022 22:29, kekr...@gmail.com wrote:
>>> > I am not seeing the frequency is more often than I expect. I am being
>>> > told a log file is being created by the scrapes in a temp directory
>>> > every minute. I am saying it is not Prometheus. So now i have to
>>> > prove it is not Prometheus.
>>> >
>>> > As an alternate solution, I am trying to use the Prometheus timestamp
>>> > function on the metric being created by the scrape in Grafana to get
>>> > the time history of the metric as proof. The thought being the time
>>> > difference between the metric history is 3 minutes. But I am having
>>> > trouble getting the value of the timestamp function to act as an epoch
>>> > date.If I use the value returned in a web epoch translator, it
>>> > translate to the correct date. If I multiple the value by 1000, as
>>> > you do every epoch date in Grafana, it actually multiplies the value
>>> > rather than putting it in human readable date format.
>>> I'm not clear if you are getting logs from these requests or not? I'd
>>> expect any request logs to include the path being requested, time &
>>> source IP. What do you see?
>>>
>>> --
>>> Stuart Clark
>>>
>>> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/4fdc7c08-bbbf-40bc-a26c-c791337155c2n%40googlegroups.com
>
> .
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/CABbyFmrgCAT%3D83%3Dba1mWCfQEHM2LqDj%2BVR6cNRYT8sYeJvuiDw%40mail.gmail.com.
Re: [prometheus-users] Target Server - Which Prometheus Server Is Scraping
Stuart, I am not sure I understand the log files question. I am not aware of any log files related to the scrape itself. We do have log files related to the exporters running on the server but they do not capture the scrapes. I am trying to get details of what is going on on the target server itself, not so concerned about what the Prometheus server has log wise. If you will add some more detail around the question, I will be glad to answer. On Tuesday, February 15, 2022 at 4:41:41 PM UTC-6 kekr...@gmail.com wrote: > I got the Grafana date problem solved and have the scrape time history for > the metric created - the info I needed. > > Thank you Stuart and Brian for the assistance. > > Kevin > > On Tuesday, February 15, 2022 at 4:32:02 PM UTC-6 Stuart Clark wrote: > >> On 15/02/2022 22:29, kekr...@gmail.com wrote: >> > I am not seeing the frequency is more often than I expect. I am being >> > told a log file is being created by the scrapes in a temp directory >> > every minute. I am saying it is not Prometheus. So now i have to >> > prove it is not Prometheus. >> > >> > As an alternate solution, I am trying to use the Prometheus timestamp >> > function on the metric being created by the scrape in Grafana to get >> > the time history of the metric as proof. The thought being the time >> > difference between the metric history is 3 minutes. But I am having >> > trouble getting the value of the timestamp function to act as an epoch >> > date.If I use the value returned in a web epoch translator, it >> > translate to the correct date. If I multiple the value by 1000, as >> > you do every epoch date in Grafana, it actually multiplies the value >> > rather than putting it in human readable date format. >> I'm not clear if you are getting logs from these requests or not? I'd >> expect any request logs to include the path being requested, time & >> source IP. What do you see? >> >> -- >> Stuart Clark >> >> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/4fdc7c08-bbbf-40bc-a26c-c791337155c2n%40googlegroups.com.
Re: [prometheus-users] Target Server - Which Prometheus Server Is Scraping
I got the Grafana date problem solved and have the scrape time history for the metric created - the info I needed. Thank you Stuart and Brian for the assistance. Kevin On Tuesday, February 15, 2022 at 4:32:02 PM UTC-6 Stuart Clark wrote: > On 15/02/2022 22:29, kekr...@gmail.com wrote: > > I am not seeing the frequency is more often than I expect. I am being > > told a log file is being created by the scrapes in a temp directory > > every minute. I am saying it is not Prometheus. So now i have to > > prove it is not Prometheus. > > > > As an alternate solution, I am trying to use the Prometheus timestamp > > function on the metric being created by the scrape in Grafana to get > > the time history of the metric as proof. The thought being the time > > difference between the metric history is 3 minutes. But I am having > > trouble getting the value of the timestamp function to act as an epoch > > date.If I use the value returned in a web epoch translator, it > > translate to the correct date. If I multiple the value by 1000, as > > you do every epoch date in Grafana, it actually multiplies the value > > rather than putting it in human readable date format. > I'm not clear if you are getting logs from these requests or not? I'd > expect any request logs to include the path being requested, time & > source IP. What do you see? > > -- > Stuart Clark > > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/6017e329-45a9-4b0a-bfa5-02de4a74713fn%40googlegroups.com.
Re: [prometheus-users] Target Server - Which Prometheus Server Is Scraping
On 15/02/2022 22:29, kekr...@gmail.com wrote: I am not seeing the frequency is more often than I expect. I am being told a log file is being created by the scrapes in a temp directory every minute. I am saying it is not Prometheus. So now i have to prove it is not Prometheus. As an alternate solution, I am trying to use the Prometheus timestamp function on the metric being created by the scrape in Grafana to get the time history of the metric as proof. The thought being the time difference between the metric history is 3 minutes. But I am having trouble getting the value of the timestamp function to act as an epoch date. If I use the value returned in a web epoch translator, it translate to the correct date. If I multiple the value by 1000, as you do every epoch date in Grafana, it actually multiplies the value rather than putting it in human readable date format. I'm not clear if you are getting logs from these requests or not? I'd expect any request logs to include the path being requested, time & source IP. What do you see? -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/e8c647e2-e295-9c59-01a7-9321b34d9962%40Jahingo.com.
Re: [prometheus-users] Target Server - Which Prometheus Server Is Scraping
I am not seeing the frequency is more often than I expect. I am being told a log file is being created by the scrapes in a temp directory every minute. I am saying it is not Prometheus. So now i have to prove it is not Prometheus. As an alternate solution, I am trying to use the Prometheus timestamp function on the metric being created by the scrape in Grafana to get the time history of the metric as proof. The thought being the time difference between the metric history is 3 minutes. But I am having trouble getting the value of the timestamp function to act as an epoch date.If I use the value returned in a web epoch translator, it translate to the correct date. If I multiple the value by 1000, as you do every epoch date in Grafana, it actually multiplies the value rather than putting it in human readable date format. Kevin On Tuesday, February 15, 2022 at 3:47:10 PM UTC-6 Stuart Clark wrote: > On 15/02/2022 18:41, kekr...@gmail.com wrote: > > My end goal is to prove monitoring is not running every minute on the > > server. My word saying it is not, there's no way, the job is not > > configured to run every minute, is not good enough. > > > > There is a possibility that the two Prometheus servers are scraping at > > the same time but there is no way the scrapes are happening every > > minute. The scrape interval is 3m with a scrape time out of 2m45s. > > So are you seeing more frequent requests than you expect? How are you > telling this? Do you have request logs & what do they say/record? > > -- > Stuart Clark > > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/9188e918-6750-43c3-80f7-7570803cad5fn%40googlegroups.com.
Re: [prometheus-users] Target Server - Which Prometheus Server Is Scraping
On 15/02/2022 18:41, kekr...@gmail.com wrote: My end goal is to prove monitoring is not running every minute on the server. My word saying it is not, there's no way, the job is not configured to run every minute, is not good enough. There is a possibility that the two Prometheus servers are scraping at the same time but there is no way the scrapes are happening every minute. The scrape interval is 3m with a scrape time out of 2m45s. So are you seeing more frequent requests than you expect? How are you telling this? Do you have request logs & what do they say/record? -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/bf39f5ed-4df9-b663-7000-94d1c58cab1d%40Jahingo.com.
Re: [prometheus-users] Target Server - Which Prometheus Server Is Scraping
My end goal is to prove monitoring is not running every minute on the server. My word saying it is not, there's no way, the job is not configured to run every minute, is not good enough. There is a possibility that the two Prometheus servers are scraping at the same time but there is no way the scrapes are happening every minute. The scrape interval is 3m with a scrape time out of 2m45s. On Tuesday, February 15, 2022 at 1:47:37 AM UTC-6 Brian Candler wrote: > You can add whatever query params you like in the scrape job, e.g. you > could add ?from=foo or ?from=bar as part of the URL being scraped. > > - job_name: blah > metrics_path: /metrics > params: > from: [ foo ] > ... > > However, I wonder how you intend to use this information. The data which > is scraped *should* be independent of who is scraping it, and it should be > possible to do additional scrapes without affecting the data (e.g. hitting > the exporter with "curl" to test it shouldn't alter the data for anyone > else). > > Therefore, I wonder if there's a better way to achieve what you're trying > to achieve. For example, if you are keeping a counter of "how many widgets > processed in the last minute", and resetting it to zero on each scrape, > then you should not be doing this; you should be keeping a counter which > just keeps incrementing. It's up to the consumer of the data to look at > data and work out the number of widgets processed per minute, or per hour > or whatever. Having the data in this format is much more useful anyway. > > If you can describe what it is you're doing, and why it matters where the > scrape is coming from, we may be able to give some alternative suggestions. > > On Tuesday, 15 February 2022 at 07:15:11 UTC Stuart Clark wrote: > >> On 15/02/2022 04:32, kekr...@gmail.com wrote: >> > >> > If you have multiple Prometheus servers using an identical target >> > list, is there a way on the target server to tell which Prometheus >> > server is scraping at the time the scrap occurs. >> > >> > For example, Prom_server_a and Prom_server_b scrape >> > target_server_123. On target_server_123 is there something on this >> > server that says Prom_server_a is scraping right now. Prom_server_b is >> > scraping right now? >> > >> The source IP address in any logs? >> >> -- >> Stuart Clark >> >> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/4136dffc-56f4-480c-ab54-0358b86864f2n%40googlegroups.com.
Re: [prometheus-users] Target Server - Which Prometheus Server Is Scraping
You can add whatever query params you like in the scrape job, e.g. you could add ?from=foo or ?from=bar as part of the URL being scraped. - job_name: blah metrics_path: /metrics params: from: [ foo ] ... However, I wonder how you intend to use this information. The data which is scraped *should* be independent of who is scraping it, and it should be possible to do additional scrapes without affecting the data (e.g. hitting the exporter with "curl" to test it shouldn't alter the data for anyone else). Therefore, I wonder if there's a better way to achieve what you're trying to achieve. For example, if you are keeping a counter of "how many widgets processed in the last minute", and resetting it to zero on each scrape, then you should not be doing this; you should be keeping a counter which just keeps incrementing. It's up to the consumer of the data to look at data and work out the number of widgets processed per minute, or per hour or whatever. Having the data in this format is much more useful anyway. If you can describe what it is you're doing, and why it matters where the scrape is coming from, we may be able to give some alternative suggestions. On Tuesday, 15 February 2022 at 07:15:11 UTC Stuart Clark wrote: > On 15/02/2022 04:32, kekr...@gmail.com wrote: > > > > If you have multiple Prometheus servers using an identical target > > list, is there a way on the target server to tell which Prometheus > > server is scraping at the time the scrap occurs. > > > > For example, Prom_server_a and Prom_server_b scrape > > target_server_123. On target_server_123 is there something on this > > server that says Prom_server_a is scraping right now. Prom_server_b is > > scraping right now? > > > The source IP address in any logs? > > -- > Stuart Clark > > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/89e37051-b5f9-4311-85c4-806721f7a921n%40googlegroups.com.
Re: [prometheus-users] Target Server - Which Prometheus Server Is Scraping
On 15/02/2022 04:32, kekr...@gmail.com wrote: If you have multiple Prometheus servers using an identical target list, is there a way on the target server to tell which Prometheus server is scraping at the time the scrap occurs. For example, Prom_server_a and Prom_server_b scrape target_server_123. On target_server_123 is there something on this server that says Prom_server_a is scraping right now. Prom_server_b is scraping right now? The source IP address in any logs? -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/17344755-91e8-954d-96d7-108040ef73c5%40Jahingo.com.
[prometheus-users] Target Server - Which Prometheus Server Is Scraping
If you have multiple Prometheus servers using an identical target list, is there a way on the target server to tell which Prometheus server is scraping at the time the scrap occurs. For example, Prom_server_a and Prom_server_b scrape target_server_123. On target_server_123 is there something on this server that says Prom_server_a is scraping right now. Prom_server_b is scraping right now? Thanks, Kevin -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/878a532a-283b-46ba-933e-0d4e8ea0c927n%40googlegroups.com.
