Re: [prometheus-users] Re: blackbox metrics scraping
Hi Stuart, Brian I checked the blackbox exporter logs as below for module check_smtp, seems only every 30 secs, the next probe started. But actually I set the scrape_interval for this job is 60s.. 07:54:34 Beginning probe 07:55:05 Beginning probe 07:55:34 Beginning probe 07:56:05 Beginning probe 07:56:34 Beginning probe 07:57:05 Beginning probe 07:57:34 Beginning probe kubectl logs prometheus-blackbox-exporter -c blackbox-exporter -n monitoring | findstr "1.1.1.1:25" ts=2022-07-26T07:54:34.872Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Beginning probe" probe=tcp timeout_seconds=9.5 ts=2022-07-26T07:54:34.872Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Resolving target address" ip_protocol=ip6 ts=2022-07-26T07:54:34.872Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Resolving target address" ip_protocol=ip4 ts=2022-07-26T07:54:34.872Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Resolved target address" ip=1.1.1.1 ts=2022-07-26T07:54:34.872Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Dialing TCP without TLS" ts=2022-07-26T07:54:34.873Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Successfully dialed" ts=2022-07-26T07:54:34.873Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Processing query response entry" entry_number=0 ts=2022-07-26T07:54:34.874Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Read line" line="220" ts=2022-07-26T07:54:34.874Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Regexp matched" regexp="^220 ([^ ]+) ESMTP( .+)?$" line="220 ESMTP" ts=2022-07-26T07:54:34.874Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Sending line" line="EHLO prober\r" ts=2022-07-26T07:54:34.874Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Processing query response entry" entry_number=1 ts=2022-07-26T07:54:34.874Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Read line" line=250 ts=2022-07-26T07:54:34.875Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Read line" line=250-PIPELINING ts=2022-07-26T07:54:34.875Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Read line" line="250-SIZE 1024" ts=2022-07-26T07:54:34.875Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Read line" line=250-ETRN ts=2022-07-26T07:54:34.875Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Read line" line=250-STARTTLS ts=2022-07-26T07:54:34.875Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Regexp matched" regexp=^250-STARTTLS line=250-STARTTLS ts=2022-07-26T07:54:34.875Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Processing query response entry" entry_number=2 ts=2022-07-26T07:54:34.875Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Read line" line="250-AUTH PLAIN LOGIN" ts=2022-07-26T07:54:34.875Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Read line" line=250-ENHANCEDSTATUSCODES ts=2022-07-26T07:54:34.875Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Read line" line=250-8BITMIME ts=2022-07-26T07:54:34.875Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Read line" line="250 DSN" ts=2022-07-26T07:54:34.875Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Regexp matched" regexp="^250 .*$" line="250 DSN" ts=2022-07-26T07:54:34.875Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Sending line" line="STARTTLS\r" ts=2022-07-26T07:54:34.875Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Processing query response entry" entry_number=3 ts=2022-07-26T07:54:34.875Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Read line" line="220 2.0.0 Ready to start TLS" ts=2022-07-26T07:54:34.875Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Regexp matched" regexp=^220 line="220 2.0.0 Ready to start TLS" ts=2022-07-26T07:54:34.882Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="TLS Handshake (client) succeeded." ts=2022-07-26T07:54:34.882Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Processing query response entry" entry_number=4 ts=2022-07-26T07:54:34.882Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Sending line" line="EHLO prober\r"
Re: [prometheus-users] Re: blackbox metrics scraping
Hi Stuart, Brian I checked the blackbox exporter logs as below for module check_smtp, seems only every 30 secs, the next probe started. But actually I set the scrape_interval for this job is 60s.. 07:54:34 Beginning probe 07:55:05 Beginning probe 07:55:34 Beginning probe 07:56:05 Beginning probe 07:56:34 Beginning probe 07:57:05 Beginning probe 07:57:34 Beginning probe kubectl logs prometheus-blackbox-exporter -c blackbox-exporter -n monitoring | findstr "1.1.1.1:25" ts=2022-07-26T07:54:34.872Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Beginning probe" probe=tcp timeout_seconds=9.5 ts=2022-07-26T07:54:34.872Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Resolving target address" ip_protocol=ip6 ts=2022-07-26T07:54:34.872Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Resolving target address" ip_protocol=ip4 ts=2022-07-26T07:54:34.872Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Resolved target address" ip=100.70.222.244 ts=2022-07-26T07:54:34.872Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Dialing TCP without TLS" ts=2022-07-26T07:54:34.873Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Successfully dialed" ts=2022-07-26T07:54:34.873Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Processing query response entry" entry_number=0 ts=2022-07-26T07:54:34.874Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Read line" line="220" ts=2022-07-26T07:54:34.874Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Regexp matched" regexp="^220 ([^ ]+) ESMTP( .+)?$" line="220 ESMTP" ts=2022-07-26T07:54:34.874Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Sending line" line="EHLO prober\r" ts=2022-07-26T07:54:34.874Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Processing query response entry" entry_number=1 ts=2022-07-26T07:54:34.874Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Read line" line=250 ts=2022-07-26T07:54:34.875Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Read line" line=250-PIPELINING ts=2022-07-26T07:54:34.875Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Read line" line="250-SIZE 1024" ts=2022-07-26T07:54:34.875Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Read line" line=250-ETRN ts=2022-07-26T07:54:34.875Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Read line" line=250-STARTTLS ts=2022-07-26T07:54:34.875Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Regexp matched" regexp=^250-STARTTLS line=250-STARTTLS ts=2022-07-26T07:54:34.875Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Processing query response entry" entry_number=2 ts=2022-07-26T07:54:34.875Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Read line" line="250-AUTH PLAIN LOGIN" ts=2022-07-26T07:54:34.875Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Read line" line=250-ENHANCEDSTATUSCODES ts=2022-07-26T07:54:34.875Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Read line" line=250-8BITMIME ts=2022-07-26T07:54:34.875Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Read line" line="250 DSN" ts=2022-07-26T07:54:34.875Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Regexp matched" regexp="^250 .*$" line="250 DSN" ts=2022-07-26T07:54:34.875Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Sending line" line="STARTTLS\r" ts=2022-07-26T07:54:34.875Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Processing query response entry" entry_number=3 ts=2022-07-26T07:54:34.875Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Read line" line="220 2.0.0 Ready to start TLS" ts=2022-07-26T07:54:34.875Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Regexp matched" regexp=^220 line="220 2.0.0 Ready to start TLS" ts=2022-07-26T07:54:34.882Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="TLS Handshake (client) succeeded." ts=2022-07-26T07:54:34.882Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Processing query response entry" entry_number=4 ts=2022-07-26T07:54:34.882Z caller=main.go:180 module=smtp_starttls_server target=1.1.1.1:25 level=debug msg="Sending line" line="EHLO
Re: [prometheus-users] Re: blackbox metrics scraping
On 2022-07-25 09:18, nina guo wrote: Thank you Stuart. May I ask why the maximum if 2.5mins? By default Prometheus will look back for a maximum of 5 minutes to find the "most recent" data point. Therefore if there was no data recorded in the past 5 minutes a "no value" would be returned, and you'd have gaps in your graphs. The recommended maximum of about 2-2.5 minutes is to allow for a single scrape failure not to result in gaps as well as all the various processing times to actually do the scrape. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/aeae2e49c46a88f80c555ca4f46a99ec%40Jahingo.com.
Re: [prometheus-users] Re: blackbox metrics scraping
Thank you Stuart. May I ask why the maximum if 2.5mins? On Monday, July 25, 2022 at 3:52:47 PM UTC+8 Stuart Clark wrote: > On 25/07/2022 08:28, nina guo wrote: > > And one more question pls, I checked the log that the probe is sending > > every 2-3 seconds, can I adjust this frequency to about 1min? > Yes that's the scrape frequency, so you can adjust the job configuration > up to a maximum of about 2.5 minutes. > > -- > Stuart Clark > > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/531832c9-9e7f-4d99-a770-4cc4ea470d55n%40googlegroups.com.
Re: [prometheus-users] Re: blackbox metrics scraping
Thank you very much Stuart Got it. On Monday, July 25, 2022 at 3:52:47 PM UTC+8 Stuart Clark wrote: > On 25/07/2022 08:28, nina guo wrote: > > And one more question pls, I checked the log that the probe is sending > > every 2-3 seconds, can I adjust this frequency to about 1min? > Yes that's the scrape frequency, so you can adjust the job configuration > up to a maximum of about 2.5 minutes. > > -- > Stuart Clark > > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/f26cf27e-22dc-4f21-be1a-246e51d99c0en%40googlegroups.com.
Re: [prometheus-users] Re: blackbox metrics scraping
On 25/07/2022 08:28, nina guo wrote: And one more question pls, I checked the log that the probe is sending every 2-3 seconds, can I adjust this frequency to about 1min? Yes that's the scrape frequency, so you can adjust the job configuration up to a maximum of about 2.5 minutes. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/4ae19a2c-8e90-bb7a-102d-f16ef8de0c8d%40Jahingo.com.
Re: [prometheus-users] Re: blackbox metrics scraping
And one more question pls, I checked the log that the probe is sending every 2-3 seconds, can I adjust this frequency to about 1min? On Monday, July 25, 2022 at 3:23:29 PM UTC+8 nina guo wrote: > Got it thank you both Brian and Stuart. > > On Monday, July 25, 2022 at 2:52:20 PM UTC+8 Stuart Clark wrote: > >> On 25/07/2022 01:08, nina guo wrote: >> > Thank you Brian. " up to T - 5 minutes ", this 5 mins is the scraping >> > interval? >> >> No. The scraping interval doesn't matter. Prometheus will by default >> look back at most 5 minutes for a value. >> >> -- >> Stuart Clark >> >> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/8d10f32e-53d3-4f38-a6a1-86db18dcf112n%40googlegroups.com.
Re: [prometheus-users] Re: blackbox metrics scraping
On 25/07/2022 01:08, nina guo wrote: Thank you Brian. " up to T - 5 minutes ", this 5 mins is the scraping interval? No. The scraping interval doesn't matter. Prometheus will by default look back at most 5 minutes for a value. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/ec5af887-abba-b67c-af27-1024aea82736%40Jahingo.com.
[prometheus-users] Re: blackbox metrics scraping
Thank you Brian. " up to T - 5 minutes ", this 5 mins is the scraping
interval?
On Monday, July 25, 2022 at 4:43:33 AM UTC+8 Brian Candler wrote:
> The alerting rules run on their own schedule, separately from the scraping
> schedule.
>
> The expression "probe_success == 0" uses the value of that metric in the
> prometheus TSDB *at the current instant of time*. However, the value of a
> metric at any given time T is the most recent value *on or before* time T
> (up to T - 5 minutes).
>
> On Friday, 22 July 2022 at 05:01:18 UTC+1 ninag...@gmail.com wrote:
>
>> Hi dears,
>>
>> I'm trying to define an alert rule as below, and the scrape and
>> evaluation interval are 3m.
>>
>>
>> I checked from log, blackbox will send probe every 2-3second, then the
>> metrics will be generated every 2-3second, and the data will be compare
>> with the alert rules to dertermine an alert will be triggered or not?
>>
>> I have a question if Prometheus compares the Live data from blackbox
>> probe, or compares the data with Prometheus scrapped in last 3 mins?
>>
>> prometheus.yml: |-
>> global:
>> scrape_interval: 3m
>> evaluation_interval: 3m
>>
>> - alert: EndpointDown
>> expr: probe_success == 0
>> for: 5s
>> labels:
>> severity: critical
>> annotations:
>> description: Service {{ $labels.instance }} is unavailable.
>> value: DOWN ({{ $value }})
>> summary: "Endpointdown {{ $labels.instance }} is down."
>>
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/a894bd0f-0253-49de-91da-5a28003c0109n%40googlegroups.com.
[prometheus-users] Re: blackbox metrics scraping
The alerting rules run on their own schedule, separately from the scraping
schedule.
The expression "probe_success == 0" uses the value of that metric in the
prometheus TSDB *at the current instant of time*. However, the value of a
metric at any given time T is the most recent value *on or before* time T
(up to T - 5 minutes).
On Friday, 22 July 2022 at 05:01:18 UTC+1 ninag...@gmail.com wrote:
> Hi dears,
>
> I'm trying to define an alert rule as below, and the scrape and evaluation
> interval are 3m.
>
>
> I checked from log, blackbox will send probe every 2-3second, then the
> metrics will be generated every 2-3second, and the data will be compare
> with the alert rules to dertermine an alert will be triggered or not?
>
> I have a question if Prometheus compares the Live data from blackbox
> probe, or compares the data with Prometheus scrapped in last 3 mins?
>
> prometheus.yml: |-
> global:
> scrape_interval: 3m
> evaluation_interval: 3m
>
> - alert: EndpointDown
> expr: probe_success == 0
> for: 5s
> labels:
> severity: critical
> annotations:
> description: Service {{ $labels.instance }} is unavailable.
> value: DOWN ({{ $value }})
> summary: "Endpointdown {{ $labels.instance }} is down."
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/cba9c7ac-e5f6-4b79-bc19-b2c7f344821fn%40googlegroups.com.
