Blackbox (0.17.0) running on Kubernetes, reporting DNS lookup failures on HTTP probes after running fine for few hours. Has anyone experienced such issue? Could someone help me to figure out the issue?
Sample probe output Logs for the probe: ts=2020-11-22T00:21:21.378113222Z caller=main.go:304 module=http_2xx target=https://arp-executor-sy-shra-arp-p.icl1p.xyz.com/actuator/health/ level=info msg="Beginning probe" probe=http timeout_seconds=9.5 ts=2020-11-22T00:21:21.378347979Z caller=http.go:323 module=http_2xx target=https://arp-executor-sy-shra-arp-p.icl1p.xyz.com/actuator/health/ level=info msg="Resolving target address" ip_protocol=ip4 ts=2020-11-22T00:21:30.878306074Z caller=http.go:323 module=http_2xx target=https://arp-executor-sy-shra-arp-p.icl1p.xyz.com/actuator/health/ level=error msg="Resolution with IP protocol failed" err="i/o timeout" ts=2020-11-22T00:21:30.878395746Z caller=main.go:119 module=http_2xx target=https://arp-executor-sy-shra-arp-p.icl1p.xyz.com/actuator/health/ level=error msg="Error resolving address" err="i/o timeout" ts=2020-11-22T00:21:30.878422453Z caller=main.go:304 module=http_2xx target=https://arp-executor-sy-shra-arp-p.icl1p.xyz.com/actuator/health/ level=error msg="Probe failed" duration_seconds=9.500237978 Metrics that would have been returned: HELP probe_dns_lookup_time_seconds Returns the time taken for probe dns lookup in secondsTYPE probe_dns_lookup_time_seconds gauge probe_dns_lookup_time_seconds 9.500014964 HELP probe_duration_seconds Returns how long the probe took to complete in secondsTYPE probe_duration_seconds gauge probe_duration_seconds 9.500237978 HELP probe_failed_due_to_regex Indicates if probe failed due to regexTYPE probe_failed_due_to_regex gauge probe_failed_due_to_regex 0 HELP probe_http_content_length Length of http content responseTYPE probe_http_content_length gauge probe_http_content_length 0 HELP probe_http_duration_seconds Duration of http request by phase, summed over all redirectsTYPE probe_http_duration_seconds gauge probe_http_duration_seconds{phase="connect"} 0 probe_http_duration_seconds{phase="processing"} 0 probe_http_duration_seconds{phase="resolve"} 0 probe_http_duration_seconds{phase="tls"} 0 probe_http_duration_seconds{phase="transfer"} 0 HELP probe_http_redirects The number of redirectsTYPE probe_http_redirects gauge probe_http_redirects 0 HELP probe_http_ssl Indicates if SSL was used for the final redirectTYPE probe_http_ssl gauge probe_http_ssl 0 HELP probe_http_status_code Response HTTP status codeTYPE probe_http_status_code gauge probe_http_status_code 0 HELP probe_http_uncompressed_body_length Length of uncompressed response bodyTYPE probe_http_uncompressed_body_length gauge probe_http_uncompressed_body_length 0 HELP probe_http_version Returns the version of HTTP of the probe responseTYPE probe_http_version gauge probe_http_version 0 HELP probe_ip_addr_hash Specifies the hash of IP address. It's useful to detect if the IP address changes.TYPE probe_ip_addr_hash gauge probe_ip_addr_hash 0 HELP probe_ip_protocol Specifies whether probe ip protocol is IP4 or IP6TYPE probe_ip_protocol gauge probe_ip_protocol 0 HELP probe_success Displays whether or not the probe was a successTYPE probe_success gauge probe_success 0 Module configuration: prober: http timeout: 10s http: valid_status_codes: - 100 - 200 - 201 - 202 - 203 - 204 - 205 - 206 - 207 - 208 - 226 - 300 - 301 - 302 - 303 - 304 - 305 - 306 - 307 - 308 valid_http_versions: - HTTP/1.1 - HTTP/2 preferred_ip_protocol: ip4 tcp: ip_protocol_fallback: true icmp: ip_protocol_fallback: true dns: ip_protocol_fallback: true -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/512f0663-12e7-4ac9-a03e-994d38974088n%40googlegroups.com.
