[prometheus-users] Re: Run two node exporters on same server
> we are getting error like context deadline exceeded You haven't show the actual error, nor where you saw it. The most likely explanation I can see is simply that prometheus cannot communicate with node_exporter - for example, you've misconfigured the target or there is some sort of firewalling in between. To prove this, login to the prometheus server (or container where prometheus is running), and do: curl -h 'http://x.x.x.x:/metrics' where x.x.x.x: is the IP address and port that you've configured as the target to scrape. On Thursday, 21 April 2022 at 07:00:24 UTC+1 chembakay...@gmail.com wrote: > Hiii > > Thanks for your reply. we were using two different ports for two node > exporters. Actually one node exporter is running in container/pod and > another node exporter is running as a binary file. we are not getting any > issue with container/pod one. But when we are running with binary one we > are getting error like context deadline exceeded. > > our prometheus config file is as follows: > > scrape_interval: 2m > evaluation_interval: 15s > scrape_timeout: 2m > > could anyone please help me out of this? > > regards, > Bharath. > On Wednesday, 20 April 2022 at 17:50:01 UTC+5:30 Brian Candler wrote: > >> You need to be clearer about what you're doing and what errors you see. >> >> Yes, you could run two node_exporters on the same server, bound to >> different ports. However this is normally completely pointless, since >> they're both monitoring the same server, and exporting the same data. >> >> It is not recommended that you run node_exporter inside a container at >> all. But if you do, see the instructions here: >> https://github.com/prometheus/node_exporter#docker >> >> On Wednesday, 20 April 2022 at 11:59:27 UTC+1 chembakay...@gmail.com >> wrote: >> >>> Hii all, >>> >>> I want to run two node exporter on same server.. One is with kubernetes >>> and another one with binary file. each node exporter has different port >>> numbers. I am getting error like time out or context deadline exceeded. >>> Will it be possible if so sould you please tell me the solution? >>> >>> thanks >>> Bharath >>> >> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/9d9e37e8-88c9-4d7e-a75a-974c0af72ba0n%40googlegroups.com.
[prometheus-users] Re: Run two node exporters on same server
Hiii Thanks for your reply. we were using two different ports for two node exporters. Actually one node exporter is running in container/pod and another node exporter is running as a binary file. we are not getting any issue with container/pod one. But when we are running with binary one we are getting error like context deadline exceeded. our prometheus config file is as follows: scrape_interval: 2m evaluation_interval: 15s scrape_timeout: 2m could anyone please help me out of this? regards, Bharath. On Wednesday, 20 April 2022 at 17:50:01 UTC+5:30 Brian Candler wrote: > You need to be clearer about what you're doing and what errors you see. > > Yes, you could run two node_exporters on the same server, bound to > different ports. However this is normally completely pointless, since > they're both monitoring the same server, and exporting the same data. > > It is not recommended that you run node_exporter inside a container at > all. But if you do, see the instructions here: > https://github.com/prometheus/node_exporter#docker > > On Wednesday, 20 April 2022 at 11:59:27 UTC+1 chembakay...@gmail.com > wrote: > >> Hii all, >> >> I want to run two node exporter on same server.. One is with kubernetes >> and another one with binary file. each node exporter has different port >> numbers. I am getting error like time out or context deadline exceeded. >> Will it be possible if so sould you please tell me the solution? >> >> thanks >> Bharath >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/51faceba-0886-416a-a9ca-75fb3e40b9dbn%40googlegroups.com.
Re: [prometheus-users] Extracting long queries from multiple histograms
Victor Sudakov wrote:
>
> There is a web app which exports its metrics as multiple histograms,
> one histogram per Web endpoint. So each set of histogram data is also
> labelled by the {endpoint} label. There are about 50 endpoints so
> about 50 histograms.
>
> I would like to detect and graph slow endpoints, that is I would like
> to know the value of {endpoint} when its {le} is over 1s or something
> like that.
>
> Can you please help with a relevant PromQL query and an idea how to
> represent it in Grafana?
>
> I don't actually want 50 heatmaps, there must be a clever way to make
> an overview of all the slow endpoints, or all the endpoints with a
> particular status code etc.
An example. The PromQL query
`app1_response_duration_bucket{external_endpoint="http://YY/XX",status_code="200",method="GET"}`
produces a histogram.
The PromQL query
`app1_response_duration_bucket{external_endpoint="http://YY/XX",status_code="200",method="POST"}`
produces another histogram.
The query `app1_response_duration_bucket{{le="0.75"}` will return a
list of endpoints which have responded faster than 0.75s.
How do I invert the "le" and find the endpoints slower than "le"?
--
Victor Sudakov VAS4-RIPE
http://vas.tomsk.ru/
2:5005/49@fidonet
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/YmBsN4x/u/Oe7ozF%40admin.sibptus.ru.
[prometheus-users] Re: black box exporter monitoring SSH and PING
blackbox_exporter monitoring TCP ports (e.g. for SSH) and ICMP (ping) works
fine.
"but black box exporter detect the recover behavior after about 5mins"
Black box exporter only performs a single test when you scrape it. It does
not by itself do any recovery detection. The problem is therefore most
likely with your prometheus scrape config or your alertmanager config.
If you're having a problem, you'll need to be more specific:
* show your blackbox_exporter config, your prometheus scrape config which
scrapes it, your alerting rules, and your alertmanager config (if using
alertmanager)
* describe more clearly the behaviour you're seeing, and what you expected
to see. (For example, are you waiting for a "recovery" E-mail from
alertmanager?)
"And after the IP table is recovered, the alert for Ping can be cleared
after about 20mins, but SSH is still there."
Either SSH is working and reachable, or it is not. You can check the
results of blackbox_exporter tests by hand using curl, and also get
additional debugging information, like this:
curl -g 'http://127.0.0.1:9115/probe?module=xxx&target=&debug=true'
Here is an example:
# *curl -g
'http://localhost:9115/probe?module=icmp&target=1.2.3.4&debug=true'*
Logs for the probe:
ts=2022-04-20T12:25:11.587855449Z caller=main.go:320 module=icmp
target=1.2.3.4 level=info msg="Beginning probe" probe=icmp timeout_seconds=3
ts=2022-04-20T12:25:11.588014456Z caller=icmp.go:91 module=icmp
target=1.2.3.4 level=info msg="Resolving target address" ip_protocol=ip6
ts=2022-04-20T12:25:11.588065658Z caller=icmp.go:91 module=icmp
target=1.2.3.4 level=info msg="Resolving target address" ip_protocol=ip4
ts=2022-04-20T12:25:11.588098688Z caller=icmp.go:91 module=icmp
target=1.2.3.4 level=info msg="Resolved target address" ip=1.2.3.4
ts=2022-04-20T12:25:11.588133368Z caller=main.go:130 module=icmp
target=1.2.3.4 level=info msg="Creating socket"
ts=2022-04-20T12:25:11.588188673Z caller=main.go:130 module=icmp
target=1.2.3.4 level=debug msg="Unable to do unprivileged listen on socket,
will attempt privileged" err="socket: permission denied"
ts=2022-04-20T12:25:11.58829848Z caller=main.go:130 module=icmp
target=1.2.3.4 level=info msg="Creating ICMP packet" seq=24581 id=190
ts=2022-04-20T12:25:11.588348917Z caller=main.go:130 module=icmp
target=1.2.3.4 level=info msg="Writing out packet"
ts=2022-04-20T12:25:11.588470176Z caller=main.go:130 module=icmp
target=1.2.3.4 level=info msg="Waiting for reply packets"
ts=2022-04-20T12:25:14.588761946Z caller=main.go:130 module=icmp
target=1.2.3.4 level=debug msg="Cannot get TTL from the received packet.
'probe_icmp_reply_hop_limit' will be missing."
ts=2022-04-20T12:25:14.588979317Z caller=main.go:130 module=icmp
target=1.2.3.4 level=warn msg="Timeout reading from socket" err="read ip
0.0.0.0: raw-read ip4 0.0.0.0: i/o timeout"
ts=2022-04-20T12:25:14.589247538Z caller=main.go:320 module=icmp
target=1.2.3.4 level=error msg="Probe failed" duration_seconds=3.001307309
Metrics that would have been returned:
# HELP probe_dns_lookup_time_seconds Returns the time taken for probe dns
lookup in seconds
# TYPE probe_dns_lookup_time_seconds gauge
probe_dns_lookup_time_seconds 0.000116077
# HELP probe_duration_seconds Returns how long the probe took to complete
in seconds
# TYPE probe_duration_seconds gauge
probe_duration_seconds 3.001307309
# HELP probe_icmp_duration_seconds Duration of icmp request by phase
# TYPE probe_icmp_duration_seconds gauge
probe_icmp_duration_seconds{phase="resolve"} 0.000116077
probe_icmp_duration_seconds{phase="rtt"} 0
probe_icmp_duration_seconds{phase="setup"} 0.000212886
# HELP probe_ip_addr_hash Specifies the hash of IP address. It's useful to
detect if the IP address changes.
# TYPE probe_ip_addr_hash gauge
probe_ip_addr_hash 3.268949123e+09
# HELP probe_ip_protocol Specifies whether probe ip protocol is IP4 or IP6
# TYPE probe_ip_protocol gauge
probe_ip_protocol 4
# HELP probe_success Displays whether or not the probe was a success
# TYPE probe_success gauge
probe_success 0
Module configuration:
prober: icmp
timeout: 3s
http:
ip_protocol_fallback: true
follow_redirects: true
tcp:
ip_protocol_fallback: true
icmp:
ip_protocol_fallback: true
dns:
ip_protocol_fallback: true
Look at "probe_success" for the overall result.
You can also use the PromQL browser in the Prometheus web interface: enter
"probe_success" as the query and look at the graph tab. You'll see the
history of your blackbox exporter probes.
On Wednesday, 20 April 2022 at 12:37:17 UTC+1 ninag...@gmail.com wrote:
> Hi guys,
>
> We are using black box exporter to monitor ssh and ping.
>
> For ssh, (we monitor the port 22) if we stop sshd service, actually the
> service will be auto-recovered, but black box exporter detect the recover
> behavior after about 5mins.
>
> For ping, we use icmp module to monitor system ping, we deleted the IP
> tables, then Prometheus triggered 2 alerts, one is SSH
[prometheus-users] Re: Run two node exporters on same server
You need to be clearer about what you're doing and what errors you see. Yes, you could run two node_exporters on the same server, bound to different ports. However this is normally completely pointless, since they're both monitoring the same server, and exporting the same data. It is not recommended that you run node_exporter inside a container at all. But if you do, see the instructions here: https://github.com/prometheus/node_exporter#docker On Wednesday, 20 April 2022 at 11:59:27 UTC+1 chembakay...@gmail.com wrote: > Hii all, > > I want to run two node exporter on same server.. One is with kubernetes > and another one with binary file. each node exporter has different port > numbers. I am getting error like time out or context deadline exceeded. > Will it be possible if so sould you please tell me the solution? > > thanks > Bharath > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/c21b5195-fc75-4eb1-b3b5-9c9b7227f1bcn%40googlegroups.com.
[prometheus-users] black box exporter monitoring SSH and PING
Hi guys, We are using black box exporter to monitor ssh and ping. For ssh, (we monitor the port 22) if we stop sshd service, actually the service will be auto-recovered, but black box exporter detect the recover behavior after about 5mins. For ping, we use icmp module to monitor system ping, we deleted the IP tables, then Prometheus triggered 2 alerts, one is SSH is failed, the other is Ping is failed. And after the IP table is recovered, the alert for Ping can be cleared after about 20mins, but SSH is still there. So it is a good approach to use blackbox exporter to monitor SSH and PING? -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/9d2f4d9a-e0e4-430f-8dbc-f21f47a94372n%40googlegroups.com.
[prometheus-users] Run two node exporters on same server
Hii all, I want to run two node exporter on same server.. One is with kubernetes and another one with binary file. each node exporter has different port numbers. I am getting error like time out or context deadline exceeded. Will it be possible if so sould you please tell me the solution? thanks Bharath -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/819ea573-c76a-48a2-a181-df237fe63be6n%40googlegroups.com.
