Re: [prometheus-users] Re: Promteheus HA different metrics

[Stuart Clark]

On 2023-09-05 14:26, Анастасия Зель wrote:

yeah, i think scrape manually it will be useful but remember that its
k8s pods :)
i only have pod ip and i cant get it from prometheus node because they
are in different subnets. Pods subnet don't have access to outside
network.
so i dont know how i can scrape manually particular pod target from
prometheus server.



That would explain why it isn't working. You need to have network 
connectivity to all of your scrape targets from the Prometheus server. 
So if you have configured Prometheus to scrape every pod (via the 
Kubernetes SD for example) the Prometheus server will either need to be 
inside the cluster or connected to the same network mechanism as the 
pods.


--
Stuart Clark

--
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/4eb0b62f043f84563619eecb8ba0c307%40Jahingo.com.

Re: [prometheus-users] Re: Promteheus HA different metrics

[Ben Kochie]

On Mon, Sep 4, 2023 at 5:00 PM Brian Candler  wrote:

> On Monday, 4 September 2023 at 15:49:25 UTC+1 Анастасия Зель wrote:
>
> Hello, we use HA prometheus with two servers.
>
> You mean, two Prometheus servers with the same config, both scraping the
> same targets?
>
>
>
> The problem is we get different metrics in dashboards from this two
> servers.
>
> Small differences are to be expected.  That's because the two servers
> won't be scraping the targets at the same points in time.  If you see more
> significant differences, then please provide some examples.
>
>
>
> And we also scrape metrics from k8s, and some pods are not scraping
> because of error context deadline exceeded
>
> That basically means "scrape timed out".  The scrape hadn't completed
> within the "scrape_timeout:" value that you've set.  You'll need to look at
> your individual exporters and the failing scrape URLs: either the target is
> not reachable at all (e.g. firewalling or network configuration issue), or
> the target is taking too long to respond.
>
>
> Its differents pods on each server. In prometheus logs we dont see any of
> errors.
>
> Where *do* you see the "context deadline exceeded" errors then?
>

Usually on the `/targets` page.

Prometheus does not log scrape errors by default. I would love this to be a
configuration option, or even better, a per-job `scrape_configs` option.


> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/34cf1354-9e58-4517-8c3d-3301d4fc0236n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CABbyFmrsM%3DDjSu2Mjvkmhzo%3D5XNJbmNvDFPN3fScuVRBOkzs%3Dg%40mail.gmail.com.