Re: [prometheus-users] Forced to use the Pushgateway as a workaround?

[a...@binoklo.com]

Hello Matthias,

Many thanks for the suggestion. I will give it a try, although I have the 
impression that the k8s API will also be off-limits. At the moment I just 
have a couple of services / pods, but it would be nice to experiment with 
agent-mode.

Cheers,

--
Adriano

On Sunday, April 17, 2022 at 7:33:47 AM UTC-3 matt...@prometheus.io wrote:

> If you can, deploy (a) Prometheus into the cluster itself. The easiest way 
> to manage that is using the Prometheus operator, but if that is not 
> possible, you can configure it directly using relabeling, as in this 
> example[0].
>
> This Prometheus can scrape the various targets. You have a few options 
> from there:
>
> You can use this directly, reaching it through the load balancer, or 
> through a Grafana deployed to the same cluster.
>
> Or use remote write to push to another Prometheus or other metric store. 
> In this case you can run the in-cluster Prometheus in the pared down agent 
> mode. This also works if you run e.g. one Prometheus per namespace.
>
> On the more complex but full featured end, you can use Thanos to tie 
> multiple servers in multiple clusters together with a long term store.
>
> What is appropriate for you depends on the size of your setup, what you 
> want monitor, and the restrictions that your admins impose. I hope this 
> gives you some pointers to discuss with them!
>
> Best,
> Matthias
>
>
>
>
> [0]: 
> https://github.com/prometheus/prometheus/blob/main/documentation/examples/prometheus-kubernetes.yml
>
> On Fri, Apr 15, 2022, 17:19 a...@binoklo.com  wrote:
>
>> Hello,
>>
>> I'd like to use Prometheus to monitor my (wrapped) k8s services, but 
>> unfortunately the admin won't let me connect directly to individual pods, I 
>> can just access them via load balancing.
>>
>> In this case, I guess I have to use the Pushgateway. It seems to be 
>> working, however timeseries for old pods persist (the "instance" label is a 
>> random string). I am thinking of creating a program to periodically delete 
>> them from the Pushgateway.
>>
>> This is not an ideal situation but I guess this is the best I can do. Or 
>> am I missing something?
>>
>> Thanks,
>>
>> --
>> Adriano
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to prometheus-use...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/prometheus-users/e009b2af-53e3-4c43-8d3c-64b3cc29041bn%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/03e43c7e-81f8-42d0-b7ab-5ba6646203fbn%40googlegroups.com.

Re: [prometheus-users] Forced to use the Pushgateway as a workaround?

[Matthias Rampke]

If you can, deploy (a) Prometheus into the cluster itself. The easiest way
to manage that is using the Prometheus operator, but if that is not
possible, you can configure it directly using relabeling, as in this
example[0].

This Prometheus can scrape the various targets. You have a few options from
there:

You can use this directly, reaching it through the load balancer, or
through a Grafana deployed to the same cluster.

Or use remote write to push to another Prometheus or other metric store. In
this case you can run the in-cluster Prometheus in the pared down agent
mode. This also works if you run e.g. one Prometheus per namespace.

On the more complex but full featured end, you can use Thanos to tie
multiple servers in multiple clusters together with a long term store.

What is appropriate for you depends on the size of your setup, what you
want monitor, and the restrictions that your admins impose. I hope this
gives you some pointers to discuss with them!

Best,
Matthias




[0]:
https://github.com/prometheus/prometheus/blob/main/documentation/examples/prometheus-kubernetes.yml

On Fri, Apr 15, 2022, 17:19 a...@binoklo.com  wrote:

> Hello,
>
> I'd like to use Prometheus to monitor my (wrapped) k8s services, but
> unfortunately the admin won't let me connect directly to individual pods, I
> can just access them via load balancing.
>
> In this case, I guess I have to use the Pushgateway. It seems to be
> working, however timeseries for old pods persist (the "instance" label is a
> random string). I am thinking of creating a program to periodically delete
> them from the Pushgateway.
>
> This is not an ideal situation but I guess this is the best I can do. Or
> am I missing something?
>
> Thanks,
>
> --
> Adriano
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/e009b2af-53e3-4c43-8d3c-64b3cc29041bn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAMV%3D_gbxxQCcF_k0qoagRwTxP4KJLd3Kd8fEVjvXLitgEpSwJA%40mail.gmail.com.

[prometheus-users] Forced to use the Pushgateway as a workaround?

[a...@binoklo.com]

Hello,

I'd like to use Prometheus to monitor my (wrapped) k8s services, but 
unfortunately the admin won't let me connect directly to individual pods, I 
can just access them via load balancing.

In this case, I guess I have to use the Pushgateway. It seems to be 
working, however timeseries for old pods persist (the "instance" label is a 
random string). I am thinking of creating a program to periodically delete 
them from the Pushgateway.

This is not an ideal situation but I guess this is the best I can do. Or am 
I missing something?

Thanks,

--
Adriano

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/e009b2af-53e3-4c43-8d3c-64b3cc29041bn%40googlegroups.com.