Re: [prometheus-users] Metrics vs log level

[Ben Kochie]

On Sat, Oct 8, 2022 at 4:22 AM Muthuveerappan Periyakaruppan <
muthu.veerap...@gmail.com> wrote:

> Please find replies inline.
>
> On Friday, 7 October, 2022 at 1:25:27 pm UTC+5:30 Stuart Clark wrote:
>
>> On 07/10/2022 04:09, Muthuveerappan Periyakaruppan wrote:
>> > we have a situation , where we have 8 to 15 million head series in
>> > each Prometheus and we have 7 instance of them (federated). Our
>> > prometheus are in a constant flooded situation handling the incoming
>> > metrics and back end recording rules.
>>
>> 8-15 million time series on a single Prometheus instance is pretty high.
>> What spec machine/pod are these?
>>
>> 90gb ram, 5000 millicores.
>

Wait, are you federating multiple Prometheus instances on multiple clusters
into one? Maybe you should look at Thanos instead. It lets you federate,
but without actually forcing you to put all the data in oneservice.

We have a Thanos setup with 250+ million metrics. Thousands of Prometheus
instances across multiple large Kubernetes clusters.


>
>
>> When you say "flooded" what are you meaning?
>>
>
> Always high usage of ram,  no oom , although missing metrics, average
> scrape duration like 35 seconds ... (may be due to no of targets/metrics)
> cpu demand/usage is not that high
>
>
>> > One thought which came to was - do we have something similar to log
>> > level for prometheus metrics ? If its there then... we can benefit
>> > from it  by configuring to run all targets in error level in
>> > production and in debug/info level in development... This will help
>> > control flooding of metrics.
>> >
>> I'm not sure what I understand what you are suggesting. What would be
>> the difference between setting this hypothetical "error" and "debug"
>> levels? Are you meaning some metrics would only be exposed on some
>> environments?
>>
>> Lets say every pod has close to 100 metrics , we may not need all of them
> in production ...
>

100 metrics per pod is not a lot. Is that really what you're using? That
means you have between 80k and 150k pods. Is that in a single cluster?

And you said you have a scrape duration of 35s. For 100 metrics per pod,
your scrape duration should be closer to 35 milliseconds.

Something in what you're saying doesn't add up.


> A developer before logging a metric can access on how useful this metric
> will be in production ...what indicators does it have - Utilization,
> Saturation, and Errors (USE) / Rate, Errors, and Duration (RED) ... based
> on this he can choose the metric level.
> Based on the level of metric,  only few can be enabled (ERROR / SEVERE
> level) in production the rest can be enabled (INFO /DEBUG Level) in
> development / testing / staging environments.
> few metrics should / are enough to troubleshoot and on demand we should
> have the option to change the metric level ...like log level at runtime to
> get more metrics
>
> --
>> Stuart Clark
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/577a43f4-3e8d-4c16-9061-3ba35699bd41n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CABbyFmpWiVoS6ZMqgcF0gnAovrfAj3D%3DGFAHZyG97D8FshGudw%40mail.gmail.com.

Re: [prometheus-users] Metrics vs log level

[Ben Kochie]

So, different opinion here.

Metrics are meant to tell you _when_ to debug. They're not meant to always
be debugging itself.

Metrics are supposed to tell you when it's time to get out other tools.
* Look at the logs
* Look at traces
* Look at profilers

Trying to get every dimension on every metric is just going to make metrics
bloated and useless for alerting. Which is what Prometheus is primarily for.


On Fri, Oct 7, 2022 at 12:05 PM 'Fabian Stäber' via Prometheus Users <
prometheus-users@googlegroups.com> wrote:

> Hi,
>
> Funny, I commented the same idea of having two separate registries on
> https://github.com/prometheus/client_java/issues/815 this morning.
>
> Currently you would just register each metric with two registries manually
>
> errorLevelRegistry.register(myCounter);
> debugLevelRegistry.register(myCounter);
>
> I'm not sure whether it's worthwhile to create API in client_java for
> making this a one-liner. Writing a custom method for this is just 4 lines
> of code.
>
> Fabian
>
> On Fri, Oct 7, 2022 at 11:43 AM Matthias Rampke 
> wrote:
>
>> > Say, If we write a wrapper on top of prometheus java client API, its
>> going to be messy
>>
>> You can make it relatively clean by creating (and incrementing) all the
>> metrics, but only calling .register() on those that you want to expose in
>> the given environment.
>>
>> Even more elaborately, you could have separate CollectorRegistry
>> instances, and register each metric with the one(s) appropriate for its
>> level. I think as it is, you will have to register "normal" level metrics
>> with both the "normal" and "debug" CollectorRegistry.
>>
>> I wonder (@fstab?) if it would make sense to have a
>> CollectorRegistryCollector, so that in effect you could do
>> `normalRegistry.register(debugRegistry)` and then decide when setting up
>> the Exporter which registry to serve, because all the "normal" metrics are
>> indirectly automatically registered with the debug registry. Or maybe that
>> exists and I couldn't find it?
>>
>> /MR
>>
>>
>> On Fri, Oct 7, 2022 at 3:09 AM Muthuveerappan Periyakaruppan <
>> muthu.veerap...@gmail.com> wrote:
>>
>>> Hi Team,
>>>
>>> we have a situation , where we have 8 to 15 million head series in each
>>> Prometheus and we have 7 instance of them (federated). Our prometheus are
>>> in a constant flooded situation handling the incoming metrics and back end
>>> recording rules.
>>>
>>> One thought which came to was - do we have something similar to log
>>> level for prometheus metrics ? If its there then... we can benefit from it
>>>  by configuring to run all targets in error level in production and in
>>> debug/info level in development... This will help control flooding of
>>> metrics.
>>>
>>> Say, If we write a wrapper on top of prometheus java client API, its
>>> going to be messy - hence wanted to check if this request makes sense or is
>>> there any other way out ?
>>>
>>> Let me know your thoughts how this can be achieved  Really need to
>>> hear from others on how this sort of situation is handled and whats the way
>>> to tackle ...
>>>
>>> fyr - We have raised the same issue @ prometheus java client project -
>>> https://github.com/prometheus/client_java/issues/815
>>>
>>>
>>> Many Thanks
>>> Muthuveerappan
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Prometheus Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to prometheus-users+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/prometheus-users/d3a1bb24-2d87-48c0-8b01-9f91a71dff7bn%40googlegroups.com
>>> 
>>> .
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to prometheus-users+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-users/CAMV%3D_gbgV5qmQmRVmqdkDuFx2gymz_vWjzxeK0h8oLfUgV51%3Dg%40mail.gmail.com
>> 
>> .
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/CAPX310iSucfPV79-5NTth-BhV2H9op_9R846AVZmsT1yFiVfgQ%40mail.gmail.com
> 

Re: [prometheus-users] Metrics vs log level

[Muthuveerappan Periyakaruppan]

Thanks a lot  Fabian, will check out and get back to you. 

On Friday, 7 October, 2022 at 3:35:28 pm UTC+5:30 fab...@fstab.de wrote:

> Hi,
>
> Funny, I commented the same idea of having two separate registries on 
> https://github.com/prometheus/client_java/issues/815 this morning.
>
> Currently you would just register each metric with two registries manually
>
> errorLevelRegistry.register(myCounter);
> debugLevelRegistry.register(myCounter);
>
> I'm not sure whether it's worthwhile to create API in client_java for 
> making this a one-liner. Writing a custom method for this is just 4 lines 
> of code.
>
> Fabian
>
> On Fri, Oct 7, 2022 at 11:43 AM Matthias Rampke  
> wrote:
>
>> > Say, If we write a wrapper on top of prometheus java client API, its 
>> going to be messy
>>
>> You can make it relatively clean by creating (and incrementing) all the 
>> metrics, but only calling .register() on those that you want to expose in 
>> the given environment.
>>
>> Even more elaborately, you could have separate CollectorRegistry 
>> instances, and register each metric with the one(s) appropriate for its 
>> level. I think as it is, you will have to register "normal" level metrics 
>> with both the "normal" and "debug" CollectorRegistry.
>>
>> I wonder (@fstab?) if it would make sense to have a 
>> CollectorRegistryCollector, so that in effect you could do 
>> `normalRegistry.register(debugRegistry)` and then decide when setting up 
>> the Exporter which registry to serve, because all the "normal" metrics are 
>> indirectly automatically registered with the debug registry. Or maybe that 
>> exists and I couldn't find it?
>>
>> /MR
>>
>>
>> On Fri, Oct 7, 2022 at 3:09 AM Muthuveerappan Periyakaruppan <
>> muthu.v...@gmail.com> wrote:
>>
>>> Hi Team,
>>>
>>> we have a situation , where we have 8 to 15 million head series in each 
>>> Prometheus and we have 7 instance of them (federated). Our prometheus are 
>>> in a constant flooded situation handling the incoming metrics and back end 
>>> recording rules.
>>>
>>> One thought which came to was - do we have something similar to log 
>>> level for prometheus metrics ? If its there then... we can benefit from it 
>>>  by configuring to run all targets in error level in production and in 
>>> debug/info level in development... This will help control flooding of 
>>> metrics.
>>>
>>> Say, If we write a wrapper on top of prometheus java client API, its 
>>> going to be messy - hence wanted to check if this request makes sense or is 
>>> there any other way out ?
>>>
>>> Let me know your thoughts how this can be achieved  Really need to 
>>> hear from others on how this sort of situation is handled and whats the way 
>>> to tackle ... 
>>>
>>> fyr - We have raised the same issue @ prometheus java client project - 
>>> https://github.com/prometheus/client_java/issues/815 
>>>
>>>
>>> Many Thanks
>>> Muthuveerappan 
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Prometheus Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to prometheus-use...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/prometheus-users/d3a1bb24-2d87-48c0-8b01-9f91a71dff7bn%40googlegroups.com
>>>  
>>> 
>>> .
>>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to prometheus-use...@googlegroups.com.
>>
> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/prometheus-users/CAMV%3D_gbgV5qmQmRVmqdkDuFx2gymz_vWjzxeK0h8oLfUgV51%3Dg%40mail.gmail.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/4a7dd007-6311-48d5-9858-f6f49942163en%40googlegroups.com.

Re: [prometheus-users] Metrics vs log level

[Muthuveerappan Periyakaruppan]

Interesting CollectorRegistryCollector 

On Friday, 7 October, 2022 at 3:13:50 pm UTC+5:30 matt...@prometheus.io 
wrote:

> > Say, If we write a wrapper on top of prometheus java client API, its 
> going to be messy
>
> You can make it relatively clean by creating (and incrementing) all the 
> metrics, but only calling .register() on those that you want to expose in 
> the given environment.
>
> Even more elaborately, you could have separate CollectorRegistry 
> instances, and register each metric with the one(s) appropriate for its 
> level. I think as it is, you will have to register "normal" level metrics 
> with both the "normal" and "debug" CollectorRegistry.
>
> I wonder (@fstab?) if it would make sense to have a 
> CollectorRegistryCollector, so that in effect you could do 
> `normalRegistry.register(debugRegistry)` and then decide when setting up 
> the Exporter which registry to serve, because all the "normal" metrics are 
> indirectly automatically registered with the debug registry. Or maybe that 
> exists and I couldn't find it?
>
> /MR
>
>
> On Fri, Oct 7, 2022 at 3:09 AM Muthuveerappan Periyakaruppan <
> muthu.v...@gmail.com> wrote:
>
>> Hi Team,
>>
>> we have a situation , where we have 8 to 15 million head series in each 
>> Prometheus and we have 7 instance of them (federated). Our prometheus are 
>> in a constant flooded situation handling the incoming metrics and back end 
>> recording rules.
>>
>> One thought which came to was - do we have something similar to log level 
>> for prometheus metrics ? If its there then... we can benefit from it  
>> by configuring to run all targets in error level in production and in 
>> debug/info level in development... This will help control flooding of 
>> metrics.
>>
>> Say, If we write a wrapper on top of prometheus java client API, its 
>> going to be messy - hence wanted to check if this request makes sense or is 
>> there any other way out ?
>>
>> Let me know your thoughts how this can be achieved  Really need to 
>> hear from others on how this sort of situation is handled and whats the way 
>> to tackle ... 
>>
>> fyr - We have raised the same issue @ prometheus java client project - 
>> https://github.com/prometheus/client_java/issues/815 
>>
>>
>> Many Thanks
>> Muthuveerappan 
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to prometheus-use...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/prometheus-users/d3a1bb24-2d87-48c0-8b01-9f91a71dff7bn%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/ba8c0c29-fd57-45da-963b-6ae8c9acae8dn%40googlegroups.com.

Re: [prometheus-users] Metrics vs log level

[Muthuveerappan Periyakaruppan]

We already dropped few that are unused ... even after that we ended up in 
that number ... 

Is there a way to find out / query unused metrics or least queried ones ? 
topk somehow does not work for us ... it gives different error every time. 
not able to spend time on that. do you have , any handy promql ? 

On Friday, 7 October, 2022 at 1:36:42 pm UTC+5:30 Brian Candler wrote:

> If you want to filter out some metrics, and you can't control this on the 
> exporter, then you can use metric relabelling 
> 
>  
> to drop any that you don't want to ingest into the database.
>
> On Friday, 7 October 2022 at 08:55:27 UTC+1 Stuart Clark wrote:
>
>> On 07/10/2022 04:09, Muthuveerappan Periyakaruppan wrote:
>> > we have a situation , where we have 8 to 15 million head series in 
>> > each Prometheus and we have 7 instance of them (federated). Our 
>> > prometheus are in a constant flooded situation handling the incoming 
>> > metrics and back end recording rules. 
>>
>> 8-15 million time series on a single Prometheus instance is pretty high. 
>> What spec machine/pod are these?
>>
>> When you say "flooded" what are you meaning?
>>
>> > One thought which came to was - do we have something similar to log 
>> > level for prometheus metrics ? If its there then... we can benefit 
>> > from it  by configuring to run all targets in error level in 
>> > production and in debug/info level in development... This will help 
>> > control flooding of metrics.
>> >
>> I'm not sure what I understand what you are suggesting. What would be 
>> the difference between setting this hypothetical "error" and "debug" 
>> levels? Are you meaning some metrics would only be exposed on some 
>> environments?
>>
>> -- 
>> Stuart Clark
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/455da200-161c-4119-85f3-a37d50e07559n%40googlegroups.com.

Re: [prometheus-users] Metrics vs log level

[Muthuveerappan Periyakaruppan]

Please find replies inline. 

On Friday, 7 October, 2022 at 1:25:27 pm UTC+5:30 Stuart Clark wrote:

> On 07/10/2022 04:09, Muthuveerappan Periyakaruppan wrote: 
> > we have a situation , where we have 8 to 15 million head series in 
> > each Prometheus and we have 7 instance of them (federated). Our 
> > prometheus are in a constant flooded situation handling the incoming 
> > metrics and back end recording rules. 
>
> 8-15 million time series on a single Prometheus instance is pretty high. 
> What spec machine/pod are these? 
>
> 90gb ram, 5000 millicores. 
 

> When you say "flooded" what are you meaning? 
>
 
Always high usage of ram,  no oom , although missing metrics, average 
scrape duration like 35 seconds ... (may be due to no of targets/metrics)
cpu demand/usage is not that high


> > One thought which came to was - do we have something similar to log 
> > level for prometheus metrics ? If its there then... we can benefit 
> > from it  by configuring to run all targets in error level in 
> > production and in debug/info level in development... This will help 
> > control flooding of metrics. 
> > 
> I'm not sure what I understand what you are suggesting. What would be 
> the difference between setting this hypothetical "error" and "debug" 
> levels? Are you meaning some metrics would only be exposed on some 
> environments? 
>
> Lets say every pod has close to 100 metrics , we may not need all of them 
in production ... 
A developer before logging a metric can access on how useful this metric 
will be in production ...what indicators does it have - Utilization, 
Saturation, and Errors (USE) / Rate, Errors, and Duration (RED) ... based 
on this he can choose the metric level. 
Based on the level of metric,  only few can be enabled (ERROR / SEVERE 
level) in production the rest can be enabled (INFO /DEBUG Level) in 
development / testing / staging environments. 
few metrics should / are enough to troubleshoot and on demand we should 
have the option to change the metric level ...like log level at runtime to 
get more metrics 

-- 
> Stuart Clark 
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/577a43f4-3e8d-4c16-9061-3ba35699bd41n%40googlegroups.com.

Re: [prometheus-users] Metrics vs log level

['Fabian Stäber' via Prometheus Users]

Hi,

Funny, I commented the same idea of having two separate registries on
https://github.com/prometheus/client_java/issues/815 this morning.

Currently you would just register each metric with two registries manually

errorLevelRegistry.register(myCounter);
debugLevelRegistry.register(myCounter);

I'm not sure whether it's worthwhile to create API in client_java for
making this a one-liner. Writing a custom method for this is just 4 lines
of code.

Fabian

On Fri, Oct 7, 2022 at 11:43 AM Matthias Rampke 
wrote:

> > Say, If we write a wrapper on top of prometheus java client API, its
> going to be messy
>
> You can make it relatively clean by creating (and incrementing) all the
> metrics, but only calling .register() on those that you want to expose in
> the given environment.
>
> Even more elaborately, you could have separate CollectorRegistry
> instances, and register each metric with the one(s) appropriate for its
> level. I think as it is, you will have to register "normal" level metrics
> with both the "normal" and "debug" CollectorRegistry.
>
> I wonder (@fstab?) if it would make sense to have a
> CollectorRegistryCollector, so that in effect you could do
> `normalRegistry.register(debugRegistry)` and then decide when setting up
> the Exporter which registry to serve, because all the "normal" metrics are
> indirectly automatically registered with the debug registry. Or maybe that
> exists and I couldn't find it?
>
> /MR
>
>
> On Fri, Oct 7, 2022 at 3:09 AM Muthuveerappan Periyakaruppan <
> muthu.veerap...@gmail.com> wrote:
>
>> Hi Team,
>>
>> we have a situation , where we have 8 to 15 million head series in each
>> Prometheus and we have 7 instance of them (federated). Our prometheus are
>> in a constant flooded situation handling the incoming metrics and back end
>> recording rules.
>>
>> One thought which came to was - do we have something similar to log level
>> for prometheus metrics ? If its there then... we can benefit from it 
>> by configuring to run all targets in error level in production and in
>> debug/info level in development... This will help control flooding of
>> metrics.
>>
>> Say, If we write a wrapper on top of prometheus java client API, its
>> going to be messy - hence wanted to check if this request makes sense or is
>> there any other way out ?
>>
>> Let me know your thoughts how this can be achieved  Really need to
>> hear from others on how this sort of situation is handled and whats the way
>> to tackle ...
>>
>> fyr - We have raised the same issue @ prometheus java client project -
>> https://github.com/prometheus/client_java/issues/815
>>
>>
>> Many Thanks
>> Muthuveerappan
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to prometheus-users+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-users/d3a1bb24-2d87-48c0-8b01-9f91a71dff7bn%40googlegroups.com
>> 
>> .
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/CAMV%3D_gbgV5qmQmRVmqdkDuFx2gymz_vWjzxeK0h8oLfUgV51%3Dg%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAPX310iSucfPV79-5NTth-BhV2H9op_9R846AVZmsT1yFiVfgQ%40mail.gmail.com.

Re: [prometheus-users] Metrics vs log level

[Matthias Rampke]

> Say, If we write a wrapper on top of prometheus java client API, its
going to be messy

You can make it relatively clean by creating (and incrementing) all the
metrics, but only calling .register() on those that you want to expose in
the given environment.

Even more elaborately, you could have separate CollectorRegistry instances,
and register each metric with the one(s) appropriate for its level. I think
as it is, you will have to register "normal" level metrics with both the
"normal" and "debug" CollectorRegistry.

I wonder (@fstab?) if it would make sense to have a
CollectorRegistryCollector, so that in effect you could do
`normalRegistry.register(debugRegistry)` and then decide when setting up
the Exporter which registry to serve, because all the "normal" metrics are
indirectly automatically registered with the debug registry. Or maybe that
exists and I couldn't find it?

/MR


On Fri, Oct 7, 2022 at 3:09 AM Muthuveerappan Periyakaruppan <
muthu.veerap...@gmail.com> wrote:

> Hi Team,
>
> we have a situation , where we have 8 to 15 million head series in each
> Prometheus and we have 7 instance of them (federated). Our prometheus are
> in a constant flooded situation handling the incoming metrics and back end
> recording rules.
>
> One thought which came to was - do we have something similar to log level
> for prometheus metrics ? If its there then... we can benefit from it 
> by configuring to run all targets in error level in production and in
> debug/info level in development... This will help control flooding of
> metrics.
>
> Say, If we write a wrapper on top of prometheus java client API, its going
> to be messy - hence wanted to check if this request makes sense or is there
> any other way out ?
>
> Let me know your thoughts how this can be achieved  Really need to
> hear from others on how this sort of situation is handled and whats the way
> to tackle ...
>
> fyr - We have raised the same issue @ prometheus java client project -
> https://github.com/prometheus/client_java/issues/815
>
>
> Many Thanks
> Muthuveerappan
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/d3a1bb24-2d87-48c0-8b01-9f91a71dff7bn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAMV%3D_gbgV5qmQmRVmqdkDuFx2gymz_vWjzxeK0h8oLfUgV51%3Dg%40mail.gmail.com.

Re: [prometheus-users] Metrics vs log level

[Brian Candler]

If you want to filter out some metrics, and you can't control this on the 
exporter, then you can use metric relabelling 

 
to drop any that you don't want to ingest into the database.

On Friday, 7 October 2022 at 08:55:27 UTC+1 Stuart Clark wrote:

> On 07/10/2022 04:09, Muthuveerappan Periyakaruppan wrote:
> > we have a situation , where we have 8 to 15 million head series in 
> > each Prometheus and we have 7 instance of them (federated). Our 
> > prometheus are in a constant flooded situation handling the incoming 
> > metrics and back end recording rules. 
>
> 8-15 million time series on a single Prometheus instance is pretty high. 
> What spec machine/pod are these?
>
> When you say "flooded" what are you meaning?
>
> > One thought which came to was - do we have something similar to log 
> > level for prometheus metrics ? If its there then... we can benefit 
> > from it  by configuring to run all targets in error level in 
> > production and in debug/info level in development... This will help 
> > control flooding of metrics.
> >
> I'm not sure what I understand what you are suggesting. What would be 
> the difference between setting this hypothetical "error" and "debug" 
> levels? Are you meaning some metrics would only be exposed on some 
> environments?
>
> -- 
> Stuart Clark
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/61e858bd-0701-4263-ac39-ed6942938c9bn%40googlegroups.com.

Re: [prometheus-users] Metrics vs log level

[Stuart Clark]

On 07/10/2022 04:09, Muthuveerappan Periyakaruppan wrote:
we have a situation , where we have 8 to 15 million head series in 
each Prometheus and we have 7 instance of them (federated). Our 
prometheus are in a constant flooded situation handling the incoming 
metrics and back end recording rules. 


8-15 million time series on a single Prometheus instance is pretty high. 
What spec machine/pod are these?


When you say "flooded" what are you meaning?

One thought which came to was - do we have something similar to log 
level for prometheus metrics ? If its there then... we can benefit 
from it  by configuring to run all targets in error level in 
production and in debug/info level in development... This will help 
control flooding of metrics.


I'm not sure what I understand what you are suggesting. What would be 
the difference between setting this hypothetical "error" and "debug" 
levels? Are you meaning some metrics would only be exposed on some 
environments?


--
Stuart Clark

--
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/17f3d5ca-4369-96ee-feb9-a4bbe0bc3ca1%40Jahingo.com.

[prometheus-users] Metrics vs log level

[Muthuveerappan Periyakaruppan]

 

Hi Team,

we have a situation , where we have 8 to 15 million head series in each 
Prometheus and we have 7 instance of them (federated). Our prometheus are 
in a constant flooded situation handling the incoming metrics and back end 
recording rules.

One thought which came to was - do we have something similar to log level 
for prometheus metrics ? If its there then... we can benefit from it  
by configuring to run all targets in error level in production and in 
debug/info level in development... This will help control flooding of 
metrics.

Say, If we write a wrapper on top of prometheus java client API, its going 
to be messy - hence wanted to check if this request makes sense or is there 
any other way out ?

Let me know your thoughts how this can be achieved  Really need to hear 
from others on how this sort of situation is handled and whats the way to 
tackle ... 

fyr - We have raised the same issue @ prometheus java client project - 
https://github.com/prometheus/client_java/issues/815 


Many Thanks
Muthuveerappan 

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/d3a1bb24-2d87-48c0-8b01-9f91a71dff7bn%40googlegroups.com.