We already dropped few that are unused ... even after that we ended up in that number ...
Is there a way to find out / query unused metrics or least queried ones ? topk somehow does not work for us ... it gives different error every time. not able to spend time on that. do you have , any handy promql ? On Friday, 7 October, 2022 at 1:36:42 pm UTC+5:30 Brian Candler wrote: > If you want to filter out some metrics, and you can't control this on the > exporter, then you can use metric relabelling > <https://prometheus.io/docs/prometheus/latest/configuration/configuration/#metric_relabel_configs> > > to drop any that you don't want to ingest into the database. > > On Friday, 7 October 2022 at 08:55:27 UTC+1 Stuart Clark wrote: > >> On 07/10/2022 04:09, Muthuveerappan Periyakaruppan wrote: >> > we have a situation , where we have 8 to 15 million head series in >> > each Prometheus and we have 7 instance of them (federated). Our >> > prometheus are in a constant flooded situation handling the incoming >> > metrics and back end recording rules. >> >> 8-15 million time series on a single Prometheus instance is pretty high. >> What spec machine/pod are these? >> >> When you say "flooded" what are you meaning? >> >> > One thought which came to was - do we have something similar to log >> > level for prometheus metrics ? If its there then... we can benefit >> > from it .... by configuring to run all targets in error level in >> > production and in debug/info level in development... This will help >> > control flooding of metrics. >> > >> I'm not sure what I understand what you are suggesting. What would be >> the difference between setting this hypothetical "error" and "debug" >> levels? Are you meaning some metrics would only be exposed on some >> environments? >> >> -- >> Stuart Clark >> >> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/455da200-161c-4119-85f3-a37d50e07559n%40googlegroups.com.
