If you only need part of the route, for example, the <unique projectid>,
you could do a label replacement, then drop the original label.
metric_relabel_configs:
- source_labels: [route]
regex: '.*/v1/project/([^/]+)/.*'
target_label: "project_id"
replacement: "$1"
- action: labeldrop
regex: ifDescr
On Friday, January 21, 2022 at 8:04:21 AM UTC-8 Brian Candler wrote:
> Dropping that label would only work if all instances of that metric have
> the same value for that label - in which case, there wasn't a cardinality
> problem in the first place. If there are distinct values, then dropping it
> will most likely end up with duplicate instances of the same timeseries,
> which will be rejected.
>
> The only way I can think of working around this is to write some
> intermediate proxy which collects all the metrics, sums across the
> appropriate dimension(s), and exposes the results as a new metric.
>
> But really, the answer is to push back. You are getting junk metrics
> which cause the TSDB to explode; therefore either the metrics need to be
> fixed, or you simply stop collecting them.
>
> On Friday, 21 January 2022 at 15:54:58 UTC Stuart Clark wrote:
>
>> On 21/01/2022 14:09, Ramachandra Bhaskar Ayalavarapu wrote:
>>
>> We have some data points like this
>> {
>> "__name__": "http_request_duration_seconds_bucket",
>> "endpoint": "apssd",
>> "instance": "10.10.77.218:8080",
>> "job": "api-service",
>> "le": "0.05",
>> "method": "GET",
>> "namespace": "foo",
>> "pod": "bar",
>> "route": "/localhsdssds/v1/project/<unique
>> projectid>/provider/WmRESTProvider/blasdsdsd/fummssd/icon",
>> "service": "api-service",
>> "status": "4XX"
>> }
>>
>>
>> unfortunately though we understand its not best practice to add "route"
>> in the label name which is driving cardinality we cannot do anything as
>> team is not ready to change code. Can i put a first aid any way to reduce
>> the cardinality. (this metric) is too important to drop
>>
>> some thing like project wise etc by writing rewrite rule. if yes please
>> advice how ?
>>
>> You could drop that label in metrics relabling.
>>
>> --
>> Stuart Clark
>>
>>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/7cf4ffcd-fd7b-475e-a7f2-3b3b6d6073a1n%40googlegroups.com.
