Re: [prometheus-users] metric_relabel_configs not dropping metrics

[Jihui Yang]

Thanks, I found the place to make the changes! Basically for each `kind: 
ServiceMonitor` add `metricRelabelings` to drop labels. Appreciate the help!

On Tuesday, February 21, 2023 at 9:51:19 AM UTC-8 Stuart Clark wrote:

> On 21/02/2023 17:29, Jihui Yang wrote:
> > I didn't find a way to adjust those. If I append scrape config jobs to 
> > the end of the config file, they should be able to overwrite existing 
> > job right?
>
> No. Job configurations are self contained. So metrics scraped by a 
> particular job will have any relabelling rules applied for that 
> particular job only. The only way you can set relabelling rules for a 
> job is by editing the job config. For the Prometheus Operator this is 
> done by adjusting the PodMonitor/ServiceMonitor objects.
>
> -- 
> Stuart Clark
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/05ff5635-840b-4d8d-8663-e495579c7664n%40googlegroups.com.

Re: [prometheus-users] metric_relabel_configs not dropping metrics

[Stuart Clark]

On 21/02/2023 17:29, Jihui Yang wrote:
I didn't find a way to adjust those. If I append scrape config jobs to 
the end of the config file, they should be able to overwrite existing 
job right?


No. Job configurations are self contained. So metrics scraped by a 
particular job will have any relabelling rules applied for that 
particular job only. The only way you can set relabelling rules for a 
job is by editing the job config. For the Prometheus Operator this is 
done by adjusting the PodMonitor/ServiceMonitor objects.


--
Stuart Clark

--
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/523da4f7-4772-b333-1c5d-2e78f345e305%40Jahingo.com.

Re: [prometheus-users] metric_relabel_configs not dropping metrics

[Jihui Yang]

I didn't find a way to adjust those. If I append scrape config jobs to the 
end of the config file, they should be able to overwrite existing job right?

On Monday, February 20, 2023 at 3:21:48 PM UTC-8 Stuart Clark wrote:

> On 20/02/2023 23:14, Jihui Yang wrote:
>
> I'm using prometheus-operator. It only allows loading 
> addionalScrapeConfigs 
> 
>  to 
> append to the end of the config file. The other config jobs were added as 
> part of loading prometheus-operator. I'm not sure I can change those.
>
> The other jobs are probably from PodMonitor & ServiceMonitor objects, so 
> you'd need to adjust those.
>
> -- 
> Stuart Clark
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/0ee9a62a-8178-4990-854c-0487aa0d054dn%40googlegroups.com.

Re: [prometheus-users] metric_relabel_configs not dropping metrics

[Stuart Clark]

On 20/02/2023 23:14, Jihui Yang wrote:
I'm using prometheus-operator. It only allows loading 
addionalScrapeConfigs 
 to 
append to the end of the config file. The other config jobs were added 
as part of loading prometheus-operator. I'm not sure I can change those.


The other jobs are probably from PodMonitor & ServiceMonitor objects, so 
you'd need to adjust those.


--
Stuart Clark

--
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/410d4240-2f08-510b-3a75-49be47cd77b9%40Jahingo.com.

Re: [prometheus-users] metric_relabel_configs not dropping metrics

[Jihui Yang]

I'm using prometheus-operator. It only allows loading addionalScrapeConfigs 

 to 
append to the end of the config file. The other config jobs were added as 
part of loading prometheus-operator. I'm not sure I can change those.
On Monday, February 20, 2023 at 3:07:25 PM UTC-8 Stuart Clark wrote:

> On 20/02/2023 22:33, Jihui Yang wrote:
> > I think these metrics are being scraped from another job. What I want 
> > is to drop any scraped metrics with names match the regex I provided
> Then you need to add the relabel config to that other job.
>
> -- 
> Stuart Clark
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/1e691810-a17f-4651-bb44-1dda3f96bf8fn%40googlegroups.com.

Re: [prometheus-users] metric_relabel_configs not dropping metrics

[Stuart Clark]

On 20/02/2023 22:33, Jihui Yang wrote:
I think these metrics are being scraped from another job. What I want 
is to drop any scraped metrics with names match the regex I provided

Then you need to add the relabel config to that other job.

--
Stuart Clark

--
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/354ed09b-b32c-4352-e786-cdca89c56a21%40Jahingo.com.

Re: [prometheus-users] metric_relabel_configs not dropping metrics

[Jihui Yang]

I think these metrics are being scraped from another job. What I want is to
drop any scraped metrics with names match the regex I provided

On Mon, Feb 20, 2023, 2:16 PM Stuart Clark  wrote:

> On 20/02/2023 19:10, Jihui Yang wrote:
> > Hi, so I added this section to match all namespaces:
> > ```
> > kubernetes_sd_configs:
> >  - role: endpoints kubeconfig_file: ""
> >follow_redirects: true
> >namespaces:
> >  names:
> >  - example1
> >  - example2
> >  - example3
> > ```
> > as well as
> > ```
> > authorization:
> >  type: Bearer
> >  credentials_file: /var/run/secrets/
> kubernetes.io/serviceaccount/token
> > ```
> > I turned on debug logging, and i'm getting
> > ```
> > ts=2023-02-20T19:08:30.169Z caller=scrape.go:1292 level=debug
> > component="scrape manager" scrape_pool=drop_response_metrics
> > target=http://10.10.188.252:25672/metrics msg="Scrape failed" err="Get
> > \"http://10.10.188.252:25672/metrics\": EOF"
> > ts=2023-02-20T19:08:30.465Z caller=scrape.go:1292 level=debug
> > component="scrape manager" scrape_pool=drop_response_metrics
> > target=http://10.10.152.96:10043/metrics msg="Scrape failed"
> > err="server returned HTTP status 500 Internal Server Error"
> > ts=2023-02-20T19:08:30.510Z caller=scrape.go:1292 level=debug
> > component="scrape manager" scrape_pool=drop_response_metrics
> > target=http://10.10.241.97:9100/metrics msg="Scrape failed"
> > err="server returned HTTP status 400 Bad Request"
> > ```
> >
> > The metrics are still not dropped
>
> I'm not really following exactly what your config is.
>
> Those errors suggest that at least some of the scrapes are failing.
>
> When you say "the metrics are still not dropped" are these metrics that
> are being scraped in this job?
>
> --
> Stuart Clark
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAG0H%2BQjT1rTguPLJGHBrrRyT9MAXjHbcDrs95_Ss6VMfVie%2BWw%40mail.gmail.com.

Re: [prometheus-users] metric_relabel_configs not dropping metrics

[Stuart Clark]

On 20/02/2023 19:10, Jihui Yang wrote:

Hi, so I added this section to match all namespaces:
```
kubernetes_sd_configs:
 - role: endpoints kubeconfig_file: ""
   follow_redirects: true
   namespaces:
     names:
         - example1
         - example2
         - example3
```
as well as
```
authorization:
     type: Bearer
     credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
```
I turned on debug logging, and i'm getting
```
ts=2023-02-20T19:08:30.169Z caller=scrape.go:1292 level=debug 
component="scrape manager" scrape_pool=drop_response_metrics 
target=http://10.10.188.252:25672/metrics msg="Scrape failed" err="Get 
\"http://10.10.188.252:25672/metrics\": EOF"
ts=2023-02-20T19:08:30.465Z caller=scrape.go:1292 level=debug 
component="scrape manager" scrape_pool=drop_response_metrics 
target=http://10.10.152.96:10043/metrics msg="Scrape failed" 
err="server returned HTTP status 500 Internal Server Error"
ts=2023-02-20T19:08:30.510Z caller=scrape.go:1292 level=debug 
component="scrape manager" scrape_pool=drop_response_metrics 
target=http://10.10.241.97:9100/metrics msg="Scrape failed" 
err="server returned HTTP status 400 Bad Request"

```

The metrics are still not dropped


I'm not really following exactly what your config is.

Those errors suggest that at least some of the scrapes are failing.

When you say "the metrics are still not dropped" are these metrics that 
are being scraped in this job?


--
Stuart Clark

--
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/4bd680fe-400e-de75-d137-7c96e25a08e0%40Jahingo.com.

Re: [prometheus-users] metric_relabel_configs not dropping metrics

[Jihui Yang]

Hi, so I added this section to match all namespaces:
```
kubernetes_sd_configs: 
 - role: endpoints kubeconfig_file: "" 
   follow_redirects: true 
   namespaces: 
 names: 
 - example1
 - example2
 - example3 
```
as well as 
```
authorization: 
 type: Bearer 
 credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
```
I turned on debug logging, and i'm getting 
```
ts=2023-02-20T19:08:30.169Z caller=scrape.go:1292 level=debug 
component="scrape manager" scrape_pool=drop_response_metrics 
target=http://10.10.188.252:25672/metrics msg="Scrape failed" err="Get 
\"http://10.10.188.252:25672/metrics\": EOF"
ts=2023-02-20T19:08:30.465Z caller=scrape.go:1292 level=debug 
component="scrape manager" scrape_pool=drop_response_metrics 
target=http://10.10.152.96:10043/metrics msg="Scrape failed" err="server 
returned HTTP status 500 Internal Server Error"
ts=2023-02-20T19:08:30.510Z caller=scrape.go:1292 level=debug 
component="scrape manager" scrape_pool=drop_response_metrics 
target=http://10.10.241.97:9100/metrics msg="Scrape failed" err="server 
returned HTTP status 400 Bad Request"
```

The metrics are still not dropped
On Monday, February 20, 2023 at 7:14:23 AM UTC-8 Stuart Clark wrote:

> On 17/02/2023 22:02, Jihui Yang wrote:
>
> I'm using prometheus-operator's addionalScrapeConfigs 
> 
>  to 
> add metric drop rules. Example: 
>
> ```
> - job_name: drop_response_metrics 
>   honor_timestamps: true 
>   scrape_interval: 30s 
>   scrape_timeout: 10s 
>   metrics_path: /metrics 
>   scheme: http 
>   follow_redirects: true 
>   metric_relabel_configs: 
>   - source_labels: [__name__] 
> separator: ; 
> regex: 
> (response_total|response_latency_ms_count|response_latency_ms_sum) 
> replacement: $1 
> action: drop
> ```
>
> The config is successfully loaded to prometheus and I can view it in 
> `/config` endpoint. But for some reason I still can see the metrics. can 
> you let me know what to do?
>
> Is that the full config? I'm not seeing a Service Discovery section (e.g. 
> Kubernetes or file based) to tell Prometheus where to scrape from.
>
> -- 
> Stuart Clark
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/9b59f300-5446-42c2-bf31-cd964dab0c24n%40googlegroups.com.

Re: [prometheus-users] metric_relabel_configs not dropping metrics

[Stuart Clark]

On 17/02/2023 22:02, Jihui Yang wrote:
I'm using prometheus-operator's addionalScrapeConfigs 
 to 
add metric drop rules. Example:


```
- job_name: drop_response_metrics
  honor_timestamps: true
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  follow_redirects: true
  metric_relabel_configs:
  - source_labels: [__name__]
    separator: ;
    regex: 
(response_total|response_latency_ms_count|response_latency_ms_sum)

    replacement: $1
    action: drop
```

The config is successfully loaded to prometheus and I can view it in 
`/config` endpoint. But for some reason I still can see the metrics. 
can you let me know what to do?


Is that the full config? I'm not seeing a Service Discovery section 
(e.g. Kubernetes or file based) to tell Prometheus where to scrape from.


--
Stuart Clark

--
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/797c0c2e-d0d2-8b1a-2067-305307d6ed2e%40Jahingo.com.