Please ignore my previous email.
I am repeating previous message again as i missed out setting under
modified configuration b):
Thank you for the response Brian,
with both config a and config b, i was unable to see the source label.
suggested config a)
metric_relabel_configs:
- source_labels: [ source, __address__ ]
regex: ";(.*)(:.*)?"
target_label: source
replacement: '${1}'
output:
*query: custom_metrics_from_server*
response:
custom_metrics_from_server{instance="localhost:9104", job="nodeexporter",
source="server1.example.com", status="pend"} 3215
custom_metrics_from_server{instance="localhost:9104", job="nodeexporter",
source="server1.example.com", status="run"} 3213
*query: up*
response:
up{instance="localhost:9104", job="nodeexporter"} 1
*query: node_cooling_device_max_state{instance="localhost:9104",
job="nodeexporter", name="0", type="Processor"}*
response:
node_cooling_device_max_state{instance="localhost:9104", job="nodeexporter"
, name="0", type="Processor"} 0
source= metrics was not available.
now to debug, i changed the query slightly , and it seems that the
__address__ is not available with metric_relabel_configs?
modified config a)
- source_labels: [ source, __address__ ]
regex: "(.*)"
target_label: source
replacement: '${1}'
*query: custom_metrics_from_server*
response:
custom_metrics_from_server{instance="localhost:9104", job="nodeexporter",
source="server1.example.com;", status="pend"} 3215
custom_metrics_from_server{instance="localhost:9104", job="nodeexporter",
source="server1.example.com;", status="run"} 3213
*query: up*
up{instance="localhost:9104", job="nodeexporter"}
*query: node_cooling_device_max_state{instance="localhost:9104",
job="nodeexporter", name="0", type="Processor"}*
node_cooling_device_max_state{instance="localhost:9104", job="nodeexporter"
, name="0", source=";", type="Processor"} 0
so from the last query, it seems that __address__ is blank, and since
source= is also blank, we see source=";" which appears to be concatenation
of 2 blank values with ; as delimiter.
Q1: here i am unable to understand that how ; was appended after source
value in *custom_metrics_from_server* .
config b) :
metric_relabel_configs:
- source_labels: [ source ]
regex: ""
target_label: __tmp_source_empty
replacement: Y
- source_labels: [ __tmp_source_empty, __address__ ]
regex: "Y;(.*)(:.*)?"
target_label: source
replacement: "${1}"
- target_label: __tmp_source_empty
replacement: ""
output:
*query: custom_metrics_from_server*
response:
custom_metrics_from_server{instance="localhost:9104", job="nodeexporter",
source="server1.example.com", status="pend"} 3215
custom_metrics_from_server{instance="localhost:9104", job="nodeexporter",
source="server1.example.com", status="run"} 3213
*query: up*
response:
up{instance="localhost:9104", job="nodeexporter"} 1
*query: node_cooling_device_max_state{instance="localhost:9104",
job="nodeexporter", name="0", type="Processor"}*
response:
node_cooling_device_max_state{instance="localhost:9104", job="nodeexporter"
, name="0", type="Processor"} 0
source= metrics was not available.
now to debug, i changed the query slightly , and it seems that the
__address__ is not available with metric_relabel_configs with config b?
modified config b)
metric_relabel_configs:
- source_labels: [ source ]
regex: ""
target_label: __tmp_source_empty
replacement: Y
- source_labels: [ __tmp_source_empty, __address__ ]
regex: "(.*)"
target_label: source
replacement: "${1}"
- target_label: __tmp_source_empty
replacement: ""
*query: custom_metrics_from_server*
custom_metrics_from_server{instance="localhost:9104", job="nodeexporter",
source=";", status="pend"} 3215
custom_metrics_from_server{instance="localhost:9104", job="nodeexporter",
source=";", status="run"} 3213
*query: up*
up{instance="localhost:9104", job="nodeexporter"}
*query: node_cooling_device_max_state{instance="localhost:9104",
job="nodeexporter", name="0", type="Processor"}*
node_cooling_device_max_state{instance="localhost:9104", job="nodeexporter"
, name="0", source="Y;", type="Processor"}
so from the last query, it seems that __address__ is blank, and since
source= is also blank, we see source=";" which appears to be concatenation
of 2 blank values with ; as delimiter.
so far , it appears that the __address__ field is not available with
metrics_relabel_config?
Please advice.
On Wednesday, March 12, 2025 at 2:00:40 AM UTC+5:30 mohan garden wrote:
> Thank you for the response Brian,
> with both config a and config b, i was unable to see the source label.
>
> suggested config a)
> metric_relabel_configs:
> - source_labels: [ source, __address__ ]
> regex: ";(.*)(:.*)?"
> target_label: source
> replacement: '${1}'
>
> output:
> *query: custom_metrics_from_server*
> response:
> custom_metrics_from_server{instance="localhost:9104", job="nodeexporter",
> source="server1.example.com", status="pend"} 3215
> custom_metrics_from_server{instance="localhost:9104", job="nodeexporter",
> source="server1.example.com", status="run"} 3213
>
> *query: up*
> response:
> up{instance="localhost:9104", job="nodeexporter"} 1
>
> *query: node_cooling_device_max_state{instance="localhost:9104",
> job="nodeexporter", name="0", type="Processor"}*
> response:
> node_cooling_device_max_state{instance="localhost:9104", job=
> "nodeexporter", name="0", type="Processor"} 0
>
>
> source= metrics was not available.
>
>
> now to debug, i changed the query slightly , and it seems that the
> __address__ is not available with metric_relabel_configs?
> modified config a)
> - source_labels: [ source, __address__ ]
> regex: "(.*)"
> target_label: source
> replacement: '${1}'
>
> *query: custom_metrics_from_server*
> response:
> custom_metrics_from_server{instance="localhost:9104", job="nodeexporter",
> source="server1.example.com;", status="pend"} 3215
> custom_metrics_from_server{instance="localhost:9104", job="nodeexporter",
> source="server1.example.com;", status="run"} 3213
>
> *query: up*
> up{instance="localhost:9104", job="nodeexporter"}
>
> *query: node_cooling_device_max_state{instance="localhost:9104",
> job="nodeexporter", name="0", type="Processor"}*
> node_cooling_device_max_state{instance="localhost:9104", job=
> "nodeexporter", name="0", source=";", type="Processor"} 0
>
> so from the last query, it seems that __address__ is blank, and since
> source= is also blank, we see source=";" which appears to be concatenation
> of 2 blank values with ; as delimiter.
> Q1: here i am unable to understand that how ; was appended after source
> value in *custom_metrics_from_server* .
>
>
>
>
> config b) :
> metric_relabel_configs:
> - source_labels: [ source ]
> regex: ""
> target_label: __tmp_source_empty
> replacement: Y
> - source_labels: [ __tmp_source_empty, __address__ ]
> regex: "Y;(.*)(:.*)?"
> target_label: source
> replacement: "${1}"
> - target_label: __tmp_source_empty
> replacement: ""
>
> output:
> *query: custom_metrics_from_server*
> response:
> custom_metrics_from_server{instance="localhost:9104", job="nodeexporter",
> source="server1.example.com", status="pend"} 3215
> custom_metrics_from_server{instance="localhost:9104", job="nodeexporter",
> source="server1.example.com", status="run"} 3213
>
> *query: up*
> response:
> up{instance="localhost:9104", job="nodeexporter"} 1
>
> *query: node_cooling_device_max_state{instance="localhost:9104",
> job="nodeexporter", name="0", type="Processor"}*
> response:
> node_cooling_device_max_state{instance="localhost:9104", job=
> "nodeexporter", name="0", type="Processor"} 0
>
>
> source= metrics was not available.
> now to debug, i changed the query slightly , and it seems that the
> __address__ is not available with metric_relabel_configs with config b?
>
>
> modified config b)
>
>
>
> *query: custom_metrics_from_server*
> custom_metrics_from_server{instance="localhost:9104", job="nodeexporter",
> source=";", status="pend"} 3215
> custom_metrics_from_server{instance="localhost:9104", job="nodeexporter",
> source=";", status="run"} 3213
>
> *query: up*
> up{instance="localhost:9104", job="nodeexporter"}
>
> *query: node_cooling_device_max_state{instance="localhost:9104",
> job="nodeexporter", name="0", type="Processor"}*
> node_cooling_device_max_state{instance="localhost:9104", job=
> "nodeexporter", name="0", source="Y;", type="Processor"}
>
>
> so from the last query, it seems that __address__ is blank, and since
> source= is also blank, we see source=";" which appears to be concatenation
> of 2 blank values with ; as delimiter.
> so far , it appears that the __address__ field is not available with
> metrics_relabel_config?
>
> Please advice.
>
>
> On Wednesday, March 12, 2025 at 12:24:11 AM UTC+5:30 Brian Candler wrote:
>
>> It's better to avoid honor_labels unless you really need it. It allows
>> the exporter to do bad things like overriding the "job" and "instance"
>> labels.
>>
>> Better to use metric_relabel_configs to relabel, at which point you have
>> the combined set of labels from service discovery and the scrape. A missing
>> label is the same as a label whose value is the empty string. So you can do
>> (untested):
>>
>> metric_relabel_configs:
>> - source_labels: [ source, __address__ ]
>> regex: ";(.*)(:.*)?"
>>
>> target_label: source
>> replacement: "${1}"
>>
>> This works as long as the source label's value cannot start with a
>> semicolon - if it can, define a different "separator" character. If you
>> don't like that, another approach is to set a temporary label as a flag:
>>
>> metric_relabel_configs:
>> - source_labels: [ source ]
>> regex: ""
>> target_label: __tmp_source_empty
>> replacement: Y
>> - source_labels: [ __tmp_source_empty, __address__ ]
>> regex: "Y;(.*)(:.*)?"
>> target_label: source
>> replacement: "${1}"
>> - target_label: __tmp_source_empty
>> replacement: ""
>>
>> On Tuesday, 11 March 2025 at 17:39:58 UTC mohan garden wrote:
>>
>>> Hi Everyone,
>>>
>>> I need to handle a scenario where I add a label to ingested metrics, but
>>> only if the label does not already exist in the collected data. If the
>>> label is present in a metric, it should remain unchanged.
>>>
>>> For example, given a set of metrics:
>>>
>>> custom_metrics_from_server{ source="server1.example.com",status="run" }
>>> 3213
>>> custom_metrics_from_server{ source="server1.example.com",status="pend"
>>> } 3215
>>>
>>> When additional metrics are collected from a node exporter, some will
>>> already have a "source=" label, while others will not:
>>>
>>> custom_metrics_from_server{ source="server1.example.com",status="run" }
>>> 3213
>>> custom_metrics_from_server{ source="server1.example.com",status="pend"
>>> } 3215
>>> node_cooling_device_max_state{name="7",type="Processor"} 0
>>>
>>> To ensure that all metrics contain the "source=" label without
>>> overriding existing ones, I figured the following approach using
>>> honor_labels and relabel_configs:
>>>
>>> honor_labels: true
>>> relabel_configs:
>>> - source_labels: [ _address_ ]
>>> regex: "(.*):(.*)"
>>> target_label: "source"
>>> replacement: '${1}'
>>>
>>> Is this the best way to handle this, or is there a more optimal approach?
>>> Please advice.
>>>
>>> Thanks!
>>>
>>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion visit
https://groups.google.com/d/msgid/prometheus-users/36fed9fb-5bc2-4e9f-b66d-e463dd66c507n%40googlegroups.com.
