[prometheus-users] Re: snmp_exporter drop metrics by metric value or label value

[Wang Yngwie]

I have tried, but it didn't work, thank you all the same, maybe I should 
read the source code deeply.

在2022年10月18日星期二 UTC+8 17:22:37 写道：

> Did it work? Which approach did you use?
>
> On Tuesday, 18 October 2022 at 07:48:06 UTC+1 yngwi...@gmail.com wrote:
>
>> Thanks  Brain, for your help.
>>
>> 在2022年10月17日星期一 UTC+8 16:45:17 写道：
>>
>>> > Hi, everyone. I want to drop some specific metrics by its value or its 
>>> label value. For example:
>>> > 1. the temperature metrics which values are 65535 meaning it's invalid
>>> > 2. the power metrics which "entPhysicalClass" label vale are not "6" 
>>> and "9"
>>>
>>> Metric relabeling 
>>> 
>>>  
>>> can be used to drop specific timeseries in the scrape response by label 
>>> value, but not by metric value.
>>>
>>> The only way I can think of dropping by metric value (without changing 
>>> the exporter output) is to use a recording rule 
>>> 
>>>  to 
>>> make a modified version of the timeseries, e.g.
>>>
>>> expr: some_temperature != 65535
>>>
>>> If you are happy to hack snmp.yml, you could try something like this 
>>> (untested):
>>>
>>> regex_extracts:
>>>   "":
>>>   - value: NaN
>>> regex: ^65535$
>>>   - value: $1
>>> regex: ^(.+)$
>>>
>>> I don't know if it's allowed to use "NaN" as a value here: source code 
>>> 
>>>  
>>> suggests it should work . However, a 
>>> time 
>>> series consisting of NaNs 
>>>  is not the 
>>> same as an empty/missing timeseries. So depending on your requirements, it 
>>> may be better to do
>>>
>>> regex_extracts:
>>>   "":
>>>   - value: INVALID
>>> regex: ^65535$
>>>   - value: $1
>>> regex: ^(.+)$
>>>
>>> although this will cause snmp_exporter to generate noisy logs at debug 
>>> level.
>>>
>>> Aside: if you look through the examples you can see regex being used to 
>>> divide a value by 10 (or by 100), e.g.:
>>>
>>> regex_extracts:
>>>   "":
>>>   - value: $1.$2
>>> regex: ^(?:(.*)(.))$
>>>
>>> > I write the snmp.yml without generator, didn't find a specification of 
>>> its syntax, does somebody know where it is?
>>>
>>>
>>> https://github.com/prometheus/snmp_exporter/blob/v0.20.0/generator/FORMAT.md
>>>
>>> Having said that, you may just want to run the generator and look at its 
>>> output to see what it emits :-)
>>>
>>> On Monday, 17 October 2022 at 08:01:08 UTC+1 yngwi...@gmail.com wrote:
>>>
 Hi, everyone. I want to drop some specific metrics by its value or its 
 label value. For example:
 1. the temperature metrics which values are 65535 meaning it's invalid
 2. the power metrics which "entPhysicalClass" label vale are not "6" 
 and "9"

 I write the snmp.yml without generator, didn't find a specification of 
 its syntax, does somebody know where it is?

 Appreciate for any help ：）

>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/3010874a-f821-45c5-857d-4ed1a6591b2en%40googlegroups.com.

Re: [prometheus-users] Guaranteed ingestion of metrics with historical timestamps

[Stuart Clark]

If you are trying to interface with another metrics system the Push Gateway 
isn't the right tool. The main use case for the Push Gateway is for batch jobs 
that aren't able to be directly scraped, but still have useful metrics. For 
systems which are constantly running you should instead look at direct 
instrumentation or the use of exporters.

Is this a custom metrics system, or something off the shelf and common? If so, 
there might already be an exporter available.

If you do need to make a custom exporter, I'd suggest looking at some of the 
similar existing ones (for example the Cloudwatch exporter) to see how they are 
made - but basically when a scrape request is received API calls would be made 
to your other metrics system to fetch the latest values, converted to 
Prometheus format (including the timestamp of that latest value from the other 
metric system) and returned. Prometheus would regularly scrape that exporter 
and add new values on a regular basis.

Alternatively, if the existing metric system already has extensive historical 
data which you'd like to be able to query (for dashboards and alerts) take a 
look at the remote read system. With this option Prometheus would use the 
remote system as an additional data source, running queries as needed (based on 
the PromQL queries it receives), combining the data with local information as 
needed. There are already remote read integrations available for some data 
stores. 

On 25 October 2022 18:24:56 BST, Omar Khazamov  wrote:
>Thanks, I'm importing metrics from our internal metrics system. Do you have
>any advice on how to push withthe explicit   timestamps?
>
>Le ven. 7 oct. 2022 à 13:30, Stuart Clark  a
>écrit :
>
>> On 07/10/2022 10:16, Omar Khazamov wrote:
>>
>> Hi Stuart,
>>
>> I can see that support of timestamps has been discontinued around November
>> 21st, 2021. Indeed, when I try
>>
>> C02G74F9Q6LR bat-datapipeline % echo "test_metric_with_timestamp 33
>> 1665623039" | curl --data-binary @- https://
>> /metrics/job/pushgateway-job
>>
>> I get  *"pushed metrics are invalid or inconsistent with existing
>> metrics: pushed metrics must not have timestamps"*
>>
>> Could you please specify how do you use timestamps in metrics? Thanks
>>
>> As mentioned before timestamps in general should not be used.
>>
>> You should always be publishing the "latest" value of any metric when
>> Prometheus scrapes the endpoint (or the push gateway in this case).
>>
>> --
>> Stuart Clark
>>
>>
>
>-- 
>Thanks,
>Omar Khazamov

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/B2EE7458-3DD5-4143-B0FF-BF45680855E2%40Jahingo.com.

Re: [prometheus-users] Guaranteed ingestion of metrics with historical timestamps

[Omar Khazamov]

Thanks, I'm importing metrics from our internal metrics system. Do you have
any advice on how to push withthe explicit   timestamps?

Le ven. 7 oct. 2022 à 13:30, Stuart Clark  a
écrit :

> On 07/10/2022 10:16, Omar Khazamov wrote:
>
> Hi Stuart,
>
> I can see that support of timestamps has been discontinued around November
> 21st, 2021. Indeed, when I try
>
> C02G74F9Q6LR bat-datapipeline % echo "test_metric_with_timestamp 33
> 1665623039" | curl --data-binary @- https://
> /metrics/job/pushgateway-job
>
> I get  *"pushed metrics are invalid or inconsistent with existing
> metrics: pushed metrics must not have timestamps"*
>
> Could you please specify how do you use timestamps in metrics? Thanks
>
> As mentioned before timestamps in general should not be used.
>
> You should always be publishing the "latest" value of any metric when
> Prometheus scrapes the endpoint (or the push gateway in this case).
>
> --
> Stuart Clark
>
>

-- 
Thanks,
Omar Khazamov

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAJs55iTbprcMSyKugNwdso92c8J%3DWOSa1L8E-rbsdjP-PK_Eiw%40mail.gmail.com.