[prometheus-users] Prometheus remote write

[Simon S]

Hi all,
I would like to create a own application with remote write endpoint to send 
data from prometheus to this application. 

Does anybody knows how the rest endpoint needs to be defined?

BR
Simon

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/38ef82c3-f1ea-4815-be3c-e5c06f50f266n%40googlegroups.com.

Re: [prometheus-users] Re: blackbox metrics scraping

[Stuart Clark]

On 2022-07-25 09:18, nina guo wrote:

Thank you Stuart.

May I ask why the maximum if 2.5mins?



By default Prometheus will look back for a maximum of 5 minutes to find 
the "most recent" data point. Therefore if there was no data recorded in 
the past 5 minutes a "no value" would be returned, and you'd have gaps 
in your graphs. The recommended maximum of about 2-2.5 minutes is to 
allow for a single scrape failure not to result in gaps as well as all 
the various processing times to actually do the scrape.


--
Stuart Clark

--
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/aeae2e49c46a88f80c555ca4f46a99ec%40Jahingo.com.

Re: [prometheus-users] Re: blackbox metrics scraping

[nina guo]

Thank you Stuart.

May I ask why the maximum if 2.5mins?

On Monday, July 25, 2022 at 3:52:47 PM UTC+8 Stuart Clark wrote:

> On 25/07/2022 08:28, nina guo wrote:
> > And one more question pls, I checked the log that the probe is sending 
> > every 2-3 seconds, can I adjust this frequency to about 1min?
> Yes that's the scrape frequency, so you can adjust the job configuration 
> up to a maximum of about 2.5 minutes.
>
> -- 
> Stuart Clark
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/531832c9-9e7f-4d99-a770-4cc4ea470d55n%40googlegroups.com.

Re: [prometheus-users] Re: blackbox metrics scraping

[nina guo]

Thank you very much Stuart Got it.

On Monday, July 25, 2022 at 3:52:47 PM UTC+8 Stuart Clark wrote:

> On 25/07/2022 08:28, nina guo wrote:
> > And one more question pls, I checked the log that the probe is sending 
> > every 2-3 seconds, can I adjust this frequency to about 1min?
> Yes that's the scrape frequency, so you can adjust the job configuration 
> up to a maximum of about 2.5 minutes.
>
> -- 
> Stuart Clark
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/f26cf27e-22dc-4f21-be1a-246e51d99c0en%40googlegroups.com.

Re: [prometheus-users] Re: blackbox metrics scraping

[Stuart Clark]

On 25/07/2022 08:28, nina guo wrote:
And one more question pls, I checked the log that the probe is sending 
every 2-3 seconds, can I adjust this frequency to about 1min?
Yes that's the scrape frequency, so you can adjust the job configuration 
up to a maximum of about 2.5 minutes.


--
Stuart Clark

--
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/4ae19a2c-8e90-bb7a-102d-f16ef8de0c8d%40Jahingo.com.

[prometheus-users] Push gateway POST vs PUT

['Markos Fragkakis' via Prometheus Users]

Hello,

I have a job that periodically pushes multiple metrics (3 different metric 
names, each metric has multiple counts, each with a different combination 
of label values).

I push to

#{push_gateway_host}/metrics/job*/my_job/*instance/my_instance

My desired behavior would be for all previous pushes to be "forgotten". 
This is where I have trouble understanding the documentation with using PUT 
vs POST . According 
to the documentation:

> PUT is used to push a group of metrics. All metrics with the grouping key 
specified in the URL are replaced by the metrics pushed with PUT.

Reading this, my understanding is that PUT is what I need. However, below I 
also read this:

> POST works exactly like the PUT method but only metrics with the same 
name as the newly pushed metrics are replaced (among those with the same 
grouping key).

What does "metrics with the same name" mean? If my push contains 30 
measurements about 3 metrics (10 measurements for each metric, each for a 
different set of labels), which metrics are replaced?

Any help is appreciated.

Cheers,

Markos

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/32acdf4c-5e08-4094-a934-1b721cb3c634n%40googlegroups.com.

[prometheus-users] Push gateway PUT vs POST

[Markos Fragkakis]

Hello,

I have a scheduled task that periodically pushes metrics to the prometheus 
gateway.

My desired behaviour would be for each "push" to overwrite previous pushes, 
even if the metrics and labels pushed are not one-for-one compared to 
previous push.

Reading the documentation on PUT 
 vs POST 
 I see this:

> PUT is used to push a group of metrics. All metrics with the grouping key 
specified in the URL are replaced by the metrics pushed with PUT.

My understanding of the above is that PUT will do exactly what I want. 
However, in the POST section I read:

> POST works exactly like the PUT method but only metrics with the same 
name as the newly pushed metrics are replaced (among those with the same 
grouping key).

What does "metrics with the same name" mean? Let's say I have 2 consecutive 
pushes with the payloads below. What using POST would do vs using PUT? 
Which are the "metrics with the same name" that would be replaced? Would it 
be `external_ats_metric1`, regardless the fact that there are different 
labels in the 2nd push?

Thank you,

Markos

PUSH #1

# TYPE external_ats_metric1 gauge
external_ats_metric1{app=\"foo\",component=\"component1\",priority=\"low\"} 
1
external_ats_metric1{app=\"foo\",component=\"component2\",priority=\"low\"} 
2
external_ats_metric1{app=\"foo\",component=\"component3\",priority=\"low\"} 
3

# TYPE external_ats_metric2 gauge
external_ats_metric2{app=\"foo\",component=\"component1\",priority=\"low\"} 
1
external_ats_metric2{app=\"foo\",component=\"component2\",priority=\"low\"} 
2
external_ats_metric2{app=\"foo\",component=\"component3\",priority=\"low\"} 
3

PUSH #2

# TYPE external_ats_metric1 gauge
external_ats_metric1{app=\"foo\",component=\"component4\",priority=\"low\"} 
1
external_ats_metric1{app=\"foo\",component=\"component5\",priority=\"low\"} 
2
external_ats_metric1{app=\"foo\",component=\"component6\",priority=\"low\"} 
3

# TYPE external_ats_metric3 gauge
external_ats_metric3{app=\"foo\",component=\"component1\",priority=\"low\"} 
1
external_ats_metric3{app=\"foo\",component=\"component2\",priority=\"low\"} 
2
external_ats_metric3{app=\"foo\",component=\"component3\",priority=\"low\"} 
3

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/7d3796e9-6592-47af-8ed4-6c593ea67d36n%40googlegroups.com.

Re: [prometheus-users] Re: blackbox metrics scraping

[nina guo]

And one more question pls, I checked the log that the probe is sending 
every 2-3 seconds, can I adjust this frequency to about 1min?

On Monday, July 25, 2022 at 3:23:29 PM UTC+8 nina guo wrote:

> Got it thank you both Brian and Stuart.
>
> On Monday, July 25, 2022 at 2:52:20 PM UTC+8 Stuart Clark wrote:
>
>> On 25/07/2022 01:08, nina guo wrote: 
>> > Thank you Brian. " up to T - 5 minutes  ", this 5 mins is the scraping 
>> > interval? 
>>
>> No. The scraping interval doesn't matter. Prometheus will by default 
>> look back at most 5 minutes for a value. 
>>
>> -- 
>> Stuart Clark 
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/8d10f32e-53d3-4f38-a6a1-86db18dcf112n%40googlegroups.com.

Re: [prometheus-users] Re: blackbox metrics scraping

[Stuart Clark]

On 25/07/2022 01:08, nina guo wrote:
Thank you Brian. " up to T - 5 minutes  ", this 5 mins is the scraping 
interval?


No. The scraping interval doesn't matter. Prometheus will by default 
look back at most 5 minutes for a value.


--
Stuart Clark

--
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/ec5af887-abba-b67c-af27-1024aea82736%40Jahingo.com.

Re: [prometheus-users] Prom QL

[BHARATH KUMAR]

I am talking about grafana dashboard. I created a custom variable as 
follows:

CPU:
All : <=100,
lt 10 && gt 0 : >0<=10,
lt 30 && gt 10 : >10<=30

So this CPU filter will be added at the top of grafana dashboard. Now If I 
select ALL in CPU filter I am not able to find the unreachable servers list 
in the grafana dashboard as those servers are not present in prometheus 
Data Source(since these are in unreachable state).

My query:
((1 - avg(irate(node_cpu_seconds_total{mode="idle",instance=~"$ip"}[5m])) by 
(instance)) * 100)  
$CPU 

Here we are comparing the value in $cpu

Is there any other way to compare the values that are not present in 
prometheus data source so that when I click ALL option I can able to see 
all servers data.

Thanks & regards,
Bharath Kumar.
On Wednesday, 20 July 2022 at 18:56:58 UTC+5:30 Brian Candler wrote:

> No idea.  You haven't said what dashboard software you're using, nor what 
> queries you're using to build that dashboard.
>
> > I want to add one filter so that I can be able to know which servers are 
> least used or most used.
>
> Not sure what you mean by a "filter" in this context.  A PromQL query 
> using min() or max() will work over all the values which are present in the 
> instant vector, which as I said before, is of variable size.  It doesn't 
> have to have a fixed number of inputs.
>
> e.g. given this data
>
> [
> node_blah{instance="foo"} 123
> node_blah{instance="bar"} 456
> node_blah{instance="baz"} 789
> ]
>
> then
>
> min(node_blah) => 123
>
> On Wednesday, 20 July 2022 at 10:28:58 UTC+1 chembakay...@gmail.com wrote:
>
>> Thanks, Clark and Brian for your reply.
>>
>> I am using two data sources in my case. i.e Prometheus and Postgres.
>>
>> In my dashboard, there is a table that contains both Prometheus and 
>> Postgres data. In this table, there is a column name %cpu used which will 
>> be obtained from Prometheus.
>>
>> As Brain said, if the server goes down, we will not get the node level 
>> metrics and for that particular server, we will have Postgres data but, not 
>> Prometheus as the server was down.
>>
>> for example, my dashboard table is as follows:
>>  
>> IPCPU   %cpu   memory   memory_used   column1
>> column2column3   
>> 1.1.1.1 4  0.4%  40gb 60%
>> a b  c
>> 1.1.1.2 8  10%80gb30%
>> d e  f
>> 1.1.1.3  
>>h  i   j
>>
>>
>> the third server goes down, so we are not able to see the CPU and memory 
>> values, my question was I want to add one filter so that I can be able to 
>> know which servers are least used or most used.
>>
>> The CPU used for the third server will be no data as that server was 
>> down. can we do any comparison for these servers(servers who went down) so 
>> that I can filter these servers whose value will be null/no_data.
>>
>> Thanks & regards,
>> Bharath Kumar.
>>
>> On Wednesday, 20 July 2022 at 14:22:17 UTC+5:30 Brian Candler wrote:
>>
>>> And just to clarify slightly, there aren't really "null values" in 
>>> prometheus. A query like "node_blah" returns a *vector* of results, that 
>>> is, a variable number of values. e.g.
>>>
>>> [
>>> node_blah{instance="foo"} 123
>>> node_blah{instance="bar"} 456
>>> node_blah{instance="baz"} 789
>>> ]
>>>
>>> If node "baz" goes down, then a query at a later point in time may return
>>>
>>> [
>>> node_blah{instance="foo"} 124
>>> node_blah{instance="bar"} 457
>>> ]
>>>
>>> If you want to test for this specific condition, i.e. there is no 
>>> "node_blah" metric present for a specific instance "baz", then you can form 
>>> a rather awkward join query using absent() in conjunction with the "up" 
>>> metric as Stuart described.
>>>
>>> But usually, you just want to query the "up" metric itself.
>>>
>>> On Wednesday, 20 July 2022 at 09:38:58 UTC+1 Stuart Clark wrote:
>>>
 On 20/07/2022 08:49, BHARATH KUMAR wrote:

 Hello all, 

 I installed node exporters on many servers (around 300). Few of the 
 servers are unreachable. So because of that, we are unable to get the CPU, 
 and memory values of those servers.

 Now I want to add a filter in the Grafana dashboard to check the least 
 CPU used, most CPU used servers. But due to unreachability, we are not 
 getting values for a few servers.

 My question is 
 "*how to compare the output of the Prometheus query is NULL"*

 Generally, I am comparing the output of the prom query like 
 I) if the CPU usage is less than 10% then I am comparing like 
 query >=0<=10%
 ii) if the CPU usage is greater than 10% and less than 30% then I am 
 comparing like
 query >10<=30
 *similarly how to check the null values using the