Re: [prometheus-users] PromQL redirection

[Ben Kochie]

This is somewhat how Thanos works.

It has a gRPC federating protocol that allows you to pull in metric data
from multiple TSDBs and glue the results together in the Thanos Query
engine.

The Thanos Sidecar already supports Prometheus remote read protocol, so it
could be a shim for what you need.

On Fri, Sep 6, 2024 at 5:28 PM Samit Jain  wrote:

>
> We've multiple segregated data systems which support PromQL, each storing
> metrics for different class of applications, infra, etc.
>
> We would like to explore the possibility of abstracting promql over these
> systems, such that user can run a query without knowing about the different
> backends. The options we considered below use something of a brute force
> approach and won't scale:
>
>1. support remote read API in all backends and configure Prometheus to
>remote read from all of them.
>2. send PromQL query to all backends and merge the results.
>
> I think a system where there is an external 'router' component which knows
> where different time series' are stored (some sort of an index table) and
> uses it to query the right backend would be worth exploring. We can presume
> for now that the time series are unique across all backends. Do you know of
> something like this exists in some form, or some literature on this that we
> could build upon?
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/f3c551fc-cfa2-496e-84e4-c96e26843244n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CABbyFmrUSJBROgERHjOFHJCN4Qdk7Vgwmovx6Rb0iFes4b2OuQ%40mail.gmail.com.

[prometheus-users] PromQL redirection

[Samit Jain]

We've multiple segregated data systems which support PromQL, each storing 
metrics for different class of applications, infra, etc.

We would like to explore the possibility of abstracting promql over these 
systems, such that user can run a query without knowing about the different 
backends. The options we considered below use something of a brute force 
approach and won't scale:

   1. support remote read API in all backends and configure Prometheus to 
   remote read from all of them.
   2. send PromQL query to all backends and merge the results.

I think a system where there is an external 'router' component which knows 
where different time series' are stored (some sort of an index table) and 
uses it to query the right backend would be worth exploring. We can presume 
for now that the time series are unique across all backends. Do you know of 
something like this exists in some form, or some literature on this that we 
could build upon?


-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/f3c551fc-cfa2-496e-84e4-c96e26843244n%40googlegroups.com.

[prometheus-users] PromQL - Check for specific value in the past

[fiala...@gmail.com]

Hi,

I have a metric bgp_state_info. Ok state is when metric has value = 3, 
other values (from 1 to 7) are considered as error.

I want to fire alert only for metrics that has value 3 at least only once. 
In other words I dont' want to fire alert for bgp that never worked.

Is it possible via promQL to do this? I have data retention 60 days and I'm 
aware of this limitations.

Thank you.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/66b0d62e-a2d7-461e-b451-abf02d7c01c3n%40googlegroups.com.

[prometheus-users] PromQL: understanding the and operator

[Puneet Singh]

Hi All, 
I have a metric called go_service_status where  i use the "sum without" 
operator to determine whether a service is up or down on a server. Now 
there can be a situation where service can be down simultaneously on 2 
master servers and I am unable to figure out a PromQL query to detect that 
situation. Example -  

*go_service_status{SERVICETYPE="grade1",SERVER_CATEGORY="db1",instance=~"server1:7878"}*
and it can have 2 possible series -
go_service_status{HOSTNAME="server1", SERVER_CATEGORY="db1", 
SERVICETYPE="grade1", USER="admin", instance="server1:7878", 
job="customprocessexporter01"} 0
go_service_status{HOSTNAME="server1", SERVER_CATEGORY="db1", 
SERVICETYPE="grade1", USER="root", instance="server1:7878", 
job="customprocessexporter01"} 1

and in the same way
*go_service_status{SERVICETYPE="grade1",SERVER_CATEGORY="db1",instance=~"server2:7878"}*
and it can have 2 possible series -
go_service_status{HOSTNAME="server2", SERVER_CATEGORY="db1", 
SERVICETYPE="grade1", USER="admin", instance="server2:7878", 
job="customprocessexporter01"} 0
go_service_status{HOSTNAME="server2", SERVER_CATEGORY="db1", 
SERVICETYPE="grade1", USER="root", instance="server2:7878", 
job="customprocessexporter01"} 0  


Here;s the query using which i figure out status of the service on 
server1.  Example - 

(sum without (USER) (
*go_service_status{HOSTNAME="server1",SERVER_CATEGORY="db1",SERVICETYPE="grade1"}*)
 
< 1)[image: Untitled.png]

so the server1's service is momentarily 0


and server2's service is always down , example - 
(sum without (USER) (
*go_lsf_service_status{HOSTNAME="server2",SERVER_CATEGORY="db1",SERVICETYPE="grade1"}*)
 
< 1)[image: Untitled.png]


Now i tried to find the time duration where both these service were 
simultaneously down / 0 on both server1 and server2 :
(sum without (USER) (
*go_service_status{HOSTNAME="server1",SERVER_CATEGORY="db1",SERVICETYPE="grade1"}*)
 
< 1) and (sum without (USER) (
*go_service_status{HOSTNAME="server2",SERVER_CATEGORY="db1",SERVICETYPE="grade1"}*)
 
< 1)


I was expecting a graph similar to the once for server2 , but i got :
[image: Untitled.png]

I think i need to ignore the HOSTNAME label , but unable to figure out the 
way to ignore the HOSTNAME label in combination with sum without clause.

Any help/hint to improve this query will be very useful for me to 
understand the and condition in context of sum without  clause.

Thanks,
Puneet

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/ba97a288-4344-4b4e-b901-807e697440c9n%40googlegroups.com.

[prometheus-users] PromQL filter based on current date

[Dipesh J]

Hi, 

Is there way to get metrics only for current date instead of using time 
like [24h] which would probably give metrics for day before too.

last 24 hours
my_metric{node="ABC"} [24h]

Something like below to give metrics for current date only.
my_metric{node="ABC"} [TODAYS_DATE]


Thanks

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/8a88c578-f187-4de6-b7df-f10cd379c77cn%40googlegroups.com.

[prometheus-users] PROMQL replace values

[Pablo Marques]

Has anyone ever done any logic like this, or would have any idea how I can 
solve this.

I use WriteLatency{DBInstanceIdentifier=~"aurora-mysql.*"} > 0 to filter 
who is the *master* of my database.

The result of *Query 1* will be: 
WriteLatency{DBInstanceIdentifier="aurora-mysql-2"} where *aurora-mysql-2* is 
the *master* database.

To validate the uptime, I use the following query 
mysql_global_status_uptime{instance=" 
"}.

I need the result of the *DBInstanceIdentifier* label in *Query 1* to be 
inserted as a value in *Query 2* in the *instance* field. That way, I can 
have a query in which I check the uptime of the master database only.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/b1a350b2-ee33-44d4-a1da-570aa168b1efn%40googlegroups.com.

Re: [prometheus-users] PromQL formatter?

['Brian Candler' via Prometheus Users]

Yes, but what guarantee do you have that the ChatGPT output is actually 
equivalent to your original input? ChatGPT is famous for making stuff up.

On Friday, 24 November 2023 at 12:05:13 UTC Sam Jewell wrote:

> I used ChatGPT today, and found it gave a SUPER neat formatting to my code.
>
> Thanks Facetoe for https://github.com/facetoe/promformat - I tried that 
> too, and I like it. But actually I found the ChatGPT formatting to be even 
> more readable (40 lines instead of 110 lines)
>
> Cheers, Sam
>
> On Saturday, 2 July 2022 at 10:05:57 UTC+1 Facetoe wrote:
>
> I wrote a formatter for PromQL people might find useful - 
> https://github.com/facetoe/promformat
>
> On Monday, December 27, 2021 at 9:23:27 PM UTC+8 bitz...@gmail.com wrote:
>
> Is there any update of this tool?
> 在2020年9月15日星期二 UTC+8 04:59:12 写道：
>
> There's a work-in-progress design doc for this:
>
>
> https://docs.google.com/document/d/1nOBjpuCk4CsrOSm2ZjfVz2EL6gmA_CFGSbHCdY0Royg/edit
>
> On Mon, Sep 14, 2020 at 8:27 PM 'pin...@hioscar.com' via Prometheus Users 
>  wrote:
>
> What's a good formatter you guys recommend for formatting complex queries?
>
> Ideally with line breaks and proper indentations after each ( or {.
>
> Thanks!
>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/prometheus-users/8b6cd4e4-24be-4066-9ee0-9d38850bbef3n%40googlegroups.com
>  
> 
> .
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/8f56366e-8a50-4b1a-bee0-26ea87a3ae51n%40googlegroups.com.

Re: [prometheus-users] PromQL formatter?

['Sam Jewell' via Prometheus Users]

I used ChatGPT today, and found it gave a SUPER neat formatting to my code.

Thanks Facetoe for https://github.com/facetoe/promformat - I tried that 
too, and I like it. But actually I found the ChatGPT formatting to be even 
more readable (40 lines instead of 110 lines)

Cheers, Sam

On Saturday, 2 July 2022 at 10:05:57 UTC+1 Facetoe wrote:

I wrote a formatter for PromQL people might find useful - 
https://github.com/facetoe/promformat

On Monday, December 27, 2021 at 9:23:27 PM UTC+8 bitz...@gmail.com wrote:

Is there any update of this tool?
在2020年9月15日星期二 UTC+8 04:59:12 写道：

There's a work-in-progress design doc for this:

https://docs.google.com/document/d/1nOBjpuCk4CsrOSm2ZjfVz2EL6gmA_CFGSbHCdY0Royg/edit

On Mon, Sep 14, 2020 at 8:27 PM 'pin...@hioscar.com' via Prometheus Users <
promethe...@googlegroups.com> wrote:

What's a good formatter you guys recommend for formatting complex queries?

Ideally with line breaks and proper indentations after each ( or {.

Thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an 
email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/8b6cd4e4-24be-4066-9ee0-9d38850bbef3n%40googlegroups.com
 

.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/e2daef7b-32fd-418f-9385-17ebc04d6377n%40googlegroups.com.

[prometheus-users] promql - what is promql for calcuate percetile

[Prashant Singh]

Hi , 

Need to be know what is promql for calculate p50th ,p90th , and p99th 
latency or perctile for bleow metrics

Note - I have no bucket metrics for histogram. 


# HELP prometheus_rule_evaluation_duration_seconds The duration for a rule 
to execute.
# TYPE prometheus_rule_evaluation_duration_seconds summary
prometheus_rule_evaluation_duration_seconds{quantile="0.5"} 6.4853e-05
prometheus_rule_evaluation_duration_seconds{quantile="0.9"} 0.00010102
prometheus_rule_evaluation_duration_seconds{quantile="0.99"} 0.000177367
prometheus_rule_evaluation_duration_seconds_sum 1.623860968846092e+06
prometheus_rule_evaluation_duration_seconds_count 1.112293682e+09

Thanks
Prashant
Thanks,
Prashant

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/04fb764d-aeda-45b5-b986-1ead519f995en%40googlegroups.com.

[prometheus-users] PromQL for location status geohash

[Nick Carlton]

I am using the below PromQL to get the percentage of devices that are UP
within sites mapped to geolocation and mapping it with the Geomap panel in
Grafana:

‘(count by (geohash,location) (up==1) / count by (geohash,location) (up)) *
100’

Is there a way to also get the value of each of the ‘count by’ instances
into the final output? So I would end up with something like:

’{geohash=“x”,location=“x”,total_devices=“x”,up_devices=“x”} 99%’

The reason for is this so that the values show up as a reference on the
Geomap tooltip when you hover over a location, no calculations will be done
with the numbers, hence wanting them as label values.

Is this possible? And if so, please could someone assist with the necessary
PromQL?


Thanks

Nick

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAC4WY5-xCVM42wYHfeG7%3DuSkbizFZv0B60nE%2BTbtJzOvZznQtA%40mail.gmail.com.

[prometheus-users] PromQL

[ritesh patel]

Hello Team,

Need help from you for PromQL query.

I want to create alert rule on prometheus alert manager and i have
prometheus query

Expr- sum by (host_name) (disk_used_percent) > 85 for warning ⚠️ level  and
same query with threshold > 95 critical level.

Now for that i need to create 2 rule for each severity. Now i have question
can we create one query for both severity like range between 85-95 warring
and 95 up critical? If yes can someone help me to write one sample query
example. So using one query i can setup alert rule.

Thanks and regards
Ritesh patel

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAPxUNF97HMKwLYmu3qEksLwY9--S5gM7T_AUP0NR4k3%2BVH%2Bgpw%40mail.gmail.com.

[prometheus-users] PromQL: multiple queries with dependent values

[marc koser]

Is it possible to have one side of a query limit the results of another 
part of the same query?

For example; I want an alert query for when the count of all instances in a 
clustered service are not equal to what the instance is reporting based on 
a label's value being equal in both sides of the query (in this case; there 
are many nodes that run the same service but are part of different 
clusters:)


*redis_cluster_known_nodes != scalar(count(up{service=~"redis-exporter"}))*

The shared label value would be something like, *group="cluster-a" *and 
should not evaluate metrics where *group="cluster-b"*

Thanks in advance!

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/12403656-6f2a-4383-ae72-8a073ab993c2n%40googlegroups.com.

[prometheus-users] Promql

[hamid daniali]

Hi everyone
i m looking for a formula that can make a query just for nights for example
from 12 pm to 12 am.
can anyone help me,
cheers.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAN%2B9LZGdk-4gXNpC%3D46e1ndRxPFR1kE01h67_FRtzyCUB2K2sA%40mail.gmail.com.

[prometheus-users] PromQL to find pods running on nodes based on architecture/instance type

[RonB]

Hello,
In the process of migrating services from amd64 to arm64 instances and was 
looking for help/guidance on a query to create a grafana dashboard.  What 
I'm trying to accomplish is displaying the number of pods running on a node 
by particular instance type/architecture.  The Karpenter "Pod Distribution 
by Instance Type" dashboard  is 
what I'm trying to eumulate.  Not using Karpenter, so can't use the promql 
expression they are using.

I did find this issue where someone joined kube_pod_labels with 
kube_node_labels.  I modified kube_state_metrics --metric-labels-allowlist 
to add the node.kubernetes.io/instance-type and tried the following promql, 
but received a many-to-many matching not allowed error:

kube_pod_info * on(pod,namespace) group_right(node) kube_pod_labels * 
on(node) group_left(label_kubernetes_io_instance_type) kube_node_labels

Any help or guidance with this would be greatly appreciated.

Thanks,
Ron

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/20a2b060-8d59-4dbc-ac02-2141127360f9n%40googlegroups.com.

[prometheus-users] PromQL Question/Help

[RB]

Hello,

I’m in the process of migrating some apps from amd64 to arm64 instances and 
was looking for help/guidance on a particular query to create a grafana 
dashboard.  What I’m trying to accomplish is displaying the number of pods 
running on a node by particular instance type/architecture.  The Karpenter 
“Pod Distribution by Instance Type” dashboard 
 is what I’m 
trying to emulate.  We are not using Karpenter, so can’t use the promql 
expression they are using.

I did find this 
 issue where 
someone tried to join kube_pod_labels with kube_node_labels.  I modified 
kube_state_metrics –metrics-allowlist to add the node.kubernetes.io 
instance_type and tried the following promql, but received a many-to-many 
matching not allowed Error:

*kube_pod_info * on(pod,namespace) group_right(node) kube_pod_labels * 
on(node) group_left(label_kubernetes_io_instance_type) kube_node_labels*

 Any help or guidance with this would be greatly appreciated.

 

Thanks,

Ron

 

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/a3373ce8-b971-405e-9061-a6694c042ff1n%40googlegroups.com.

Re: [prometheus-users] PromQL formatter?

[Facetoe]

I wrote a formatter for PromQL people might find useful - 
https://github.com/facetoe/promformat

On Monday, December 27, 2021 at 9:23:27 PM UTC+8 bitz...@gmail.com wrote:

> Is there any update of this tool?
> 在2020年9月15日星期二 UTC+8 04:59:12 写道：
>
>> There's a work-in-progress design doc for this:
>>
>>
>> https://docs.google.com/document/d/1nOBjpuCk4CsrOSm2ZjfVz2EL6gmA_CFGSbHCdY0Royg/edit
>>
>> On Mon, Sep 14, 2020 at 8:27 PM 'pin...@hioscar.com' via Prometheus 
>> Users  wrote:
>>
>>> What's a good formatter you guys recommend for formatting complex 
>>> queries?
>>>
>>> Ideally with line breaks and proper indentations after each ( or {.
>>>
>>> Thanks!
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Prometheus Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to prometheus-use...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/prometheus-users/8b6cd4e4-24be-4066-9ee0-9d38850bbef3n%40googlegroups.com
>>>  
>>> 
>>> .
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/83814c6c-8124-44f1-b3d8-fdb6523bd870n%40googlegroups.com.

Re: [prometheus-users] PromQL formatter?

[Facetoe]

I wrote a PromQL formatter people might find useful - 
https://github.com/facetoe/promformat

On Monday, December 27, 2021 at 9:23:27 PM UTC+8 bitz...@gmail.com wrote:

> Is there any update of this tool?
> 在2020年9月15日星期二 UTC+8 04:59:12 写道：
>
>> There's a work-in-progress design doc for this:
>>
>>
>> https://docs.google.com/document/d/1nOBjpuCk4CsrOSm2ZjfVz2EL6gmA_CFGSbHCdY0Royg/edit
>>
>> On Mon, Sep 14, 2020 at 8:27 PM 'pin...@hioscar.com' via Prometheus 
>> Users  wrote:
>>
>>> What's a good formatter you guys recommend for formatting complex 
>>> queries?
>>>
>>> Ideally with line breaks and proper indentations after each ( or {.
>>>
>>> Thanks!
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Prometheus Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to prometheus-use...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/prometheus-users/8b6cd4e4-24be-4066-9ee0-9d38850bbef3n%40googlegroups.com
>>>  
>>> 
>>> .
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/60c47374-be4d-409d-b544-18b514f6031bn%40googlegroups.com.

Re: [prometheus-users] PromQL formatter?

[Zhu Xiangbin]

Is there any update of this tool?
在2020年9月15日星期二 UTC+8 04:59:12 写道：

> There's a work-in-progress design doc for this:
>
>
> https://docs.google.com/document/d/1nOBjpuCk4CsrOSm2ZjfVz2EL6gmA_CFGSbHCdY0Royg/edit
>
> On Mon, Sep 14, 2020 at 8:27 PM 'pin...@hioscar.com' via Prometheus Users 
>  wrote:
>
>> What's a good formatter you guys recommend for formatting complex queries?
>>
>> Ideally with line breaks and proper indentations after each ( or {.
>>
>> Thanks!
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to prometheus-use...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/prometheus-users/8b6cd4e4-24be-4066-9ee0-9d38850bbef3n%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/2e7667ec-8d84-4a84-88a6-fd9b30d86625n%40googlegroups.com.

[prometheus-users] promql optimization

[ishu...@gmail.com]

Hi Team,

Any suggestion is very much appreciated.

I have an alert for breach of 95 percentile threshold on API requests. Now 
problem is, different API requests has different threshold, so I cannot 
hard code the alerts as I would have to add 10 alerts for 10 different APIs 
with their thresholds. 

The alerts would like something like this
es_exporter_percentiles_values_95_0{httppath="path1",app="app1"} > 10
es_exporter_percentiles_values_95_0{httppath="path2",app="app2"} > 5
es_exporter_percentiles_values_95_0{httppath="path3",app="app3"} > 3

Doesn't want to use or as again wanted to do this in a better smarter way. 
Tried using recording rules, but that would result in adding recording 
rules for 20 apps * 4 env * 5 http paths. 

Any better way, so that I have only one alert definition but can be applied 
against all apps, paths and envs. 

Thanks
Eswar


-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/c93fbde9-1633-4fda-9a1d-e603875e9e2an%40googlegroups.com.

[prometheus-users] Promql JOIN many-to-many matching

[Oltion Alimerko]

 

Hi all,

is it possible to use in promql the JOIN with many-to-many matching?I 
merged 2 metrics using group_left but it looks like the many-to-many 
matching is not supported because i get the error:

*"Error executing query: found duplicate series for the match group {} on 
the left hand-side of the operation. Many-to-many matching not allowed: 
matching labels must be unique on one side"*

On both sides of JOIN i have multitiple rows 

Metric 1

kube_deployment_labels{label_chart=~".*sdc-5.17.*",project!=""}


Result of metric 1 

kube_deployment_labels{deployment="sdc", instance="cfor-aks-dev", job=
"metrics-forwarder", label_app="sdc", label_app_kubernetes_io_managed_by=
"Helm", label_chart="sdc-5.17.2-HF01", label_heritage="Helm", label_release=
"sdc", namespace="dev-workloads", project="C4R"} 1 
kube_deployment_labels{deployment="sdc", instance="sop-aks-dev", job=
"metrics-forwarder", label_app="sdc", label_app_kubernetes_io_managed_by=
"Helm", label_chart="sdc-5.17.1-b03", label_heritage="Helm", label_release=
"sdc", namespace="dev-workloads", project="SOP"} 1 
kube_deployment_labels{deployment="sdc", instance="sop-aks-dev", job=
"metrics-forwarder", label_app="sdc", label_app_kubernetes_io_managed_by=
"Helm", label_chart="sdc-5.17.1-b03", label_heritage="Helm", label_release=
"sdc", namespace="test-workloads", project="SOP"} 1 
kube_deployment_labels{deployment="sdc", instance="stu-aks-dev", job=
"metrics-forwarder", label_app="sdc", label_app_kubernetes_io_managed_by=
"Helm", label_chart="sdc-5.17.2-HF04-b01", label_heritage="Helm", 
label_release="sdc", namespace="dev-workloads", project="STU"}

Metric2 

kube_node_info{container_runtime_version="containerd://1.4.4+azure", 
instance="cfor-aks-dev", job="metrics-forwarder", kernel_version=
"5.4.0-1051-azure", kubelet_version="v1.20.7", kubeproxy_version="v1.20.7", 
node="aks-default-13254112-vmss01", os_image="Ubuntu 18.04.5 LTS", 
pod_cidr="10.244.2.0/24", project="C4R", provider_id=
"azure:///subscriptions/693c9868-a960-4590-b23d-7220a5a8ba04/resourceGroups/mc_rg_aks-c4r-aks-dev_c4r-aks-dev_westeurope/providers/Microsoft.Compute/virtualMachineScaleSets/aks-default-13254112-vmss/virtualMachines/1"}
 
1 kube_node_info{container_runtime_version="containerd://1.4.4+azure", 
instance="cfor-aks-dev", job="metrics-forwarder", kernel_version=
"5.4.0-1051-azure", kubelet_version="v1.20.7", kubeproxy_version="v1.20.7", 
node="aks-default-13254112-vmss04", os_image="Ubuntu 18.04.5 LTS", 
pod_cidr="10.244.3.0/24", project="C4R", provider_id=
"azure:///subscriptions/693c9868-a960-4590-b23d-7220a5a8ba04/resourceGroups/mc_rg_aks-c4r-aks-dev_c4r-aks-dev_westeurope/providers/Microsoft.Compute/virtualMachineScaleSets/aks-default-13254112-vmss/virtualMachines/4"}
 
1 kube_node_info{container_runtime_version="containerd://1.4.4+azure", 
instance="stu-aks-dev", job="metrics-forwarder", kernel_version=
"5.4.0-1047-azure", kubelet_version="v1.20.7", kubeproxy_version="v1.20.7", 
node="aks-default-36930916-vmss02", os_image="Ubuntu 18.04.5 LTS", 
pod_cidr="10.244.3.0/24", project="STU", provider_id=
"azure:///subscriptions/3fb69224-6feb-4c6c-9f55-0b233b82d4a2/resourceGroups/mc_rg_aks-stu-aks-dev_stu-aks-dev_westeurope/providers/Microsoft.Compute/virtualMachineScaleSets/aks-default-36930916-vmss/virtualMachines/2"
}

Now i want all fields from metric1 and the label “kubelet_version” from 2nd 
metric,let say i want to join kubelet_version to the 1st metric result. 

I have wrote this metric: 

kube_deployment_labels{label_chart=~".*sdc-5.17.*",project!=""} * on () 
group_right (kubelet_version)kube_node_info{}


Since both metrics give me more then 1 timeseries data it looks like that i 
want to achieve a many-to-many match.Would be this possible? 


-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/9585b6b9-2fff-46f5-b369-40f8d61b64c1n%40googlegroups.com.

Re: [prometheus-users] PromQL to identify targets which are failing continuously in last 5 minutes

[Ben Kochie]

avg_over_time(probe_success[5m]) == 0

On Fri, Sep 24, 2021 at 4:42 AM Narendra Gudipudi 
wrote:

> Hi,
>
> I am using blackbox exporter to probe my targets. I am trying to write a
> PromQL to identify and alert every 5 minutes with details of targets which
> are continuously failing in the last 5 minutes. Any help here is greatly
> appreciated.
>
> I run my probers every 1 minute;
> *9:00 AM*
> probe_success {url="https:/url1"} = 1
> probe_success {url="https:/url2"} = 0
> probe_success {url="https:/url3"} = 0
> probe_success {url="https:/url4"} = 1
>
> *9:01 AM*
> probe_success {url="https:/url1"} = 1
> probe_success {url="https:/url2"} = 0
> probe_success {url="https:/url3"} = 0
> probe_success {url="https:/url4"} = 0
>
> *9:02 AM*
> probe_success {url="https:/url1"} = 1
> probe_success {url="https:/url2"} = 0
> probe_success {url="https:/url3"} = 0
> probe_success {url="https:/url4"} = 1
>
> *9:03 AM*
> probe_success {url="https:/url1"} = 1
> probe_success {url="https:/url2"} = 0
> probe_success {url="https:/url3"} = 0
> probe_success {url="https:/url4"} = 1
>
> *9:04 AM*
> probe_success {url="https:/url1"} = 1
> probe_success {url="https:/url2"} = 0
> probe_success {url="https:/url3"} = 0
> probe_success {url="https:/url4"} = 0
>
> *9:05 AM*
> probe_success {url="https:/url1"} = 1
> probe_success {url="*https:/url2*"} = 0
> probe_success {url="*https:/url3*"} = 0
> probe_success {url="https:/url4"} = 1
>
> I am looking for some help with query which gives me these two URLs -
>
> *https:/url2**https:/url3*
> *as these were continuously failing in the last 5 mins.*
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/2eebbd5f-08df-4694-b589-ab5cb2c7031en%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CABbyFmrc7%2B-x-%3DHxpvFFDQpqA0Cc4Pu094cFPftyCttW0F_kWA%40mail.gmail.com.

[prometheus-users] PromQL to identify targets which are failing continuously in last 5 minutes

[Narendra Gudipudi]

Hi,

I am using blackbox exporter to probe my targets. I am trying to write a 
PromQL to identify and alert every 5 minutes with details of targets which 
are continuously failing in the last 5 minutes. Any help here is greatly 
appreciated.

I run my probers every 1 minute;
*9:00 AM*
probe_success {url="https:/url1"} = 1
probe_success {url="https:/url2"} = 0
probe_success {url="https:/url3"} = 0
probe_success {url="https:/url4"} = 1

*9:01 AM*
probe_success {url="https:/url1"} = 1
probe_success {url="https:/url2"} = 0
probe_success {url="https:/url3"} = 0
probe_success {url="https:/url4"} = 0

*9:02 AM*
probe_success {url="https:/url1"} = 1
probe_success {url="https:/url2"} = 0
probe_success {url="https:/url3"} = 0
probe_success {url="https:/url4"} = 1

*9:03 AM*
probe_success {url="https:/url1"} = 1
probe_success {url="https:/url2"} = 0
probe_success {url="https:/url3"} = 0
probe_success {url="https:/url4"} = 1

*9:04 AM*
probe_success {url="https:/url1"} = 1
probe_success {url="https:/url2"} = 0
probe_success {url="https:/url3"} = 0
probe_success {url="https:/url4"} = 0

*9:05 AM*
probe_success {url="https:/url1"} = 1
probe_success {url="*https:/url2*"} = 0
probe_success {url="*https:/url3*"} = 0
probe_success {url="https:/url4"} = 1

I am looking for some help with query which gives me these two URLs -

*https:/url2**https:/url3*
*as these were continuously failing in the last 5 mins.*

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/2eebbd5f-08df-4694-b589-ab5cb2c7031en%40googlegroups.com.

[prometheus-users] Promql query

[Kshitija Patil]

Hi all,
I wanted to ask a question , I am trying to perform cpu_cores and 
memory_bytes query on the same graph for a particular namespace.How can i 
write a query for this .I am new to promql programming.Would appreciate it 
if anyone could help.
Thanks 

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/ae97e5b4-ddd2-4c70-9378-32e55c3d1b3dn%40googlegroups.com.

Re: [prometheus-users] PromQL formatter?

[Shuxiong Xu]

plz, Is there any update of this tools?

在2021年3月13日星期六 UTC+8 上午1:49:02 写道：

> Anyone found a good formatter yet? Thanks
>
> On Tuesday, September 15, 2020 at 1:24:07 AM UTC-4 pin...@hioscar.com 
> wrote:
>
>> Thanks. Before that's shipped, I'm hoping I can find a formatter that can 
>> do similar things, i.e. as simple as "adding line breaks and proper 
>> indentations after each ( or {."
>>
>> On Monday, September 14, 2020 at 4:59:12 PM UTC-4 sup...@gmail.com wrote:
>>
>>> There's a work-in-progress design doc for this:
>>>
>>>
>>> https://docs.google.com/document/d/1nOBjpuCk4CsrOSm2ZjfVz2EL6gmA_CFGSbHCdY0Royg/edit
>>>
>>> On Mon, Sep 14, 2020 at 8:27 PM 'pin...@hioscar.com' via Prometheus 
>>> Users  wrote:
>>>
 What's a good formatter you guys recommend for formatting complex 
 queries?

 Ideally with line breaks and proper indentations after each ( or {.

 Thanks!

 -- 
 You received this message because you are subscribed to the Google 
 Groups "Prometheus Users" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to prometheus-use...@googlegroups.com.
 To view this discussion on the web visit 
 https://groups.google.com/d/msgid/prometheus-users/8b6cd4e4-24be-4066-9ee0-9d38850bbef3n%40googlegroups.com
  
 
 .

>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/ad9d73b6-ff10-4264-9c94-25c97f03f52dn%40googlegroups.com.

[prometheus-users] PromQL query to find metric for which label values are different

[grze...@gmail.com]

Hi all,

let's say I have metric

   - some_metric{secret="aaa", some="ccc"}
   - other_metric{secret="aaa", some="ccc"}

and I want to have a query which will tell me that for those metrics, for 
`secret="aaa"` values for `some` label are different.

To give some context, I have some k8s metrics in prom which exposes labels 
on kubernetes objects, and I want to detect case when labels ("some") for 
two different object matched by `secret` labels is different.


Any advice will be appreciated,


Regards,


Grzegorz

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/714314b5-1e8e-48d6-a664-6a3a505fa236n%40googlegroups.com.

Re: [prometheus-users] PromQL Regex and the left bracket [ character

[Patrick Mackey]

Yes, that works. Thanks, Matthias!

On Thursday, 22 April 2021 at 08:13:25 UTC matt...@prometheus.io wrote:

> Try using [[] (match any of the following characters: [): 
> https://play.golang.org/p/ynvSW3lIHDY
>
> /MR
>
> On Thu, Apr 22, 2021 at 3:02 AM Patrick Mackey <
> patrick...@deniedaccess.org> wrote:
>
>> Hi, all.
>>
>> I'm trying to match the left bracket character '[' with a regex and seem 
>> to be hitting some issues. I've tried with the octal sequence \133 and the 
>> hex sequence \x5B but both return either false positive matches or the 
>> following error:
>>
>> Error executing query: invalid parameter "query": 1:19: parse error: 
>> error parsing regexp: missing closing ]: `[.*)$`
>>
>> I'm running 2.26.0 from Docker. 
>>
>> My query is variations of:
>> ifOperStatus{ifAlias=~"\133.*"}
>>
>> Matching the right bracket ']' with \135 and \x5D works fine.
>>
>> Any ideas?
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to prometheus-use...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/prometheus-users/083e9924-483c-4506-aba6-f35be725798an%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/031a8c61-b00f-4e82-9081-d36cac545b46n%40googlegroups.com.

Re: [prometheus-users] PromQL Regex and the left bracket [ character

[Matthias Rampke]

Try using [[] (match any of the following characters: [):
https://play.golang.org/p/ynvSW3lIHDY

/MR

On Thu, Apr 22, 2021 at 3:02 AM Patrick Mackey <
patrick.mac...@deniedaccess.org> wrote:

> Hi, all.
>
> I'm trying to match the left bracket character '[' with a regex and seem
> to be hitting some issues. I've tried with the octal sequence \133 and the
> hex sequence \x5B but both return either false positive matches or the
> following error:
>
> Error executing query: invalid parameter "query": 1:19: parse error: error
> parsing regexp: missing closing ]: `[.*)$`
>
> I'm running 2.26.0 from Docker.
>
> My query is variations of:
> ifOperStatus{ifAlias=~"\133.*"}
>
> Matching the right bracket ']' with \135 and \x5D works fine.
>
> Any ideas?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/083e9924-483c-4506-aba6-f35be725798an%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAMV%3D_ganJQjaJjP%3DHUOLFC7j3PoCFXaVoi_rBwVOM6v8q1ZWdw%40mail.gmail.com.

[prometheus-users] PromQL Regex and the left bracket [ character

[Patrick Mackey]

Hi, all.

I'm trying to match the left bracket character '[' with a regex and seem to 
be hitting some issues. I've tried with the octal sequence \133 and the hex 
sequence \x5B but both return either false positive matches or the 
following error:

Error executing query: invalid parameter "query": 1:19: parse error: error 
parsing regexp: missing closing ]: `[.*)$`

I'm running 2.26.0 from Docker. 

My query is variations of:
ifOperStatus{ifAlias=~"\133.*"}

Matching the right bracket ']' with \135 and \x5D works fine.

Any ideas?

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/083e9924-483c-4506-aba6-f35be725798an%40googlegroups.com.

Re: [prometheus-users] PromQL - Absenth - labels

[Matt Doughty]

Absent doesn't work that way. It just tells you that there are no
timeseries that match. As a result, when it fires there are no labels to
give back to you.

If you make a broad regex, the alert won't fire unless there are no matches
at all. For that reason, you will need a discrete alert for every process
you want to monitor.

--Matt



On Thu, Apr 15, 2021 at 10:52 AM fiala...@gmail.com 
wrote:

> I wanted to make this alert with lots of processes.
>
> Eg.: absent(process_cpu_usage{command=~"/usr/sbin/rsyslogd.*") or
> absent(process_cpu_usage{command=~"/usr/sbin/mysql") or absent()
> On Thursday, April 15, 2021 at 4:06:28 PM UTC+2 mdou...@gmail.com wrote:
>
>> Absent doesn't return labels because there aren't any. Add the labels you
>> need for the alert identification in the labels section.
>>
>> --Matt
>>
>> On Thu, Apr 15, 2021, 07:50 fiala...@gmail.com 
>> wrote:
>>
>>> Hi,
>>>
>>> I have cpu usage exporter with these metrics for every running processes.
>>> process_cpu_usage{command="/usr/bin/rsyslog -n"} 0.12
>>>
>>> I would like to use this exporter for monitoring if process is running.
>>>
>>> I've tried to use absent -  but with regexp match it doesn't return
>>> "command" label. And I need it for alert description
>>> Eg:
>>> absent(process_cpu_usage{command=~"/usr/sbin/rsyslogd.*", instance="
>>> my-server.com"})
>>>
>>> Is there a way how to add label to vector(0)?
>>>
>>> Or do you know any other way, how to solve it?
>>>
>>> Thank you.
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Prometheus Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to prometheus-use...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/prometheus-users/42c7e7dc-e32b-42f8-b7e9-a19d2ec0f485n%40googlegroups.com
>>> 
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/08bd910d-d8ad-47af-a273-b0b6a5f8260dn%40googlegroups.com
> 
> .
>


-- 
--Matt

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAGyBzcz%3DxJ_WAOWwoQw6tNzUMY80jQckUUXNP7zQ7OHnm%3Dxfgw%40mail.gmail.com.

Re: [prometheus-users] PromQL - Absenth - labels

[fiala...@gmail.com]

I wanted to make this alert with lots of processes.

Eg.: absent(process_cpu_usage{command=~"/usr/sbin/rsyslogd.*") or 
absent(process_cpu_usage{command=~"/usr/sbin/mysql") or absent()
On Thursday, April 15, 2021 at 4:06:28 PM UTC+2 mdou...@gmail.com wrote:

> Absent doesn't return labels because there aren't any. Add the labels you 
> need for the alert identification in the labels section.
>
> --Matt
>
> On Thu, Apr 15, 2021, 07:50 fiala...@gmail.com  wrote:
>
>> Hi,
>>
>> I have cpu usage exporter with these metrics for every running processes.
>> process_cpu_usage{command="/usr/bin/rsyslog -n"} 0.12
>>
>> I would like to use this exporter for monitoring if process is running.
>>
>> I've tried to use absent -  but with regexp match it doesn't return 
>> "command" label. And I need it for alert description
>> Eg: 
>> absent(process_cpu_usage{command=~"/usr/sbin/rsyslogd.*", instance="
>> my-server.com"})
>>
>> Is there a way how to add label to vector(0)?
>>
>> Or do you know any other way, how to solve it?
>>
>> Thank you.
>>
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to prometheus-use...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/prometheus-users/42c7e7dc-e32b-42f8-b7e9-a19d2ec0f485n%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/08bd910d-d8ad-47af-a273-b0b6a5f8260dn%40googlegroups.com.

Re: [prometheus-users] PromQL - Absenth - labels

[Matt Doughty]

Absent doesn't return labels because there aren't any. Add the labels you
need for the alert identification in the labels section.

--Matt

On Thu, Apr 15, 2021, 07:50 fiala...@gmail.com 
wrote:

> Hi,
>
> I have cpu usage exporter with these metrics for every running processes.
> process_cpu_usage{command="/usr/bin/rsyslog -n"} 0.12
>
> I would like to use this exporter for monitoring if process is running.
>
> I've tried to use absent -  but with regexp match it doesn't return
> "command" label. And I need it for alert description
> Eg:
> absent(process_cpu_usage{command=~"/usr/sbin/rsyslogd.*", instance="
> my-server.com"})
>
> Is there a way how to add label to vector(0)?
>
> Or do you know any other way, how to solve it?
>
> Thank you.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/42c7e7dc-e32b-42f8-b7e9-a19d2ec0f485n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAGyBzcyEsX2bm_AQP3yTtq5YyACcxx5P52ijoNE4Eor%2Bb4O%2BnQ%40mail.gmail.com.

[prometheus-users] PromQL - Absenth - labels

[fiala...@gmail.com]

Hi,

I have cpu usage exporter with these metrics for every running processes.
process_cpu_usage{command="/usr/bin/rsyslog -n"} 0.12

I would like to use this exporter for monitoring if process is running.

I've tried to use absent -  but with regexp match it doesn't return 
"command" label. And I need it for alert description
Eg: 
absent(process_cpu_usage{command=~"/usr/sbin/rsyslogd.*", 
instance="my-server.com"})

Is there a way how to add label to vector(0)?

Or do you know any other way, how to solve it?

Thank you.


-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/42c7e7dc-e32b-42f8-b7e9-a19d2ec0f485n%40googlegroups.com.

Re: [prometheus-users] PromQL query to count how many series have {label=XYZ}

[Julien Pivotto]

On 24 Mar 19:15, John Dexter wrote:
> I would like to count how many time-series have a given label value for a
> given metric.
> 
> I am looking at something like:
> group by (job) (my_updates_total{job="Job123"})

count by (job) (my_updates_total{job="Job123"})

> 
> It seems to work but feels a bit odd, is it right?
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/prometheus-users/CAGJRanjBKf5TQNhqdV1_63%3DrfRRjoKO_XvkO-5UnErr6abQnCg%40mail.gmail.com.

-- 
Julien Pivotto
@roidelapluie

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/20210325001059.GA452144%40oxygen.

[prometheus-users] PromQL query to count how many series have {label=XYZ}

[John Dexter]

I would like to count how many time-series have a given label value for a
given metric.

I am looking at something like:
group by (job) (my_updates_total{job="Job123"})

It seems to work but feels a bit odd, is it right?

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAGJRanjBKf5TQNhqdV1_63%3DrfRRjoKO_XvkO-5UnErr6abQnCg%40mail.gmail.com.

Re: [prometheus-users] PromQL query decode

[Unni Sathyarajan]

Thanks, David, for the clarification!!

On Tue, Mar 16, 2021 at 2:08 PM David Leadbeater  wrote:

> It’s because the annotation uses $value, so the annotation can report a
> more useful number to the user, rather than rate per second.
>
> (Although using increase() rather than rate() may be better for this case,
> as it avoids the need to keep the window in sync with the maths.)
>
> David
>
> On Tue, 16 Mar 2021 at 09:31, Unni Sathyarajan 
> wrote:
>
>> Hi All,
>>
>> Could someone please explain this PromQL query?
>>
>> rate(kube_pod_container_status_restarts_total{job="kube-state-metrics"}[15m])
>> * 60 * 5 > 0
>>
>> Cannot get my head around why we multiply ** 60 * 5 * in this metric.
>>
>> Would the following query be fine to identify and alert any crash looping
>> containers?
>>
>> rate(kube_pod_container_status_restarts_total{job="kube-state-metrics"}[15m])
>> > 0
>>
>> This is in reference to the following alert
>> https://monitoring.mixins.dev/kubernetes/
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to prometheus-users+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-users/ddd22d69-2213-4647-87d8-cadd7f7f7211n%40googlegroups.com
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAEtPgHNjjs%2Bn_0h1Of_9o1xyoj_NdDk6r7fSMY%2BQ3nueE%2B1oDA%40mail.gmail.com.

Re: [prometheus-users] PromQL query decode

[David Leadbeater]

It’s because the annotation uses $value, so the annotation can report a
more useful number to the user, rather than rate per second.

(Although using increase() rather than rate() may be better for this case,
as it avoids the need to keep the window in sync with the maths.)

David

On Tue, 16 Mar 2021 at 09:31, Unni Sathyarajan 
wrote:

> Hi All,
>
> Could someone please explain this PromQL query?
>
> rate(kube_pod_container_status_restarts_total{job="kube-state-metrics"}[15m])
> * 60 * 5 > 0
>
> Cannot get my head around why we multiply ** 60 * 5 * in this metric.
>
> Would the following query be fine to identify and alert any crash looping
> containers?
>
> rate(kube_pod_container_status_restarts_total{job="kube-state-metrics"}[15m])
> > 0
>
> This is in reference to the following alert
> https://monitoring.mixins.dev/kubernetes/
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/ddd22d69-2213-4647-87d8-cadd7f7f7211n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAP9KPhA7xW4Sdo0n5jhJOMuXak5iJJvkLnxAoWkQ9%2BYQ3n0bJg%40mail.gmail.com.

Re: [prometheus-users] PromQL formatter?

['ping...@hioscar.com' via Prometheus Users]

Anyone found a good formatter yet? Thanks

On Tuesday, September 15, 2020 at 1:24:07 AM UTC-4 ping...@hioscar.com 
wrote:

> Thanks. Before that's shipped, I'm hoping I can find a formatter that can 
> do similar things, i.e. as simple as "adding line breaks and proper 
> indentations after each ( or {."
>
> On Monday, September 14, 2020 at 4:59:12 PM UTC-4 sup...@gmail.com wrote:
>
>> There's a work-in-progress design doc for this:
>>
>>
>> https://docs.google.com/document/d/1nOBjpuCk4CsrOSm2ZjfVz2EL6gmA_CFGSbHCdY0Royg/edit
>>
>> On Mon, Sep 14, 2020 at 8:27 PM 'pin...@hioscar.com' via Prometheus 
>> Users  wrote:
>>
>>> What's a good formatter you guys recommend for formatting complex 
>>> queries?
>>>
>>> Ideally with line breaks and proper indentations after each ( or {.
>>>
>>> Thanks!
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Prometheus Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to prometheus-use...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/prometheus-users/8b6cd4e4-24be-4066-9ee0-9d38850bbef3n%40googlegroups.com
>>>  
>>> 
>>> .
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/8d4e4f15-0741-4798-9161-f199723c68f5n%40googlegroups.com.

Re: [prometheus-users] promQL duration window

['M S' via Prometheus Users]

I believe [1h] in that context means use the data series of 1h intervals.

I wanted to specify the amount of time in the graph.

I did manage to do it — next to the data source, you can select query options, 
I set relative time to 7 days to get a week of data. 

> On 9 Mar 2021, at 10:24, Charls P John  wrote:
> 
> Hi MS,
> 
> Did you try metric_name{label1="value1"}[1h] ?
> 
> On Tue, Mar 9, 2021, 14:16 'M S' via Prometheus Users 
>  > wrote:
> How do I specify that I want see the last 1 hour of data only, or I want to 
> see 2 days of data?
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to prometheus-users+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/prometheus-users/D462A65B-689E-406E-877A-C5F3CA6BDD54%40yahoo.com
>  
> .
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to prometheus-users+unsubscr...@googlegroups.com 
> .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/prometheus-users/CAJ7d%2BpQ1pTjzQ1y_9qLm-%3D20axYyn%2B0HjVvGB2eoWikRXjeZpg%40mail.gmail.com
>  
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/E6276B78-92AD-4BB2-B48A-A17FF7B1EADB%40yahoo.com.

Re: [prometheus-users] promQL duration window

[Charls P John]

Hi MS,

Did you try *metric_name{label1="value1"}[1h]* ?

On Tue, Mar 9, 2021, 14:16 'M S' via Prometheus Users <
prometheus-users@googlegroups.com> wrote:

> How do I specify that I want see the last 1 hour of data only, or I want
> to see 2 days of data?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/D462A65B-689E-406E-877A-C5F3CA6BDD54%40yahoo.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAJ7d%2BpQ1pTjzQ1y_9qLm-%3D20axYyn%2B0HjVvGB2eoWikRXjeZpg%40mail.gmail.com.

[prometheus-users] promQL duration window

['M S' via Prometheus Users]

How do I specify that I want see the last 1 hour of data only, or I want to see 
2 days of data?

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/D462A65B-689E-406E-877A-C5F3CA6BDD54%40yahoo.com.

[prometheus-users] PromQL Query Using Hour Function

[Chad Thielen]

Hello,

I'm having some trouble writing a query for an alerting rule that we only 
want to run at certain times of the day. This is what I original had, which 
works, but not in the way we now need:

> count(up{job="calculator"} == 0) and count(hour(vector(time())) >= 1 and 
hour(vector(time())) < 13)

Now our problem is that we need the instance label from the up metric to 
persist to the alert so we can include it in the pager duty incident, 
otherwise our engineers don't know what instance is down without checking 
prometheus first. This is where I'm having problems, using count strips all 
those labels and just basically gives a true or false of if the alert is 
firing.

Any suggestions?

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/ea198656-1b5d-4eac-b8b1-c6e962b91fa4n%40googlegroups.com.

Re: [prometheus-users] PromQL prolong alert 30 minutes

[Matthias Rampke]

Try `min_over_time` on a subquery:

min_over_time(( some_traffic_metric / some_traffic_metrics offset
15m)[30m:]) < 0.8

/MR

On Thu, Jan 28, 2021 at 7:37 AM fiala...@gmail.com 
wrote:

> Hi,
>
> I have a simple rule for monitoring traffic drop (by 20% in last 15
> minutes):
>
> some_traffic_metric < (some_traffic_metric offset 15m * 0.8)
>
> I need this alert to be active for at least 30 minutes.
>
> Please help, how to rewrite the query.
>
> Thank you.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/1def3a6c-4804-4782-af0e-0109b05011dfn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAMV%3D_ga9c9z5dwXPyH6%3DNDUivwwUuYJH8ieWS090S_zRLNA7xQ%40mail.gmail.com.

[prometheus-users] PromQL prolong alert 30 minutes

[fiala...@gmail.com]

Hi,

I have a simple rule for monitoring traffic drop (by 20% in last 15 
minutes):

some_traffic_metric < (some_traffic_metric offset 15m * 0.8)

I need this alert to be active for at least 30 minutes. 

Please help, how to rewrite the query.

Thank you.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/1def3a6c-4804-4782-af0e-0109b05011dfn%40googlegroups.com.

Re: [prometheus-users] PromQL: many-to-many matching not allowed: matching labels must be unique on one side

[Ramprasath A]

 Hi Julius,

It is an honor and my pleasure to receive a reply from you. 

Thanks a lot for pointing me in the right direction.

The query worked flawlessly when I added the label: clusterName.

The modified query looks like this:

count by (cluster, node) (sum by (node, cpu) (
node_cpu_seconds_total{job="node-exporter",clusterName="cluster1"}
  * on (namespace, pod) group_left(node)
node_namespace_pod:kube_pod_info:{clusterName="cluster1"}
  ))

Sincerely,
Ramprasath A

On Tuesday, January 26, 2021 at 5:08:09 PM UTC+5:30 juliu...@promlabs.com 
wrote:

> Hi Ramprasath,
>
> When you do a N-to-1 (or many-to-one) match, you need to ensure that your 
> matching labels (in your case, "namespace" and "pod") don't produce 
> multiple matches on the "many" (right) side for any entry on the "one" 
> (left) side. The error message lists two series from the right side which 
> have identical namespace + pod labels, but differ on at least the "replica" 
> label ("thanos-receive-prod-0" vs. "thanos-receive-poc-0"). Maybe you only 
> wanted to select series from one Thanos receive replica on the right side 
> and that would fix the issue? Or you could aggregate away the extra label 
> somehow before applying the binary operator.
>
> Regards,
> Julius
>
> On Tue, Jan 26, 2021 at 5:49 AM Ramprasath A  
> wrote:
>
>> Hi,
>>
>> I have an expression of a recording rule named *node:node_num_cpu:sum *
>> and I am planning to add this rule to Thanos-Rule component but it fails 
>> in Thanos-Query itself with the error message: *many-to-many matching 
>> not allowed: matching labels must be unique on one side* (attached 
>> screenshot here with too. Please find the attachment.)
>>
>> Here is the query:
>> count by (cluster, node) (sum by (node, cpu) (
>> node_cpu_seconds_total{job="node-exporter"}
>> * on (namespace, pod) group_left(node)
>> node_namespace_pod:kube_pod_info:
>> )) 
>>
>> Similarly, there are around about 50 other recording and alerting rules 
>> which are affected.
>>
>> Thank you for your time and consideration.
>>
>> Sincerely,
>> Ramprasath A
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to prometheus-use...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/prometheus-users/c100c985-58ec-40e6-b15b-10d6d704b1bfn%40googlegroups.com
>>  
>> 
>> .
>>
>
>
> -- 
> Julius Volz
> PromLabs - promlabs.com
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/1aaf4728-5ed8-4f01-ad3a-285cdf08ec0fn%40googlegroups.com.

Re: [prometheus-users] PromQL: many-to-many matching not allowed: matching labels must be unique on one side

[Ramprasath A]

Thank you for the clarification. I will check this out and get back to you.

On Tue, Jan 26, 2021, 5:08 PM Julius Volz  wrote:

> Hi Ramprasath,
>
> When you do a N-to-1 (or many-to-one) match, you need to ensure that your
> matching labels (in your case, "namespace" and "pod") don't produce
> multiple matches on the "many" (right) side for any entry on the "one"
> (left) side. The error message lists two series from the right side which
> have identical namespace + pod labels, but differ on at least the "replica"
> label ("thanos-receive-prod-0" vs. "thanos-receive-poc-0"). Maybe you only
> wanted to select series from one Thanos receive replica on the right side
> and that would fix the issue? Or you could aggregate away the extra label
> somehow before applying the binary operator.
>
> Regards,
> Julius
>
> On Tue, Jan 26, 2021 at 5:49 AM Ramprasath A 
> wrote:
>
>> Hi,
>>
>> I have an expression of a recording rule named *node:node_num_cpu:sum *
>> and I am planning to add this rule to Thanos-Rule component but it fails
>> in Thanos-Query itself with the error message: *many-to-many matching
>> not allowed: matching labels must be unique on one side* (attached
>> screenshot here with too. Please find the attachment.)
>>
>> Here is the query:
>> count by (cluster, node) (sum by (node, cpu) (
>> node_cpu_seconds_total{job="node-exporter"}
>> * on (namespace, pod) group_left(node)
>> node_namespace_pod:kube_pod_info:
>> ))
>>
>> Similarly, there are around about 50 other recording and alerting rules
>> which are affected.
>>
>> Thank you for your time and consideration.
>>
>> Sincerely,
>> Ramprasath A
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to prometheus-users+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-users/c100c985-58ec-40e6-b15b-10d6d704b1bfn%40googlegroups.com
>> 
>> .
>>
>
>
> --
> Julius Volz
> PromLabs - promlabs.com
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAKr3sbWcCwd0FB%3DmnFQjuLz5OAPpNibdTmXHb_GrGVviwhJKVA%40mail.gmail.com.

Re: [prometheus-users] PromQL: many-to-many matching not allowed: matching labels must be unique on one side

[Julius Volz]

Hi Ramprasath,

When you do a N-to-1 (or many-to-one) match, you need to ensure that your
matching labels (in your case, "namespace" and "pod") don't produce
multiple matches on the "many" (right) side for any entry on the "one"
(left) side. The error message lists two series from the right side which
have identical namespace + pod labels, but differ on at least the "replica"
label ("thanos-receive-prod-0" vs. "thanos-receive-poc-0"). Maybe you only
wanted to select series from one Thanos receive replica on the right side
and that would fix the issue? Or you could aggregate away the extra label
somehow before applying the binary operator.

Regards,
Julius

On Tue, Jan 26, 2021 at 5:49 AM Ramprasath A 
wrote:

> Hi,
>
> I have an expression of a recording rule named *node:node_num_cpu:sum *and
>  I am planning to add this rule to Thanos-Rule component but it fails in
> Thanos-Query itself with the error message: *many-to-many matching not
> allowed: matching labels must be unique on one side* (attached screenshot
> here with too. Please find the attachment.)
>
> Here is the query:
> count by (cluster, node) (sum by (node, cpu) (
> node_cpu_seconds_total{job="node-exporter"}
> * on (namespace, pod) group_left(node)
> node_namespace_pod:kube_pod_info:
> ))
>
> Similarly, there are around about 50 other recording and alerting rules
> which are affected.
>
> Thank you for your time and consideration.
>
> Sincerely,
> Ramprasath A
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/c100c985-58ec-40e6-b15b-10d6d704b1bfn%40googlegroups.com
> 
> .
>


-- 
Julius Volz
PromLabs - promlabs.com

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAObpH5xNAS%3DzN_48X%3DNgux48HxvfgncuQBCawajzwvZ0Kvfi8g%40mail.gmail.com.

Re: [prometheus-users] PromQL - vector cannot contain metrics with the same labelset

[Dan S]

Awesome Julian, thanks!  Works great. 

On Monday, January 25, 2021 at 2:40:06 PM UTC+2 juliu...@promlabs.com wrote:

> Hi,
>
> The problem you're running into is that a binary operator (division, in 
> your case) removes the metric name (because the sample value no longer has 
> that same metric name meaning after the transformation), but the metric 
> name is the only distinguishing label in your case on these series. You 
> could work around that by first copying the metric name to a non-special 
> label, like "metric":
>
> label_join(count by (__name__) ({job="kubernetes-pods", app="scylla"}), 
> "metric", "", "__name__") / 70
>
> Regards,
> Julius
>
> On Sun, Jan 24, 2021 at 10:37 PM Dan S  wrote:
>
>> Hi,
>>
>> Not sure what I'm doing wrong when trying to divide
>>
>> *count by (__name__) ({job="kubernetes-pods", app="scylla"})*
>>
>> scrape_series_added{} 70
>> scrape_samples_scraped{}  70
>> scylla_node_operation_mode{}  70
>> scrape_samples_post_metric_relabeling{} 70
>> scrape_duration_seconds{} 70
>> up{}  70
>>
>> When I try to divide by 70 (to find number of time series per pod), I get 
>> an error:
>>
>> *count by (__name__) ({job="kubernetes-pods", app="scylla"}) / 70*
>> *Error executing query: vector cannot contain metrics with the same 
>> labelset*
>>
>> how can I get a basic divide working? so that I end up with something 
>> like:
>>
>> scrape_series_added{} 1
>> scrape_samples_scraped{}  1
>> scylla_node_operation_mode{}  1
>> scrape_samples_post_metric_relabeling{} 1
>> scrape_duration_seconds{} 1
>> up{}  1
>>
>> Basically I want to know exactly how many time series per metric each 
>> "app" is generating per pod
>>
>> Thanks
>> Dan
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to prometheus-use...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/prometheus-users/d3562084-7a0f-4012-82d2-218036b730a5n%40googlegroups.com
>>  
>> 
>> .
>>
>
>
> -- 
> Julius Volz
> PromLabs - promlabs.com
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/81f23a3a-2245-43a1-bbea-24b620942a1an%40googlegroups.com.

Re: [prometheus-users] PromQL - vector cannot contain metrics with the same labelset

[Julius Volz]

Hi,

The problem you're running into is that a binary operator (division, in
your case) removes the metric name (because the sample value no longer has
that same metric name meaning after the transformation), but the metric
name is the only distinguishing label in your case on these series. You
could work around that by first copying the metric name to a non-special
label, like "metric":

label_join(count by (__name__) ({job="kubernetes-pods", app="scylla"}),
"metric", "", "__name__") / 70

Regards,
Julius

On Sun, Jan 24, 2021 at 10:37 PM Dan S  wrote:

> Hi,
>
> Not sure what I'm doing wrong when trying to divide
>
> *count by (__name__) ({job="kubernetes-pods", app="scylla"})*
>
> scrape_series_added{} 70
> scrape_samples_scraped{}  70
> scylla_node_operation_mode{}  70
> scrape_samples_post_metric_relabeling{} 70
> scrape_duration_seconds{} 70
> up{}  70
>
> When I try to divide by 70 (to find number of time series per pod), I get
> an error:
>
> *count by (__name__) ({job="kubernetes-pods", app="scylla"}) / 70*
> *Error executing query: vector cannot contain metrics with the same
> labelset*
>
> how can I get a basic divide working? so that I end up with something like:
>
> scrape_series_added{} 1
> scrape_samples_scraped{}  1
> scylla_node_operation_mode{}  1
> scrape_samples_post_metric_relabeling{} 1
> scrape_duration_seconds{} 1
> up{}  1
>
> Basically I want to know exactly how many time series per metric each
> "app" is generating per pod
>
> Thanks
> Dan
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/d3562084-7a0f-4012-82d2-218036b730a5n%40googlegroups.com
> 
> .
>


-- 
Julius Volz
PromLabs - promlabs.com

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAObpH5yASQBDcuPiqa-cvaB1DdQ%2BTNaPJRzdv0MnWu%2BsGf5Bxg%40mail.gmail.com.

[prometheus-users] PromQL - vector cannot contain metrics with the same labelset

[Dan S]

Hi,

Not sure what I'm doing wrong when trying to divide

*count by (__name__) ({job="kubernetes-pods", app="scylla"})*

scrape_series_added{} 70
scrape_samples_scraped{}  70
scylla_node_operation_mode{}  70
scrape_samples_post_metric_relabeling{} 70
scrape_duration_seconds{} 70
up{}  70

When I try to divide by 70 (to find number of time series per pod), I get 
an error:

*count by (__name__) ({job="kubernetes-pods", app="scylla"}) / 70*
*Error executing query: vector cannot contain metrics with the same 
labelset*

how can I get a basic divide working? so that I end up with something like:

scrape_series_added{} 1
scrape_samples_scraped{}  1
scylla_node_operation_mode{}  1
scrape_samples_post_metric_relabeling{} 1
scrape_duration_seconds{} 1
up{}  1

Basically I want to know exactly how many time series per metric each "app" 
is generating per pod

Thanks
Dan

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/d3562084-7a0f-4012-82d2-218036b730a5n%40googlegroups.com.

Re: [prometheus-users] PromQL one-to-many matching with different label sets

[Julius Volz]

Hi Simon,

Aggregation and binary operators are quite different beasts and only
sometimes happen to be usable to reach the same query outcome...

The "on(foo, bar)" label matching modifier you provided tells PromQL to
match series from the left and right sides only when they have exactly the
same values for *both* of those labels. But in your case, the left series
has {foo="a", bar="c"} and the right series has only {foo="a"}, which is
equivalent to {foo="a", bar=""}, so a different (non-existent) value for
the bar label. Thus, those two series do not find each other as a match at
all and don't become part of the output. The "group_left()" modifier only
helps to allow one-to-many matching in situations where your matching
criteria end up matching each series from the left with multiple series
from the right, but it cannot create matches that weren't there in the
first place.

Regards,
Julius

On Thu, Jan 21, 2021 at 12:22 PM Simon Let  wrote:

> I have some metrics with different label sets, for example:
> ```
> label_metric{foo="a", bar="b"} 0
> label_metric{foo="a", bar="c"} 0
>
> my_metric{foo="a", bar="b"} 5
> my_metric{foo="a"} 3
> ```
>
> When I use the following aggregation:
> ```
> min by (foo, bar) (
>   my_metric
> )
> ```
> I get an expected result:
> ```
> my_metric{foo="a", bar="b"} 5
> my_metric{foo="a"} 3
> ```
> But when I instead use one-to-many operation:
> ```
> label_metric
> + on (foo, bar) group_left()
> my_metric
> ```
> I would expect a following result:
> ```
> {foo="a", bar="b"} 5
> {foo="a", bar="c"} 3
> ```
> as it would be consistent with the aggregation operation behavior.
>
> But instead, I get the following result:
> ```
> {foo="a", bar="b"} 5
> ```
>
> This behavior seems inconsistent with the aggregation operation to me.
> Is there a reason why `group_left()` silently drops part of the data
> instead of matching it with the right-hand side?
>
> It looks like one-to-many matching only works with the largest label set
> and drops the rest of the data. This is very confusing and dangerous imho.
>
> Am I missing something?
>
>
> Thanks in advance
> Simon Let
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/97e321fd-846f-4bf2-b380-ca72924af0f3n%40googlegroups.com
> 
> .
>


-- 
Julius Volz
PromLabs - promlabs.com

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAObpH5wLpJ85iPzYAnUP-s2fn6-%2BiAqzYivLcbMX0YKxg-A_qg%40mail.gmail.com.

[prometheus-users] Promql related query - probe_icmp_duration_seconds{phase =~ "resolve"} > 0 and probe_success == 0

[Siju Vasudevan]

Usecase : i want to trigger hostdown alert only if the dns resolve is 
success  and icmp check got failed. I was trying to create a promql query 
for this usecase but prometheus was not giving any result. Is it possible?

probe_icmp_duration_seconds{phase =~ "resolve"} > 0 and probe_success == 0  

Regards,
Siju

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/e9321d44-f867-4a4a-8415-ae1dedba8e93n%40googlegroups.com.

[prometheus-users] PromQL one-to-many matching with different label sets

[Simon Let]

I have some metrics with different label sets, for example:
```
label_metric{foo="a", bar="b"} 0
label_metric{foo="a", bar="c"} 0

my_metric{foo="a", bar="b"} 5
my_metric{foo="a"} 3
```

When I use the following aggregation:
```
min by (foo, bar) (
  my_metric
)
```
I get an expected result:
```
my_metric{foo="a", bar="b"} 5
my_metric{foo="a"} 3
```
But when I instead use one-to-many operation:
```
label_metric
+ on (foo, bar) group_left()
my_metric
```
I would expect a following result:
```
{foo="a", bar="b"} 5
{foo="a", bar="c"} 3
```
as it would be consistent with the aggregation operation behavior.

But instead, I get the following result:
```
{foo="a", bar="b"} 5
```

This behavior seems inconsistent with the aggregation operation to me. 
Is there a reason why `group_left()` silently drops part of the data 
instead of matching it with the right-hand side? 

It looks like one-to-many matching only works with the largest label set 
and drops the rest of the data. This is very confusing and dangerous imho.

Am I missing something? 


Thanks in advance
Simon Let

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/97e321fd-846f-4bf2-b380-ca72924af0f3n%40googlegroups.com.

Re: [prometheus-users] PromQL step parameter

[Julius Volz]

Hi Kiran,

On Fri, Jan 15, 2021 at 2:01 PM kiran  wrote:

> Hello all
>
> 1. In PromQL range query what step value is used if I don’t give step
> parameter?
>

The server returns an error if you use the API directly and don't set the
step parameter. However, when using the Prometheus web UI and don't set a
step manually, it sets a default step (
https://github.com/prometheus/prometheus/blob/e483cff61ff1b68e8925db1059393c4d36af9df5/web/ui/react-app/src/pages/graph/Panel.tsx#L114)
that depends on the graph range. It's basically the graph range divided by
250, so a 250-minute graph range would get a 1-minute step and a 500-minute
time range would get a 2-minute step.


> 2. Is there a way to determine for the given start and end dates, what is
> the step value I can give so it’s with the limit where query can handle?
>

No, at least not without first looking at the data more and determining how
many samples there would be under a given graph range. You need to somehow
keep the total number of returned points <=11k.

E.g I used a query for last 30 days and I gave 1 min as step and it threw a
> limitation error. So wondering if there is a way I could know what step
> value is appropriate for the given period.
>

Note that the 11k-point limit only applies to range queries. If your goal
is to extract raw data at the resolution that it was scraped over 30d, you
could run an instant query like "my_metric[30d]".

Regards,
Julius

-- 
Julius Volz
PromLabs - promlabs.com

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAObpH5ydfm00mBWoet%2BA4oLwQRfWpSbsOMoHibE6uu97T2qH%3DQ%40mail.gmail.com.

[prometheus-users] PromQL step parameter

[kiran]

Hello all

1. In PromQL range query what step value is used if I don’t give step
parameter?
2. Is there a way to determine for the given start and end dates, what is
the step value I can give so it’s with the limit where query can handle?
E.g I used a query for last 30 days and I gave 1 min as step and it threw a
limitation error. So wondering if there is a way I could know what step
value is appropriate for the given period.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAOnWYZVQssvfO%2BQ9jPmNnjHsLm1rRO0H%3D_7fnsugbdS1dzo%3D3Q%40mail.gmail.com.

Re: [prometheus-users] PROMQL deriv()

[Rodolphe Ghio]

Thanks a lot ! 
That was exactly what I was looking for !
Have a great day !

Le jeudi 7 janvier 2021 à 10:31:18 UTC-5, Julien Pivotto a écrit :

> On 07 Jan 06:56, Rodolphe Ghio wrote:
> > Hi everyone ! 
> > First of all I wish you a happy new year.
> > 
> > I would like to use the deriv() function but this function needs a range 
> > vector so my query should look like this: 
> > deriv([]). But I don't really understand how 
> can I 
> > choose an acurate time_range in square brackets.
> > Can someone help me to find out how can choose the right time_range ?
> > 
> > King regards,
> > GHIO Rodolphe
>
> Please look at
> https://www.robustperception.io/what-range-should-i-use-with-rate
>
> or
>
> https://grafana.com/blog/2020/09/28/new-in-grafana-7.2-__rate_interval-for-prometheus-rate-queries-that-just-work/
>
>
> > 
> > 
> > 
> > -- 
> > You received this message because you are subscribed to the Google 
> Groups "Prometheus Users" group.
> > To unsubscribe from this group and stop receiving emails from it, send 
> an email to prometheus-use...@googlegroups.com.
> > To view this discussion on the web visit 
> https://groups.google.com/d/msgid/prometheus-users/9f285f7f-3afc-4a5b-b2de-ffb8a2dafe7dn%40googlegroups.com
> .
>
>
> -- 
> Julien Pivotto
> @roidelapluie
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/6fcb0e76-22ab-4b65-b91a-c3e977257ccan%40googlegroups.com.

Re: [prometheus-users] PROMQL deriv()

[Julien Pivotto]

On 07 Jan 06:56, Rodolphe Ghio wrote:
> Hi everyone ! 
> First of all I wish you a happy new year.
> 
> I would like to use the deriv() function but this function needs a range 
> vector so my query should look like this: 
> deriv([]). But I don't really understand how can I 
> choose an acurate time_range in square brackets.
> Can someone help me to find out how can choose the right time_range ?
> 
> King regards,
> GHIO Rodolphe

Please look at
https://www.robustperception.io/what-range-should-i-use-with-rate

or
https://grafana.com/blog/2020/09/28/new-in-grafana-7.2-__rate_interval-for-prometheus-rate-queries-that-just-work/


> 
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/prometheus-users/9f285f7f-3afc-4a5b-b2de-ffb8a2dafe7dn%40googlegroups.com.


-- 
Julien Pivotto
@roidelapluie

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/20210107153112.GA660193%40oxygen.

[prometheus-users] PROMQL deriv()

[Rodolphe Ghio]

Hi everyone ! 
First of all I wish you a happy new year.

I would like to use the deriv() function but this function needs a range 
vector so my query should look like this: 
deriv([]). But I don't really understand how can I 
choose an acurate time_range in square brackets.
Can someone help me to find out how can choose the right time_range ?

King regards,
GHIO Rodolphe



-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/9f285f7f-3afc-4a5b-b2de-ffb8a2dafe7dn%40googlegroups.com.

Re: [prometheus-users] Promql is a bit complicated. Is there any way to reduce the difficulty of use？

[Allenzh li]

Just now, I have search some other monitoring systems and find sysdig which 
have provided a cookbook which how to monitoring kubernetes.

https://sysdig.com/blog/alerting-kubernetes/ 

This is very enlightening to me.


Kind regards, 

Allenzh li
在2020年12月9日星期三 UTC+8 下午5:46:08 写道：

> I agree with you. 
>
>
> PromQL is a power query language and the web page can’t easily represent 
> the power.
>
>
> It is also a good idea to provide examples or cookbooks.
>
>
> Kind regards,
> Allenzh li
>
>
> 在 2020年12月9日 16:50，Stuart Clark 写道：
>
> On 09/12/2020 08:34, Allenzh li wrote:
> > The key problem is that ordinary users need to learn PromQL, even if 
> > grafana is provided, the cost of promql learning cannot be reduced.
> >
> > It is not easy for everyone to learn PromQL.
>
> If users need to create/update dashboards or alerts then yes they need 
> to understand PromQL. They also need to understand the metrics and their 
> meanings.
>
> I would say this is the case for any database type system, for example 
> needing to know SQL for MariaDB or KQL for Kibana & Elasticsearch.
>
> For both of those examples tools to allow very simple queries to be 
> constructed graphically do exist, but you very quickly outgrow them - 
> all these systems are only fully useful if you are able to use the power 
> of their different query languages.
>
> The sort of query which can be easily represented on a web page is so 
> basic that it generally isn't that useful, and trying to represent more 
> complex queries would quickly become more confusing than seeing the 
> underlying PromQL. For example how would you visually represent queries 
> which use sums of rates or join metrics together?
>
> I would suggest as far as query languages go, PromQL isn't that complex. 
> The basics should be relatively straightforward to learn, with more 
> complex ideas being things you can add over time to your understanding. 
> One thing that can be useful for less experienced users are example 
> queries or cookbooks - often you might find a set of alerts/queries for 
> dashboards have similarities.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/b00a5d55-67b6-46fc-a808-ff16f02525f4n%40googlegroups.com.

Re: [prometheus-users] Promql is a bit complicated. Is there any way to reduce the difficulty of use？

[lizhihua0925]

I agree with you. 


PromQL is a power query language and the web page can’t easily represent the 
power.


It is also a good idea to provide examples or cookbooks.


Kind regards, Allenzh li


在 2020年12月9日 16:50，Stuart Clark 写道：


On 09/12/2020 08:34, Allenzh li wrote: > The key problem is that ordinary users 
need to learn PromQL, even if > grafana is provided, the cost of promql 
learning cannot be reduced. > > It is not easy for everyone to learn PromQL. If 
users need to create/update dashboards or alerts then yes they need to 
understand PromQL. They also need to understand the metrics and their meanings. 
I would say this is the case for any database type system, for example needing 
to know SQL for MariaDB or KQL for Kibana & Elasticsearch. For both of those 
examples tools to allow very simple queries to be constructed graphically do 
exist, but you very quickly outgrow them - all these systems are only fully 
useful if you are able to use the power of their different query languages. The 
sort of query which can be easily represented on a web page is so basic that it 
generally isn't that useful, and trying to represent more complex queries would 
quickly become more confusing than seeing the underlying PromQL. For example 
how would you visually represent queries which use sums of rates or join 
metrics together? I would suggest as far as query languages go, PromQL isn't 
that complex. The basics should be relatively straightforward to learn, with 
more complex ideas being things you can add over time to your understanding. 
One thing that can be useful for less experienced users are example queries or 
cookbooks - often you might find a set of alerts/queries for dashboards have 
similarities.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/tencent_59C3911CC8972C18218C5747%40qq.com.

Re: [prometheus-users] Promql is a bit complicated. Is there any way to reduce the difficulty of use？

[Stuart Clark]

On 09/12/2020 08:34, Allenzh li wrote:
The key problem is that ordinary users need to learn PromQL, even if 
grafana is provided, the cost of promql learning cannot be reduced.


It is not easy for everyone to learn PromQL.


If users need to create/update dashboards or alerts then yes they need 
to understand PromQL. They also need to understand the metrics and their 
meanings.


I would say this is the case for any database type system, for example 
needing to know SQL for MariaDB or KQL for Kibana & Elasticsearch.


For both of those examples tools to allow very simple queries to be 
constructed graphically do exist, but you very quickly outgrow them - 
all these systems are only fully useful if you are able to use the power 
of their different query languages.


The sort of query which can be easily represented on a web page is so 
basic that it generally isn't that useful, and trying to represent more 
complex queries would quickly become more confusing than seeing the 
underlying PromQL. For example how would you visually represent queries 
which use sums of rates or join metrics together?


I would suggest as far as query languages go, PromQL isn't that complex. 
The basics should be relatively straightforward to learn, with more 
complex ideas being things you can add over time to your understanding. 
One thing that can be useful for less experienced users are example 
queries or cookbooks - often you might find a set of alerts/queries for 
dashboards have similarities.


--
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/c0239984-8aa1-6fcc-345c-858b2a6a2d1e%40Jahingo.com.

Re: [prometheus-users] Promql is a bit complicated. Is there any way to reduce the difficulty of use？

[Allenzh li]

The key problem is that ordinary users need to learn PromQL, even if 
grafana is provided, the cost of promql learning cannot be reduced.

It is not easy for everyone to learn PromQL.

Kind regards, 
Allenzh li

在2020年12月9日星期三 UTC+8 下午4:21:24 写道：

> On 09/12/2020 06:50, Allenzh li wrote: 
> > When create a prometheus alerting rule, the promql is a bit 
> > complicated for ordinary users. 
> > 
> > Since I want to hide native promql expr and labels, ordinary users 
> > only select some metric name or labels in a web page then can create a 
> > alerting rule. 
> > 
> > Is there any best practice? 
> > 
> I would be cautious about hiding PromQL. You could create a simple web 
> page for some types of very basic alerts quite easily, for example a 
> very simple fixed threshold for a single metric, but in reality those 
> sort of rules are rarely very common. 
>
> One of the big advantages of Prometheus above other systems is the more 
> expressive query language, which allows for much more useful alerting 
> rules. 
>
> Common alerting rules might contain calculations (e.g. a ratio between 
> metrics), multiple metrics (e.g. comparing a value in two parts of the 
> system), aggregation (sum or sum_over_time) and rates (increase, rate, 
> predict_linear). It would be very difficult to represent such queries 
> via a web page, other than allowing for a text box input. 
>
> However, there can be advantages of some extra help via an interface. 
> Grafana for example allows you to quickly find metrics and gives instant 
> visualisation as you write your query. 
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/8a6edcf4-b6a2-4717-b7d3-54b2ef77ae9an%40googlegroups.com.

Re: [prometheus-users] Promql is a bit complicated. Is there any way to reduce the difficulty of use？

[Stuart Clark]

On 09/12/2020 06:50, Allenzh li wrote:
When create a prometheus alerting rule, the promql is a bit 
complicated for ordinary users.


Since I want to hide native promql expr and labels, ordinary users 
only select some metric name or labels in a web page then can create a 
alerting rule.


Is there any best practice?

I would be cautious about hiding PromQL. You could create a simple web 
page for some types of very basic alerts quite easily, for example a 
very simple fixed threshold for a single metric, but in reality those 
sort of rules are rarely very common.


One of the big advantages of Prometheus above other systems is the more 
expressive query language, which allows for much more useful alerting rules.


Common alerting rules might contain calculations (e.g. a ratio between 
metrics), multiple metrics (e.g. comparing a value in two parts of the 
system), aggregation (sum or sum_over_time) and rates (increase, rate, 
predict_linear). It would be very difficult to represent such queries 
via a web page, other than allowing for a text box input.


However, there can be advantages of some extra help via an interface. 
Grafana for example allows you to quickly find metrics and gives instant 
visualisation as you write your query.


--
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/82611f4b-f9ec-6d6a-529e-5ff854950275%40Jahingo.com.

[prometheus-users] Promql is a bit complicated. Is there any way to reduce the difficulty of use？

[Allenzh li]

When create a prometheus alerting rule, the promql is a bit complicated for 
ordinary users.

Since I want to hide native promql expr and labels, ordinary users only 
select some metric name or labels in a web page then can create a alerting 
rule.

Is there any best practice?


Kind regards, 
Allenzh li

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/2573229f-2dad-48f3-9a1b-f44867ee4026n%40googlegroups.com.

Re: [prometheus-users] Promql dort by timestamp

[Julien Pivotto]

On 06 Dec 07:10, kiran wrote:
> Hello all
> 
> From a promql query
> 1. I would like to get 5 most recent(sorted by descending time stamp)
> results.
> 2. Also is there a way to return the time stamp in the results in human
> readable ISO format?


1. This is not trivial, and it would only work for plain selector.  It
also depend on non-trivial internal promql that are not guaranteed to
work, but I think this should work:

0*topk(5,timestamp(my_metric)) + my_metric

2. no

> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/prometheus-users/CAOnWYZWa9pmjwv4%2BK8AzQmjiEBEjHBhP4qL8F%2ByEhnwvyjYF%2BQ%40mail.gmail.com.

-- 
Julien Pivotto
@roidelapluie

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/20201206121352.GA54688%40oxygen.

[prometheus-users] Promql dort by timestamp

[kiran]

Hello all

>From a promql query
1. I would like to get 5 most recent(sorted by descending time stamp)
results.
2. Also is there a way to return the time stamp in the results in human
readable ISO format?

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAOnWYZWa9pmjwv4%2BK8AzQmjiEBEjHBhP4qL8F%2ByEhnwvyjYF%2BQ%40mail.gmail.com.

Re: [prometheus-users] PromQL query to sort by numbers

[Tom005]

Ok thanks guys. It's a bit annoying I will say. 
My use case is that I'm monitoring CPU cores, and it goes 0, 1, 10, 11 
which is just annoying as I said. 
Look forward to the release. 

On Monday, 30 November 2020 at 14:13:07 UTC Julien Pivotto wrote:

> That kind of sort is on the roadmap but it will still take a few time
> before we have it.
>
> On 30 Nov 14:02, Stuart Clark wrote:
> > On 30/11/2020 13:44, Tom005 wrote:
> > > Ok yeah, I can see that and I can do that.
> > > I'm actually using the graph visualisation though, which I need. Is
> > > there a way to sort from that?
> > > 
> > I'm not an expert, so you might be better asking the Grafana people, but
> > google does mention how to do it via the JSON, but I'm sure there is a 
> way
> > in the UI:
> > 
> > https://github.com/grafana/grafana/issues/2640#issuecomment-606012950
> > 
> > -- 
> > You received this message because you are subscribed to the Google 
> Groups "Prometheus Users" group.
> > To unsubscribe from this group and stop receiving emails from it, send 
> an email to prometheus-use...@googlegroups.com.
> > To view this discussion on the web visit 
> https://groups.google.com/d/msgid/prometheus-users/f4451964-3c3d-af4b-f406-4bb58597f516%40Jahingo.com
> .
>
> -- 
> Julien Pivotto
> @roidelapluie
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/3b705d40-f535-42da-998a-8c645903acfbn%40googlegroups.com.

Re: [prometheus-users] PromQL query to sort by numbers

[Julien Pivotto]

That kind of sort is on the roadmap but it will still take a few time
before we have it.

On 30 Nov 14:02, Stuart Clark wrote:
> On 30/11/2020 13:44, Tom005 wrote:
> > Ok yeah, I can see that and I can do that.
> > I'm actually using the graph visualisation though, which I need. Is
> > there a way to sort from that?
> > 
> I'm not an expert, so you might be better asking the Grafana people, but
> google does mention how to do it via the JSON, but I'm sure there is a way
> in the UI:
> 
> https://github.com/grafana/grafana/issues/2640#issuecomment-606012950
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/prometheus-users/f4451964-3c3d-af4b-f406-4bb58597f516%40Jahingo.com.

-- 
Julien Pivotto
@roidelapluie

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/20201130141256.GA618220%40oxygen.

Re: [prometheus-users] PromQL query to sort by numbers

[Stuart Clark]

On 30/11/2020 13:44, Tom005 wrote:

Ok yeah, I can see that and I can do that.
I'm actually using the graph visualisation though, which I need. Is 
there a way to sort from that?


I'm not an expert, so you might be better asking the Grafana people, but 
google does mention how to do it via the JSON, but I'm sure there is a 
way in the UI:


https://github.com/grafana/grafana/issues/2640#issuecomment-606012950

--
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/f4451964-3c3d-af4b-f406-4bb58597f516%40Jahingo.com.

Re: [prometheus-users] PromQL query to sort by numbers

[Tom005]

Ok yeah, I can see that and I can do that. 
I'm actually using the graph visualisation though, which I need. Is there a 
way to sort from that? 

On Monday, 30 November 2020 at 13:38:12 UTC Stuart Clark wrote:

> On 30/11/2020 13:31, Tom005 wrote:
> > Right, I am using grafana to display. Is there a way to do that do you 
> > know?
> > Appreciate that it might be going slightly off topic away from 
> > Prometheus.
> >
> Exactly how you do it will depend on the visualisation you are using and 
> the version. For example for a table you can use the transform options 
> to convert labels into fields and decide what to show, click on the 
> field to sort. You can also set the default sorting.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/d81fb343-c894-422e-b60a-48967f7023c7n%40googlegroups.com.

Re: [prometheus-users] PromQL query to sort by numbers

[Stuart Clark]

On 30/11/2020 13:31, Tom005 wrote:
Right, I am using grafana to display. Is there a way to do that do you 
know?
Appreciate that it might be going slightly off topic away from 
Prometheus.


Exactly how you do it will depend on the visualisation you are using and 
the version. For example for a table you can use the transform options 
to convert labels into fields and decide what to show, click on the 
field to sort. You can also set the default sorting.


--
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/063be734-545c-e2ba-7f92-5b73915023fb%40Jahingo.com.

Re: [prometheus-users] PromQL query to sort by numbers

[Tom005]

Right, I am using grafana to display. Is there a way to do that do you 
know? 
Appreciate that it might be going slightly off topic away from Prometheus. 

On Monday, 30 November 2020 at 13:29:59 UTC Stuart Clark wrote:

> On 30/11/2020 13:06, Tom005 wrote:
> > Sorry, yeah I actually meant the metric labels, I'd like to sort from 
> > the metric label in that order.
> >
> How data is displayed is up to whatever system is doing the API query. 
> For example Grafana has various sort options.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/d2d435c5-ff38-44ec-98c7-7b38d96caf84n%40googlegroups.com.

Re: [prometheus-users] PromQL query to sort by numbers

[Stuart Clark]

On 30/11/2020 13:06, Tom005 wrote:
Sorry, yeah I actually meant the metric labels, I'd like to sort from 
the metric label in that order.


How data is displayed is up to whatever system is doing the API query. 
For example Grafana has various sort options.


--
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/105462e9-02b1-0c80-e985-6b50f071a9c0%40Jahingo.com.

Re: [prometheus-users] PromQL query to sort by numbers

[Tom005]

Sorry, yeah I actually meant the metric labels, I'd like to sort from the 
metric label in that order.

On Monday, 30 November 2020 at 12:27:25 UTC Stuart Clark wrote:

> On 30/11/2020 11:52, Tom005 wrote:
> > I have a metric that I'd like to sort by numbers.
> > Currently it is sorted like this:
> > 0, 1, 11, 12, 13... 2, 20,21... etc.
> > I'd like it to go like this:
> > 0, 1, 2, 3, 3...10, 11, 12... etc.
> >
> > Is that possible?
>
> How do you mean you have a metric sorted like that?
>
> Do you mean that you have a metric with the given values for a 
> particular time period?
>
> Or are you referring to the value of one of the metric's labels?
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/26b6071a-1576-4b19-8005-d5bb92f6bc7cn%40googlegroups.com.

Re: [prometheus-users] PromQL query to sort by numbers

[Stuart Clark]

On 30/11/2020 11:52, Tom005 wrote:

I have a metric that I'd like to sort by numbers.
Currently it is sorted like this:
0, 1, 11, 12, 13... 2, 20,21... etc.
I'd like it to go like this:
0, 1, 2, 3, 3...10, 11, 12... etc.

Is that possible?


How do you mean you have a metric sorted like that?

Do you mean that you have a metric with the given values for a 
particular time period?


Or are you referring to the value of one of the metric's labels?

--
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/473ab349-4c5c-a10c-335b-70089e05a20e%40Jahingo.com.

[prometheus-users] PromQL query to sort by numbers

[Tom005]

I have a metric that I'd like to sort by numbers.
Currently it is sorted like this:
0, 1, 11, 12, 13... 2, 20,21... etc.
I'd like it to go like this:
0, 1, 2, 3, 3...10, 11, 12... etc.

Is that possible?
Thanks 

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/da593369-a8a3-4fc4-a42a-88d57a43ac80n%40googlegroups.com.

Re: [prometheus-users] PromQL query to find the duration of each firing alert

[Chris Siebenmann]

> I am looking for a solution to calculate the total duration of each
> firing alert since it started firing. Following is the query I tried,
> but i see the value for all the firing alert is 86400
> 
> (avg_over_time(customer_ALERTS{alertstate="firing",severity="critical"}[24h]))
> *24 * 3600

 You can only use avg_over_time() this way on metrics that are always
present and are either 0 or 1; the classical metric for this is 'up',
whether scrapes succeed, but there are many others that at least
approximate this (such as Blackbox 'probe_success'). Unfortunately
alerts are not like this; when an ALERTS metric is present at all, it's
always one. Therefor the average over the time it's present is 1.

 There are two possible queries to deal with this, depending on how
many assumptions you want to make about your setup. The simple version
is to count how many times the metric is present over your time period
and then multiply this by your Prometheus alert checking interval. For
example, if you check all alert rules every 15 seconds (a common default),
you would do the following to generate the total duration of every alert
in seconds over the past 24 hours:

  count_over_time(ALERTS{alertstate="firing",severity="critical"}[24h]) \
 * 15

Every single count of a particular ALERTS metric being present
represents a 15 second period when that particular alert was firing,
so the total amount of time (in seconds) is just that number times 15
(seconds). Note that this is subtly different from 'the total duration
of each alert from when it started firing', which Prometheus cannot
readily calculate. If a single alert fired twice during the past 24
hours, this PromQL expression will give you the total time across both
incidents, not two separate duration metrics (one for each incident).

 If you don't want to embed knowledge of your alert rule interval
into your dashboard (perhaps different alert rules have different
check frequency, perhaps you want to change them later), you need to
use a subquery, which lets you explicitly specify the interval. Here
you would change the '[24h]' to '[24h:15s]'. This will likely cause
Prometheus to do more work but you probably have few enough ALERTS
metrics that this doesn't matter.

 Generally you want to pick a subquery interval (the '15s') that is
at least as small as your smallest real interval. This insures you
won't miss-count metrics that only appeared briefly.

- cks

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/20201127200739.5D1C03201C0%40apps1.cs.toronto.edu.

[prometheus-users] PromQL query to find the duration of each firing alert

[Shilpa Akhilesh]

Hi,

I am looking for a solution to calculate the total duration of each firing
alert  since it started firing. Following is the query I tried, but i see
the value for all the firing alert is 86400

(avg_over_time(customer_ALERTS{alertstate="firing",severity="critical"}[24h]))
*24 * 3600

Can somebody confirm if the query I used is right or  please guide me to
derive the right value.

Once I have the right query, then the next step is to visualize in Grafana
with two columns, one with the alert name and then the duration for each
alert name.

Regards,
Shilpa

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CA%2BW8-JZoYmc1YtXJBF2hKT%3D7Bof5mZApLP1Vz8uqRtmu-xe_Kg%40mail.gmail.com.

[prometheus-users] PROMQL: how to add values when there is no data returned?

['John Tipper' via Prometheus Users]

Hi all,

I have a data model where some metrics are namespaced by client, 
environment and deployment name. I am interested in creating a summary of 
each deployment, where that summary is based on the number of alerts that 
are present for each deployment.

I can get the deployments in the dev, uat, and prod environments using this 
query:
group by(tenant, environment, deployment)(up{environment=~"dev|uat|prod"}) 
- 1 # returns the following by way of example: 
{deployment="default",environment="dev",tenant="tenant1"} 0 
{deployment="default",environment="prod",tenant="tenant3"} 0 
{deployment="default",environment="prod",tenant="tenant2"} 0 
{deployment="default",environment="uat",tenant="tenant1"} 0 

So we can see that tenant 1 has 2 deployments in 2 different environments, 
whereas the other 2 have only one.  group by returns a value of 1, so we 
subtract 1 to get 0 for each deployment and I now wish to add to this the 
number of alerts that are applicable to each deployment.

To get the alerts, I do this:
ALERTS{severity="warning"} 

# returns something like this when there is an alert, the details in the 
alert will vary, but will always have the `tenant`, `environment` and 
`deployment` labels 

ALERTS{alertname="HostSystemdServiceCrashed",alertstate="firing",instance="example",job="node",deployment="default",environment="dev",tenant="tenant1",name="example.service",severity="warning",state="failed",type="oneshot"}
 
1 

# however, when there are no alerts, I get "no data" returned 

I can't work out how to add the alerts to the deployments whilst retaining 
the deployments for which there were no alerts returned:
(group by(tenant, environment, deployment)(up{environment=~"dev|uat|prod"}) 
-1) + on(tenant, environment, deployment) (ALERTS{severity="warning"}) 

# returns only data for the deployment for which there is an alert 
{deployment="default",environment="dev",tenant="tenant1"} 1  

# if there are no alerts, I get no data returned at all 

What I want as output is this:
{deployment="default",environment="dev",tenant="tenant1"} 1 
{deployment="default",environment="uat",tenant="tenant1"} 0 
{deployment="default",environment="prod",tenant="tenant2"} 0 
{deployment="default",environment="prod",tenant="tenant3"} 0 

How can I achieve this?

*NOTE:*

If I use sum with or, then I get this, depending on the order of the 
arguments to or:
(group by(tenant, environment, deployment)(up{environment=~"dev|uat|prod"}) 
-1) or sum by (tenant, environment, deployment) (ALERTS{severity="warning"} 
)

# returns this, note the value in `tenant1|dev|default` 

{deployment="default",environment="dev",tenant="tenant1"} 0 
{deployment="default",environment="uat",tenant="tenant1"} 0 
{deployment="default",environment="prod",tenant="tenant2"} 0 
{deployment="default",environment="prod",tenant="tenant3"} 0 

If I reverse the order of the parameters to or, I get what I am after:
{deployment="default",environment="dev",tenant="tenant1"} 1 
{deployment="default",environment="uat",tenant="tenant1"} 0 
{deployment="default",environment="prod",tenant="tenant2"} 0 
{deployment="default",environment="prod",tenant="tenant3"} 0 

But I'm stuck now if I want to do something like apply a weight to alerts 
of a different severity level, e.g. (pseudocode):
summary = 0 + sum(warning alerts) + 2*sum(alerts(critical alerts)) 

This gives the same single value series, or no data if there are no alerts.

Question is also posted to Stack Overflow, in case someone would prefer to 
answer there: 
https://stackoverflow.com/questions/64803483/promql-how-to-add-values-when-there-is-no-data-returned


Many thanks,

John

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/550eac3e-45f2-42c9-adc3-c5d2b3480a14n%40googlegroups.com.

Re: [prometheus-users] promql to cut label name

[Stuart Clark]

On 10/11/2020 14:32, ishu...@gmail.com wrote:

Hi,

I have a query with promql. I have prom sd configuration to scrape 
metrics from my kube cluster. I want to know if it is possible to cut 
string and apply it as label name.


My project has test01, test02 and preprod environments but namespaces 
will be test01-internal, test02-internal, prodref-internal and 
prodref-external .Because of prometheus sd config the env is coming as 
test01-internal where i want only test01. Is it possible to extract 
only this string and using replace i can send this as env label.




Yes you can do that with relabeling.

Use a regular expression match, with a () around the bit you want to 
keep. That will then be stored in $1, which is the default already for 
the "replacement:" field.



https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config

--
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/ef42fa58-c41d-9bc3-0119-aa4656d4ce5b%40Jahingo.com.

[prometheus-users] promql to cut label name

[ishu...@gmail.com]

Hi,

I have a query with promql. I have prom sd configuration to scrape metrics 
from my kube cluster. I want to know if it is possible to cut string and 
apply it as label name.

My project has test01, test02 and preprod environments but namespaces will 
be test01-internal, test02-internal, prodref-internal and prodref-external 
.Because of prometheus sd config the env is coming as test01-internal where 
i want only test01. Is it possible to extract only this string and using 
replace i can send this as env label.

Thanks
Eswar

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/1d87f2a6-a075-4482-86b2-25f0d8dfe555n%40googlegroups.com.

[prometheus-users] promQL: getting data from multiple metrics in single query

[kiran]

Hello all

My use case is to present to the user data of each instance in a table
format where each column indicates a metric, the first column being
instance id. And I have to refresh the table data every minute. How do I
handle getting data from multiple queries?

Is not there an elegant way of getting information from multiple metrics in
just one query which I believe is efficient rather than running one query
per metric and putting the data together. Running multiple queries every
minute(or even lesser interval) seems inefficient theoretically.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAOnWYZUKQg5iZR-TN%2Bfd8-%3D3Cpax1z77BskPZ%3DyU1Oux0gi5LA%40mail.gmail.com.

[prometheus-users] PromQL to get list of cronjobs if they miss to run at scheduled time

['Hemal Solgama' via Prometheus Users]

Hello there,

I am trying to create one grafana query for cronjobs.

I am trying to get all those cronjobs which missed their schedule time and 
couldn't run due to any reason. In prometheus, there are few metrics such 
as, kube_cronjob_next_schedule_time, kube_job_status_start_time, etc. but I 
don't know how to use them in grafana query to achieve this result.

Can anyone help me on this?

Thank you..

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/7eb9831c-fe9e-4ee4-9fd9-310cf1f773cdn%40googlegroups.com.

Re: [prometheus-users] PromQL formatter?

['ping...@hioscar.com' via Prometheus Users]

Thanks. Before that's shipped, I'm hoping I can find a formatter that can 
do similar things, i.e. as simple as "adding line breaks and proper 
indentations after each ( or {."

On Monday, September 14, 2020 at 4:59:12 PM UTC-4 sup...@gmail.com wrote:

> There's a work-in-progress design doc for this:
>
>
> https://docs.google.com/document/d/1nOBjpuCk4CsrOSm2ZjfVz2EL6gmA_CFGSbHCdY0Royg/edit
>
> On Mon, Sep 14, 2020 at 8:27 PM 'pin...@hioscar.com' via Prometheus Users 
>  wrote:
>
>> What's a good formatter you guys recommend for formatting complex queries?
>>
>> Ideally with line breaks and proper indentations after each ( or {.
>>
>> Thanks!
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to prometheus-use...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/prometheus-users/8b6cd4e4-24be-4066-9ee0-9d38850bbef3n%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/b2f7d6b1-f77c-4c40-b49b-c41cf4208aa6n%40googlegroups.com.

Re: [prometheus-users] PromQL formatter?

[Ben Kochie]

There's a work-in-progress design doc for this:

https://docs.google.com/document/d/1nOBjpuCk4CsrOSm2ZjfVz2EL6gmA_CFGSbHCdY0Royg/edit

On Mon, Sep 14, 2020 at 8:27 PM 'ping...@hioscar.com' via Prometheus Users <
prometheus-users@googlegroups.com> wrote:

> What's a good formatter you guys recommend for formatting complex queries?
>
> Ideally with line breaks and proper indentations after each ( or {.
>
> Thanks!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/8b6cd4e4-24be-4066-9ee0-9d38850bbef3n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CABbyFmqQEEaf3D3qRNeUOB3ZMu_bADAVwq8-1tNAs9d8aUL_Xg%40mail.gmail.com.

[prometheus-users] PromQL formatter?

['ping...@hioscar.com' via Prometheus Users]

What's a good formatter you guys recommend for formatting complex queries?

Ideally with line breaks and proper indentations after each ( or {.

Thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/8b6cd4e4-24be-4066-9ee0-9d38850bbef3n%40googlegroups.com.

[prometheus-users] PromQL group by

[Masnu]

Hello,

We have labeled all our nodes with the "tier" they belong to. I have the 
following query:

*(100 * node_memory_MemAvailable_bytes{node=~".*"} / 
node_memory_MemTotal_bytes{node=~".*"} < 10) * on (node) 
group_left(label_tier) kube_node_labels*

And would like *label_tier* added to the query so we can handle the alert 
differently depending on the node tier. I keep getting *found duplicate 
series for the match group* error. What am I doing wrong? 

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/6e6fc87d-59d6-4d77-b407-385ca6bfec4an%40googlegroups.com.

Re: [prometheus-users] PromQL - Possible to query label by list of values?

[Julien Pivotto]

On 12 Aug 09:12, Michael Black wrote:
> Is is possible in PromQL to query a label with a list of values? For 
> example this is a how you'd do it in SQL:
> 
> SELECT *
> FROM my_table
> WHERE column_1 IN ('A', 'B', 'C')
> 
> Is there a similar syntax in PromQL? I would have imagined something like 
> this:
> 
> `my_metric{Lable_1=["A", "B", "C"]}`

Yes that is possible with a regex:

my_metric{Lable_1=~"A|B|C"}`

> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/prometheus-users/5a5448e8-f054-4ca5-8400-8eb9100a9555n%40googlegroups.com.


-- 
Julien Pivotto
@roidelapluie

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/20200812161753.GA880615%40oxygen.

[prometheus-users] PromQL - Possible to query label by list of values?

[Michael Black]

Is is possible in PromQL to query a label with a list of values? For 
example this is a how you'd do it in SQL:

SELECT *
FROM my_table
WHERE column_1 IN ('A', 'B', 'C')

Is there a similar syntax in PromQL? I would have imagined something like 
this:

`my_metric{Lable_1=["A", "B", "C"]}`

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/5a5448e8-f054-4ca5-8400-8eb9100a9555n%40googlegroups.com.

Re: [prometheus-users] PromQL stateless issue

[fiala...@gmail.com]

Thank you!

It helps.

On Wednesday, August 12, 2020 at 9:05:17 AM UTC+2 Julien Pivotto wrote:

> On 12 Aug 00:00, fiala...@gmail.com wrote:
> > Hi, 
> > 
> > I would like to write a rule for alerting e.g. packet discards in 
> percent.
> > 
> > My requirements are:
> > - trigger an alert when discards are at least 10 minutes higher than X
> > - close alert when discards are more then 60m equals to zero
> > 
> > I'm not sure, how to define it in PromQL. Because I haven't information 
> > about state of the alert in promql. Have you any suggestions?
>
>
> https://fr.slideshare.net/roidelapluie/improved-alerting-with-prometheus-and-alertmanager
>
> See the part about Hysteresis
>
> Regards,
>
> > 
> > Thank you,
> > 
> > -- 
> > You received this message because you are subscribed to the Google 
> Groups "Prometheus Users" group.
> > To unsubscribe from this group and stop receiving emails from it, send 
> an email to prometheus-use...@googlegroups.com.
> > To view this discussion on the web visit 
> https://groups.google.com/d/msgid/prometheus-users/e3331264-0f0c-4887-b951-53cc90c17844n%40googlegroups.com
> .
>
>
> -- 
> Julien Pivotto
> @roidelapluie
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/91113fc5-916c-4cf2-8cfb-52256372db2cn%40googlegroups.com.

Re: [prometheus-users] PromQL stateless issue

[Julien Pivotto]

On 12 Aug 00:00, fiala...@gmail.com wrote:
> Hi, 
> 
> I would like to write a rule for alerting e.g. packet discards in percent.
> 
> My requirements are:
> - trigger an alert when discards are at least 10 minutes higher than X
> - close alert when discards are more then 60m equals to zero
> 
> I'm not sure, how to define it in PromQL. Because I haven't information 
> about state of the alert in promql. Have you any suggestions?

https://fr.slideshare.net/roidelapluie/improved-alerting-with-prometheus-and-alertmanager

See the part about Hysteresis

Regards,

> 
> Thank you,
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/prometheus-users/e3331264-0f0c-4887-b951-53cc90c17844n%40googlegroups.com.


-- 
Julien Pivotto
@roidelapluie

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/20200812070510.GA7362%40oxygen.

[prometheus-users] PromQL stateless issue

[fiala...@gmail.com]

Hi, 

I would like to write a rule for alerting e.g. packet discards in percent.

My requirements are:
- trigger an alert when discards are at least 10 minutes higher than X
- close alert when discards are more then 60m equals to zero

I'm not sure, how to define it in PromQL. Because I haven't information 
about state of the alert in promql. Have you any suggestions?

Thank you,

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/e3331264-0f0c-4887-b951-53cc90c17844n%40googlegroups.com.

[prometheus-users] promql

[raghu padaki]

need help for writing promql query to populate metrics
i am using prometheus to integrate mongodb

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/965b284e-88a4-474a-b143-7eb2277239cfo%40googlegroups.com.

Re: [prometheus-users] PromQL to merge two metrics data

[Brian Candler]

I'd suggest that's *not* the prometheus way, because it's rare to have two 
metrics which actually mean exactly the same thing (and if they did, they 
ought to have the same metric name, and different labels)

Therefore, a single query which mixes multiple metrics into one result set 
is usually a bad idea.

In this particular instance - total processes and running processes - it 
might be OK; but even then, if you had (say) 3 systems being shown, which 
means you get 6 metrics, having them mixed together is usually not useful.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/54661e8f-9690-408a-b379-84a816261ff9o%40googlegroups.com.

Re: [prometheus-users] PromQL to merge two metrics data

[Harkishen Singh]

Actually, this is the Prometheus way,

{__name__=~"system_total_processes|system_running_processes", 
instance=~"$host"}

for your use case!

On Tuesday, June 23, 2020 at 12:26:49 AM UTC+5:30 Harkishen Singh wrote:

> Either you can go the grafana way, or you can query something smarter like,
>
> {__name__=~"system_.*"}
>
> but that will give you all the queries starting with systems. So, you can 
> add some or/and conditions there as well and make it more productive.
>
> On Monday, June 22, 2020 at 2:32:06 PM UTC+5:30 neel...@gmail.com wrote:
>
>> Hi Julien,
>>
>> Yes, you are right. Thanks for your input. Appreciated.
>>
>> Thanks,
>> Neel Patel
>>
>> On Mon, Jun 22, 2020 at 2:17 PM Julien Pivotto  
>> wrote:
>>
>>> On 22 Jun 14:14, neel patel wrote:
>>> > Hi,
>>>
>>> Hi,
>>>
>>> > 
>>> > How can we merge two metrics data in a prometheus query so that grafana
>>> > will plot it in a single graph rather than individual data.
>>> > 
>>> > I am using the below query to plot the data in my dashboard.
>>> > 
>>> > system_total_processes{instance=~"$host"}
>>> > system_running_processes{instance=~"$host"}
>>> > 
>>> > I read the PromQL docs but didn't find anything regarding merging 2 
>>> metrics
>>> > So how to merge both above metrics using a single query so that we can 
>>> get
>>> > the data with 2 rows. Any pointers will be appreciated?
>>>
>>> You are looking into adding multiple queries in Grafana. Grafana
>>> supports up to 26 queries per panel.
>>>
>>> Look for + Query button:
>>>
>>> https://grafana.com/static/img/docs/queries/prometheus-query-editor-7-0.png
>>>
>>>
>>> regards,
>>>
>>>
>>> > 
>>> > Thanks in Advance
>>> > 
>>> > -- 
>>> > You received this message because you are subscribed to the Google 
>>> Groups "Prometheus Users" group.
>>> > To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to prometheus-use...@googlegroups.com.
>>> > To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/prometheus-users/CAPTvBtvbn0wqTnnLKEFrik3fVMu1LOzeOs1rNEaUBi3fpi8wXg%40mail.gmail.com
>>> .
>>>
>>> -- 
>>> Julien Pivotto
>>> @roidelapluie
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/e49d3c74-f87a-4095-9b51-0facefdc74afn%40googlegroups.com.

Re: [prometheus-users] PromQL to merge two metrics data

[Harkishen Singh]

Either you can go the grafana way, or you can query something smarter like,

{__name__=~"system_.*"}

but that will give you all the queries starting with systems. So, you can 
add some or/and conditions there as well and make it more productive.

On Monday, June 22, 2020 at 2:32:06 PM UTC+5:30 neel...@gmail.com wrote:

> Hi Julien,
>
> Yes, you are right. Thanks for your input. Appreciated.
>
> Thanks,
> Neel Patel
>
> On Mon, Jun 22, 2020 at 2:17 PM Julien Pivotto  
> wrote:
>
>> On 22 Jun 14:14, neel patel wrote:
>> > Hi,
>>
>> Hi,
>>
>> > 
>> > How can we merge two metrics data in a prometheus query so that grafana
>> > will plot it in a single graph rather than individual data.
>> > 
>> > I am using the below query to plot the data in my dashboard.
>> > 
>> > system_total_processes{instance=~"$host"}
>> > system_running_processes{instance=~"$host"}
>> > 
>> > I read the PromQL docs but didn't find anything regarding merging 2 
>> metrics
>> > So how to merge both above metrics using a single query so that we can 
>> get
>> > the data with 2 rows. Any pointers will be appreciated?
>>
>> You are looking into adding multiple queries in Grafana. Grafana
>> supports up to 26 queries per panel.
>>
>> Look for + Query button:
>>
>> https://grafana.com/static/img/docs/queries/prometheus-query-editor-7-0.png
>>
>>
>> regards,
>>
>>
>> > 
>> > Thanks in Advance
>> > 
>> > -- 
>> > You received this message because you are subscribed to the Google 
>> Groups "Prometheus Users" group.
>> > To unsubscribe from this group and stop receiving emails from it, send 
>> an email to prometheus-use...@googlegroups.com.
>> > To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/prometheus-users/CAPTvBtvbn0wqTnnLKEFrik3fVMu1LOzeOs1rNEaUBi3fpi8wXg%40mail.gmail.com
>> .
>>
>> -- 
>> Julien Pivotto
>> @roidelapluie
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/1741c2cd-7f89-47ae-b07c-611d0be24755n%40googlegroups.com.

Re: [prometheus-users] PromQL to merge two metrics data

[neel patel]

Hi Julien,

Yes, you are right. Thanks for your input. Appreciated.

Thanks,
Neel Patel

On Mon, Jun 22, 2020 at 2:17 PM Julien Pivotto 
wrote:

> On 22 Jun 14:14, neel patel wrote:
> > Hi,
>
> Hi,
>
> >
> > How can we merge two metrics data in a prometheus query so that grafana
> > will plot it in a single graph rather than individual data.
> >
> > I am using the below query to plot the data in my dashboard.
> >
> > system_total_processes{instance=~"$host"}
> > system_running_processes{instance=~"$host"}
> >
> > I read the PromQL docs but didn't find anything regarding merging 2
> metrics
> > So how to merge both above metrics using a single query so that we can
> get
> > the data with 2 rows. Any pointers will be appreciated?
>
> You are looking into adding multiple queries in Grafana. Grafana
> supports up to 26 queries per panel.
>
> Look for + Query button:
> https://grafana.com/static/img/docs/queries/prometheus-query-editor-7-0.png
>
>
> regards,
>
>
> >
> > Thanks in Advance
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "Prometheus Users" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to prometheus-users+unsubscr...@googlegroups.com.
> > To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/CAPTvBtvbn0wqTnnLKEFrik3fVMu1LOzeOs1rNEaUBi3fpi8wXg%40mail.gmail.com
> .
>
> --
> Julien Pivotto
> @roidelapluie
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAPTvBttWtGoNV14SbSDnKEVxwJu3uEo0sRHEXc3Loj7mtyfomg%40mail.gmail.com.

Re: [prometheus-users] PromQL to merge two metrics data

[Julien Pivotto]

On 22 Jun 14:14, neel patel wrote:
> Hi,

Hi,

> 
> How can we merge two metrics data in a prometheus query so that grafana
> will plot it in a single graph rather than individual data.
> 
> I am using the below query to plot the data in my dashboard.
> 
> system_total_processes{instance=~"$host"}
> system_running_processes{instance=~"$host"}
> 
> I read the PromQL docs but didn't find anything regarding merging 2 metrics
> So how to merge both above metrics using a single query so that we can get
> the data with 2 rows. Any pointers will be appreciated?

You are looking into adding multiple queries in Grafana. Grafana
supports up to 26 queries per panel.

Look for + Query button:
https://grafana.com/static/img/docs/queries/prometheus-query-editor-7-0.png


regards,


> 
> Thanks in Advance
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/prometheus-users/CAPTvBtvbn0wqTnnLKEFrik3fVMu1LOzeOs1rNEaUBi3fpi8wXg%40mail.gmail.com.

-- 
Julien Pivotto
@roidelapluie

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/20200622084710.GA196956%40oxygen.

[prometheus-users] PromQL to merge two metrics data

[neel patel]

Hi,

How can we merge two metrics data in a prometheus query so that grafana
will plot it in a single graph rather than individual data.

I am using the below query to plot the data in my dashboard.

system_total_processes{instance=~"$host"}
system_running_processes{instance=~"$host"}

I read the PromQL docs but didn't find anything regarding merging 2 metrics
So how to merge both above metrics using a single query so that we can get
the data with 2 rows. Any pointers will be appreciated?

Thanks in Advance

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CAPTvBtvbn0wqTnnLKEFrik3fVMu1LOzeOs1rNEaUBi3fpi8wXg%40mail.gmail.com.

[prometheus-users] Promql for filtering pods restert for the last workload deployed

[Neha Gupta]

Hi,

I have a specific requirement to set rules for pods restart notifications 
only for the most recent workload deployed.

Any help would be highly appreciated.

Thanks
Neha

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/fead1df8-c0e2-4857-b6e6-5b7423122bcc%40googlegroups.com.

Re: [prometheus-users] PromQL: comparison to custom metric and/or static threshold

[Harald Koch]

On Mon, May 11, 2020, at 11:47, Justin W wrote:
> Hi all,
> 
> I'm trying to compare the calculated filesystem used to either a static 
> threshold, or a custom threshold if a certain metric is present. What I have 
> so far is:
> 
> ((avg(node_filesystem_used_bytes{mountpoint=~".*foo.*"}) by 
> (site)/avg(node_filesystem_size_bytes{mountpoint=~".*foo.*"}) by (site)) > 
> my_custom_threshold) or 
> ((avg(node_filesystem_used_bytes{mountpoint=~".*foo.*"}) by 
> (site)/avg(node_filesystem_size_bytes{mountpoint=~".*foo.*"}) by (site)) > 
> 0.7)
> 
> What I want this query to do is "If the my_custom_threshold is present for 
> the (site), compare against that, else compare against 70%".


I wrote a long and incoherent article about this:

https://www.haraldkoch.ca/blog/index.php/2020/03/14/prometheus-alerting-rules-and-metadata/

Suggestions welcome!!

-- 
Harald

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/a210390c-d853-4e9f-86fd-d431a54cd848%40www.fastmail.com.