[prometheus-users] Can we have multiple alertmanager config in kube prom stack

2023-09-18 Thread 'Laxman Soma' via Prometheus Users 
Hello All - We recently implemented alertmanagerconfig using global 
alertmanager configuration in monitoring namespace , but we had a recent 
requirement to have multiple alertmanager config and wanted to check if 
anyone implemented the same as we are not able to send alerts

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/ee0f3485-a655-429a-b7b7-defe4de30cben%40googlegroups.com.

Re: [prometheus-users] Not able add remote_write in Prometheus config for k8s monitoring

2023-09-18 Thread Prashant Singh 
Yes , got solution , below code is working for config and also working
remote_write .

config file :
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  namespace: monitoring
metadata:
name: prometheus-server-conf
namespace: monitoring
labels:
name: prometheus-server-conf

thanks,
prashant

On Mon, Sep 18, 2023 at 12:52 PM Stuart Clark 
wrote:

> On 2023-09-18 07:20, Prashant Singh wrote:
> > Hello,
> >
> > I am not able add remote_write detail in k8s prometheus config file.
> > kubectl version: 1.27
> >
> > error :
> > Error from server (BadRequest): error when creating "config-map.yaml":
> > ConfigMap in version "v1" cannot be handled as a ConfigMap: strict
> > decoding error: unknown field "remote_write"
> >
> > prometheus config file:
> > apiVersion: v1
> > kind: ConfigMap
> > metadata:
> >   name: prometheus-server-conf
> >   labels:
> > name: prometheus-server-conf
> >   namespace: monitoring
> >
> > remote_write:
> >- url: "http://x.x.x.x:31000/api/prom/push;
> >
>
> Not really a Prometheus thing, but that isn't a valid ConfigMap. It is
> expecting top level fields called "metatdata" and "data", with a
> filename within the data section that then contains whatever data you
> are wanting (which for Prometheus is a YAML config file).
>
> --
> Stuart Clark
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/762c7be3990926a7c2867148f68db2f0%40Jahingo.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CALGft8xXX9MvHQD1_ziNcUpSaF8TuW%3DbRRSgT4OqN2Yc9zJDcg%40mail.gmail.com.

Re: [prometheus-users] Optimizing query with many duplicate calls

2023-09-18 Thread Brian Candler 
One possibility is to use a recording rule for the expensive repeated query.

If I 
rewrite avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) 
to X, I get:

((50 - 0) / (12 - 0) * ((X <= 12) - 0) + 0) or
((100 - 51) / (35.4 - 12.1) * ((X > 12 and X <= 35.4) - 12.1) + 51) or
((150 - 101) / (55.4 - 35.5) * ((X > 35.4 and X <= 55.4) - 35.5) + 101) or
((200 - 151) / (150.4 - 55.5) * ((X > 55.4 and X <= 150.4) - 55.5) + 151) or
((300 - 201) / (250.4 - 150.5) * ((X > 150.4 and X <= 250.4) - 150.5) + 
201) or
((400 - 301) / (350.4 - 250.5) * ((X > 250.4 and X <= 350.4) - 250.5) + 
301) or
((500 - 401) / (500.4 - 350.5) * ((X > 350.4 and X <= 500.4) - 350.5) + 
401) or
clamp_max(X, 600)

I guess you're trying to apply different scaling for different ranges of X:
- if X is between 0 and 12 (or negative) then rescale to 0 to 50
- if X is between 12 and 35.4 then rescale to 50(?) to 100
- if X is between 35.4 and 55.4 then rescale to 100(?) to 150
etc (except there seem to be some small discontinuities at the boundaries, 
e.g. 12 versus 12.1, 50 versus 51)

"A or B" will suppress elements in the B vector where the A vector has a 
value (i.e. with a matching label set). That means it's unnecessary to test 
the lower bounds, and I think your expression could simplify to something 
like this:

(X <= 12) * k1 + o1 or
(X <= 35.4) * k2 + o2 or
(X <= 55.4) * k3 + o3 or
(X <= 150.4) * k4  + o4 or
(X <= 250.4) * k5 + o5 or
(X <= 350.4) * k6 + o6 or
(X <= 500.4) * k7 + o7 or
clamp_max(X, 600)

That would roughly halve the number of the subexpressions X.

On Monday, 18 September 2023 at 07:01:50 UTC+1 Ben Kochie wrote:

> One thing you can do to speed things up is to eliminate the `=~` in your 
> query. Using regexp matching means it has to do a string search over every 
> instance in your Prometheus for each metric. Using exact matching (`=`) 
> will speed things up a lot. Although you won't be able to do multiple 
> matching if you want that in your dashboard variables.
>
> On Mon, Sep 18, 2023 at 7:58 AM David Leibovic  
> wrote:
>
>> Hi there, I'm trying to optimize a slow query of this form:
>>
>> (1 * avg_over_time(foo{instance=~"$i"}[$interval]) <= 10) or 
>> (2 * avg_over_time(foo{instance=~"$i"}[$interval]) <= 20) or 
>> (3 * avg_over_time(foo{instance=~"$i"}[$interval]) <= 30) or 
>> (10 * avg_over_time(foo{instance=~"$i"}[$interval]))
>>
>> I suspect it's slow because of the many duplicate calls to 
>> avg_over_time(foo{instance=~"$i"}[$interval])
>>
>> Is there some way to only call the avg_over_time function once and 
>> re-use the results subsequently? I'm using Prometheus in conjunction with 
>> Grafana, in case it's relevant.
>>
>> The full query I'm trying to optimize is much more complicated, but I 
>> figured the above would be enough to understand the problem. But in case 
>> it's helpful, here is the full query I am trying to optimize (it's an Air 
>> Quality Index computation):
>>
>> ((50 - 0) / (12 - 0) * 
>> ((avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) <= 
>> 12) - 0) + 0) or 
>> ((100 - 51) / (35.4 - 12.1) * 
>> ((avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) > 12 
>> and avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) <= 
>> 35.4) - 12.1) + 51) or 
>> ((150 - 101) / (55.4 - 35.5) * 
>> ((avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) > 
>> 35.4 and 
>> avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) <= 
>> 55.4) - 35.5) + 101) or 
>> ((200 - 151) / (150.4 - 55.5) * 
>> ((avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) > 
>> 55.4 and 
>> avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) <= 
>> 150.4) - 55.5) + 151) or 
>> ((300 - 201) / (250.4 - 150.5) * 
>> ((avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) > 
>> 150.4 and 
>> avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) <= 
>> 250.4) - 150.5) + 201) or 
>> ((400 - 301) / (350.4 - 250.5) * 
>> ((avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) > 
>> 250.4 and 
>> avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) <= 
>> 350.4) - 250.5) + 301) or 
>> ((500 - 401) / (500.4 - 350.5) * 
>> ((avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) > 
>> 350.4 and 
>> avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) <= 
>> 500.4) - 350.5) + 401) or 
>> clamp_max(avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]),
>>  
>> 600)
>>
>> Thanks for any help you can provide!
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to prometheus-use...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/prometheus-users/664689b3-9f45-4b05-9438-4225e2dce773n%40googlegroups.com
>>  
>>

Re: [prometheus-users] Not able add remote_write in Prometheus config for k8s monitoring

2023-09-18 Thread Stuart Clark 

On 2023-09-18 07:20, Prashant Singh wrote:

Hello,

I am not able add remote_write detail in k8s prometheus config file.
kubectl version: 1.27

error :
Error from server (BadRequest): error when creating "config-map.yaml":
ConfigMap in version "v1" cannot be handled as a ConfigMap: strict
decoding error: unknown field "remote_write"

prometheus config file:
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-server-conf
  labels:
name: prometheus-server-conf
  namespace: monitoring

remote_write:
   - url: "http://x.x.x.x:31000/api/prom/push;



Not really a Prometheus thing, but that isn't a valid ConfigMap. It is 
expecting top level fields called "metatdata" and "data", with a 
filename within the data section that then contains whatever data you 
are wanting (which for Prometheus is a YAML config file).


--
Stuart Clark

--
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/762c7be3990926a7c2867148f68db2f0%40Jahingo.com.

[prometheus-users] Not able add remote_write in Prometheus config for k8s monitoring

2023-09-18 Thread Prashant Singh 
Hello,

I am not able add remote_write detail in k8s prometheus config file. 
kubectl version: 1.27
error : 
*Error from server (BadRequest): error when creating "config-map.yaml": 
ConfigMap in version "v1" cannot be handled as a ConfigMap: strict decoding 
error: unknown field "remote_write"*

prometheus config file:
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-server-conf
  labels:
name: prometheus-server-conf
  namespace: monitoring

remote_write:
   - url: "http://x.x.x.x:31000/api/prom/push;


Thanks,
prashant

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/da1b6519-5095-4ec5-b5f1-ee5da0850350n%40googlegroups.com.

Re: [prometheus-users] Optimizing query with many duplicate calls

2023-09-18 Thread Ben Kochie 
One thing you can do to speed things up is to eliminate the `=~` in your
query. Using regexp matching means it has to do a string search over every
instance in your Prometheus for each metric. Using exact matching (`=`)
will speed things up a lot. Although you won't be able to do multiple
matching if you want that in your dashboard variables.

On Mon, Sep 18, 2023 at 7:58 AM David Leibovic 
wrote:

> Hi there, I'm trying to optimize a slow query of this form:
>
> (1 * avg_over_time(foo{instance=~"$i"}[$interval]) <= 10) or
> (2 * avg_over_time(foo{instance=~"$i"}[$interval]) <= 20) or
> (3 * avg_over_time(foo{instance=~"$i"}[$interval]) <= 30) or
> (10 * avg_over_time(foo{instance=~"$i"}[$interval]))
>
> I suspect it's slow because of the many duplicate calls to
> avg_over_time(foo{instance=~"$i"}[$interval])
>
> Is there some way to only call the avg_over_time function once and re-use
> the results subsequently? I'm using Prometheus in conjunction with Grafana,
> in case it's relevant.
>
> The full query I'm trying to optimize is much more complicated, but I
> figured the above would be enough to understand the problem. But in case
> it's helpful, here is the full query I am trying to optimize (it's an Air
> Quality Index computation):
>
> ((50 - 0) / (12 - 0) *
> ((avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) <=
> 12) - 0) + 0) or
> ((100 - 51) / (35.4 - 12.1) *
> ((avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) > 12
> and avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) <=
> 35.4) - 12.1) + 51) or
> ((150 - 101) / (55.4 - 35.5) *
> ((avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) >
> 35.4 and
> avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) <=
> 55.4) - 35.5) + 101) or
> ((200 - 151) / (150.4 - 55.5) *
> ((avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) >
> 55.4 and
> avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) <=
> 150.4) - 55.5) + 151) or
> ((300 - 201) / (250.4 - 150.5) *
> ((avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) >
> 150.4 and
> avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) <=
> 250.4) - 150.5) + 201) or
> ((400 - 301) / (350.4 - 250.5) *
> ((avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) >
> 250.4 and
> avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) <=
> 350.4) - 250.5) + 301) or
> ((500 - 401) / (500.4 - 350.5) *
> ((avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) >
> 350.4 and
> avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]) <=
> 500.4) - 350.5) + 401) or
> clamp_max(avg_over_time(ambient_pm25_env{instance=~"$room.*"}[$aqi_interval]),
> 600)
>
> Thanks for any help you can provide!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/664689b3-9f45-4b05-9438-4225e2dce773n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/CABbyFmrPbE_fjixveYZ4%3D0s2GdKEEADout9odJsu%2Byq7kymyqw%40mail.gmail.com.