[prometheus-users] Create custom alert for a specific service

[Himanshu Gude]

Hello all, There is a requirement by the client where they want the alert 
for a particular service (app), V2. is there any way, we can create a 
custom alert for monitoring the particular deployment. 

This is the current the alert that we have for monitoring the deployments 
(pods)

*name: KubePodNotReady 
*
*expr: 
(kube_pod_status_phase{job="kubernetes-service-endpoints",phase=~"Pending|Unknown"})
 
> 0 
*
*for: 1m*
*labels:*
*cluster: Prometheus_Cups_k8s_ipccprod*
*resource: IKS*
*severity: warning*
*annotations:*
*summary: {{ $labels.pod }} is not ready*

This a high priority requirement by the client as v2 service is important. 
kindly provide me some lead... Thanks in advance.

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/db4dff3d-329b-447c-800f-720140477134n%40googlegroups.com.

Re: [prometheus-users] Scrape only a particular namespace and ignore all other namespaces

[Himanshu Gude]

Hello Julien, 
I have used the method which you have mentioned. I think this way will stop 
the prometheus from scraping metrics for all the other namespaces except 
devsecops. If this works it will also fix the issue that i m facing. 
But I think supppressing the alerts from alertmanager will be a better fix 
for me. If you can suggest something for that it will help. 

Thanks again!

On Monday, March 27, 2023 at 12:24:34 AM UTC+5:30 Julien Pivotto wrote:

> You can update the prometheus config:
>
>
> scrape_configs:
> - job_name: 'kubernetes-pods'
> kubernetes_sd_configs:
> - role: pod
> namespaces:
> names:
> - devsecops
>
> On 25 Mar 23:00, Himanshu Gude wrote:
> > Hello guys,
> > 
> > There is a requirement in my project which states that, the client only 
> > wants alerts from "devsecops" namespace and alerts from all other 
> > namespaces should be ignored. which means alerts from namespaces other 
> than 
> > "devsecops" should not be captured by the alertmanager and should not 
> > trigger tickets in ServiceNow.
> > 
> > In short, prometheus should only scrape metrics from only one namespace 
> ie; 
> > devsecops
> > 
> > would really appreciate some sort of hel here.
> > Thanks in advance
> > 
> > -- 
> > You received this message because you are subscribed to the Google 
> Groups "Prometheus Users" group.
> > To unsubscribe from this group and stop receiving emails from it, send 
> an email to prometheus-use...@googlegroups.com.
> > To view this discussion on the web visit 
> https://groups.google.com/d/msgid/prometheus-users/e4ef8176-930d-4743-9118-90bbab5f2164n%40googlegroups.com
> .
>
>
> -- 
> Julien Pivotto
> @roidelapluie
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/91c46a6e-7232-45dd-8139-03ef44e38ecen%40googlegroups.com.

[prometheus-users] Re: Scrape only a particular namespace and ignore all other namespaces

[Himanshu Gude]

Thank you Brian. This is exactly what I was looking for. 

On Monday, March 27, 2023 at 12:30:31 PM UTC+5:30 Brian Candler wrote:

> There are actually a a couple of options.
>
> In each alerting rule (in prometheus) you can match on labels in the 
> actual alerting expression: e.g.
>
> expr: node_boot_time_seconds > (node_boot_time_seconds offset 5m + 
> 5)
> becomes
> expr: node_boot_time_seconds*{namespace="devsecops"}* > 
> (node_boot_time_seconds offset 5m + 5)
>
> But I'd say it's simpler and more flexible to allow all the alerts to 
> fire, but suppress the notifications in alertmanager routing rules:
>
> route:
>   receiver: noc
>   routes:
> - receiver: discard
>   matchers:
> - namespace!="devsecops"
> # more routing rules if desired go here
>
> receivers:
>   - name: discard
>   - name: noc
> email_configs:
>   - to: n...@example.com
> send_resolved: false
>
> Then you can *see* alerts firing in other environments in the prometheus 
> web UI (which helps you test and tune the rules), but notifications aren't 
> sent out for anything apart from devsecops.
>
> This does require that all the alerting expressions pass through a 
> namespace label. Pasting each alerting expression into the PromQL browser 
> in the Prometheus web UI will let you see exactly what timeseries values it 
> generates (including the labels), as long as it's generating at least one 
> alert of course.
>
> On Monday, 27 March 2023 at 06:56:58 UTC+1 Himanshu Gude wrote:
>
>> yes, I m new to prometheus env. could you pls give an example for each 
>> case, which will help me to figure out what changes to be made in the 
>> configurations. Your first case is what I am looking for. 
>> Thanks in advance! 
>>
>> On Sunday, March 26, 2023 at 9:20:38 PM UTC+5:30 Brian Candler wrote:
>>
>>> That's two different things you're asking.
>>>
>>> * alerting only from "devsecops" namespace: ensure every metric has a 
>>> label with the namespace, and configure alerting rules to match only where 
>>> namespace=devsecops
>>>
>>> * scraping only from one namespace: if you can't configure the scrape 
>>> URL to fetch only one namespace, then you can use metric_relabel_rules to 
>>> discard all metrics except the ones of interest (again, matching on some 
>>> namespace label to identify which ones to keep or drop)
>>>
>>> On Sunday, 26 March 2023 at 07:28:36 UTC+1 Himanshu Gude wrote:
>>>
>>>> Hello guys,
>>>>
>>>> There is a requirement in my project which states that, the client only 
>>>> wants alerts from "devsecops" namespace and alerts from all other 
>>>> namespaces should be ignored. which means alerts from namespaces other 
>>>> than 
>>>> "devsecops" should not be captured by the alertmanager and should not 
>>>> trigger tickets in ServiceNow.
>>>>
>>>> In short, prometheus should only scrape metrics from only one namespace 
>>>> ie; devsecops
>>>>
>>>> would really appreciate some sort of hel here.
>>>> Thanks in advance
>>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/98a6a9fa-50f4-4144-9877-015e6aae8243n%40googlegroups.com.

[prometheus-users] Re: Scrape only a particular namespace and ignore all other namespaces

[Himanshu Gude]

yes, I m new to prometheus env. could you pls give an example for each 
case, which will help me to figure out what changes to be made in the 
configurations. Your first case is what I am looking for. 
Thanks in advance! 

On Sunday, March 26, 2023 at 9:20:38 PM UTC+5:30 Brian Candler wrote:

> That's two different things you're asking.
>
> * alerting only from "devsecops" namespace: ensure every metric has a 
> label with the namespace, and configure alerting rules to match only where 
> namespace=devsecops
>
> * scraping only from one namespace: if you can't configure the scrape URL 
> to fetch only one namespace, then you can use metric_relabel_rules to 
> discard all metrics except the ones of interest (again, matching on some 
> namespace label to identify which ones to keep or drop)
>
> On Sunday, 26 March 2023 at 07:28:36 UTC+1 Himanshu Gude wrote:
>
>> Hello guys,
>>
>> There is a requirement in my project which states that, the client only 
>> wants alerts from "devsecops" namespace and alerts from all other 
>> namespaces should be ignored. which means alerts from namespaces other than 
>> "devsecops" should not be captured by the alertmanager and should not 
>> trigger tickets in ServiceNow.
>>
>> In short, prometheus should only scrape metrics from only one namespace 
>> ie; devsecops
>>
>> would really appreciate some sort of hel here.
>> Thanks in advance
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/fe5bd5ed-ec7b-47c4-b326-5ca8e5b387fdn%40googlegroups.com.

[prometheus-users] Scrape only a particular namespace and ignore all other namespaces

[Himanshu Gude]

Hello guys,

There is a requirement in my project which states that, the client only 
wants alerts from "devsecops" namespace and alerts from all other 
namespaces should be ignored. which means alerts from namespaces other than 
"devsecops" should not be captured by the alertmanager and should not 
trigger tickets in ServiceNow.

In short, prometheus should only scrape metrics from only one namespace ie; 
devsecops

would really appreciate some sort of hel here.
Thanks in advance

-- 
You received this message because you are subscribed to the Google Groups 
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/prometheus-users/e4ef8176-930d-4743-9118-90bbab5f2164n%40googlegroups.com.