Re: [prometheus-users] Incorrect Node Labels in kube_pod_container_info

2023-01-20 Thread Murali Krishna Kanagala
relabeling simulator / explainer tool at > https://relabeler.promlabs.com/ (shameless plug) to enter your relabeling > rules and the target labels before relabeling (both in YAML format) to see > what effect the rules have on the labels. > > Regards, > Julius > > On Thu, Ja

[prometheus-users] Incorrect Node Labels in kube_pod_container_info

2023-01-19 Thread Murali Krishna Kanagala
default scrape config. Any suggestions/ recommendations? -- Regards, Murali Krishna Kanagala -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to promet

[prometheus-users] GaugeMetricFamily Timestamp with Pushgateway

2020-08-12 Thread Murali Krishna Kanagala
push gateway. Can someone help me with this? -- Regards, Murali Krishna Kanagala -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-user

[prometheus-users] Re: Alertmanager webhook timeout

2020-07-08 Thread Murali Krishna Kanagala
Guys, Can I get some help with this? On Mon, Jul 6, 2020, 10:25 AM Murali Krishna Kanagala < kanagalamur...@gmail.com> wrote: > Hello everyone, > > Can someone help me on how can > I change the webhook timeout settings for a webhook? > Also in an alert route if AM is

[prometheus-users] Alertmanager webhook timeout

2020-07-06 Thread Murali Krishna Kanagala
Hello everyone, Can someone help me on how can I change the webhook timeout settings for a webhook? Also in an alert route if AM is waiting for a webhook response does it holds of the rest of the pipeline until that Thanks! -- You received this message because you are subscribed to the Google

Re: [prometheus-users] Relabeling Question

2020-06-29 Thread Murali Krishna Kanagala
Thanks for the replies guys! We have chosen to switch to windows exporter in one shot and adjust grafana queries and alerts accordingly. On Fri, Jun 26, 2020, 2:27 AM Ben Kochie wrote: > The wmi_exporter was renamed to windows_exporter when we moved it to the > Prometheus Community org. It origi

[prometheus-users] Relabeling Question

2020-06-25 Thread Murali Krishna Kanagala
Hello everyone, We are moving from wmi_exporter to windows_exporter and trying to do it in a smooth manner. Our Alerts and Graphs are heavily dependent on WMI metrics and moving to Windows exporter may cause some issues because of a different metric prefix (wmi to windows). We are looking for a te

Re: [prometheus-users] Re: Prometheus not able to start and gives error "opening storage failed:: found unsequential head chunk error"

2020-06-11 Thread Murali Krishna Kanagala
Try deleting the contents WAL folder and start the service. Ideally you can delete the last WAL where the service is failing to start. It is in the log just before Prometheus stopping. On Thu, Jun 11, 2020, 2:26 AM neel patel wrote: > Hi Julien, > > Unfortunately i don't have previous logs. And

Re: [prometheus-users] correlating counters across services

2020-06-08 Thread Murali Krishna Kanagala
Metrics use the labels from kubernetes. If your kubernetes tenant use specific labels like namespace etc. those will be used in the metrics automatically. On Mon, Jun 8, 2020, 4:16 PM Manoj Guglani wrote: > Hi, > > I am working on integrating prometheus for our microservices platform > deployed

Re: [prometheus-users] Ways to Mitigate on huge label value pair during high cardinalities

2020-06-04 Thread Murali Krishna Kanagala
This should be taken care by the exporter you are collecting metrics from. If you are writing your own exporter then make validate what labels stay unique all the time. For ex. If you are collecting ngnix request metrics then passing request uuid as a label makes the metrics highly cardinal. On Th

Re: [prometheus-users] Re: (Alertmanager) Ignore instance label to prevent same alert multiple times

2020-06-04 Thread Murali Krishna Kanagala
I guess your scrape config has some static labels like environment: prod, region: central, etc. so add another static label with cluster name. Then do a label replacement on your alert query to replace the instance value to your cluster name. Then you have 1 instance name for the entire cluster. If

Re: [prometheus-users] Windows Exporter SetUp

2020-06-04 Thread Murali Krishna Kanagala
9 > > I have just downloaded the wmi_exporter on clients , what else need to be > done on clients windows servers? to get the Metrics displayed on Grafana, > Also I have Linux and Windows as client servers on Grafana. > > > > On Thursday, June 4, 2020 at 12:43:44 PM UTC-5, Murali

Re: [prometheus-users] Prometheus Querying

2020-06-04 Thread Murali Krishna Kanagala
Take a look these queries https://www.robustperception.io/which-are-my-biggest-metrics Based on these you can write queries to get the data you are looking for. On Thu, Jun 4, 2020, 5:44 AM Nishant Ketu wrote: > Hi All , > > I need few values about my Prometheus server for calculation the RAM >

Re: [prometheus-users] Windows Exporter SetUp

2020-06-04 Thread Murali Krishna Kanagala
Wmi exporter is the windows equivalent of node exporter. You need to run it as a service and add that machine to scrape config. Never heard of windows exporter. On Thu, Jun 4, 2020, 11:24 AM Freddy Mack wrote: > I have Download the wmi_exporter on localhost and clients- but I see the > blogs tha

Re: [prometheus-users] Converting value from process exporter to percent in grafana for better representation

2020-05-28 Thread Murali Krishna Kanagala
The metrics you have shared above are actual memory usage in bytes. To calculate the percentage we need to the memory capacity of the server you are monitoring. There should be another metric giving that value. Then you can calculate the percent usage by 100* (memory_used_bytes/Total memory bytes)

Re: [prometheus-users] Alert Manager Wrong Resolution

2020-05-26 Thread Murali Krishna Kanagala
The alert should resolve if the query does not give any results. On Tue, May 26, 2020, 11:22 AM Arnav Bose wrote: > I have this scenario with prometheus-alert manager alerting: > > > alert condition: K_status != 3 > > > The alert gets triggered fine and when the value changes back to 3, the > a

Re: [prometheus-users] Re: I want to monitor ssl cert expiry of my urls using ssl exporter.Need help

2020-05-15 Thread Murali Krishna Kanagala
dff-0a56189f48fa%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- Regards, Murali Krishna Kanagala -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails

Re: [prometheus-users] Re: I want to monitor ssl cert expiry of my urls using ssl exporter.Need help

2020-05-13 Thread Murali Krishna Kanagala
PM UTC+5:30, Murali Krishna Kanagala > wrote: >> >> What Dashboard are you using? Do you have a link from Grafana website? >> If you use the metric *probe_ssl_earliest_cert_expiry *you should >> multiply it with 1000 and use -MM-DD HH:mm:ss (on any date specific >>

Re: [prometheus-users] Re: I want to monitor ssl cert expiry of my urls using ssl exporter.Need help

2020-05-13 Thread Murali Krishna Kanagala
491847f%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> >>> >>> -- >>> Julius Volz >>> PromLabs - promlabs.com >>> >> -- > You received this message because you are subscribed to the

Re: [prometheus-users] Prometheus Adapter

2020-04-29 Thread Murali Krishna Kanagala
sgid/prometheus-users/efde4ac0-1376-4504-bc15-504d16a11956%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/efde4ac0-1376-4504-bc15-504d16a11956%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- Regards, Murali Krishna Kanagala -- Yo

Re: [prometheus-users] Blackbox exporter probe_ssl_earliest_cert_expiry giving wrong value in grafana

2020-04-23 Thread Murali Krishna Kanagala
iest_cert_expiry{instance=~"$target"} - time() > > > [image: probe.PNG] > > > Regards, > James > > On Thursday, April 23, 2020 at 6:05:24 PM UTC+8, Murali Krishna Kanagala > wrote: >> >> Hi James, >> >> Multiply the expiry value you

Re: [prometheus-users] Blackbox exporter probe_ssl_earliest_cert_expiry giving wrong value in grafana

2020-04-23 Thread Murali Krishna Kanagala
Hi James, Multiply the expiry value you receive from probe_ssl_earliest_cert_expiry with thousand when you use it on Grafana. We can directly apply the units on Grafana without subtracting the time() from it. Thanks, Murali Kanagala On Thu, Apr 23, 2020, 4:26 AM James Eduard Andaya < jameseduard

Re: [prometheus-users] Prometheus dashboard exposed over alb-ingress-controller is not loading fully

2020-04-22 Thread Murali Krishna Kanagala
nothing... Still same... >> >> Thanks again for helping me dude... >> >> On Tuesday, April 21, 2020 at 9:10:30 PM UTC-5, Murali Krishna Kanagala >> wrote: >>> >>> You can add the arguments here. >>> >>> >>> https://github

Re: [prometheus-users] Prometheus dashboard exposed over alb-ingress-controller is not loading fully

2020-04-21 Thread Murali Krishna Kanagala
the page is loading (I just want to reiterate here). Its just the > additional *.css & *.js files are not loading... > > Thanks > > On Tuesday, April 21, 2020 at 6:29:33 PM UTC-5, Murali Krishna Kanagala > wrote: >> >> Try setting the --web.external-url in the container

Re: [prometheus-users] Prometheus dashboard exposed over alb-ingress-controller is not loading fully

2020-04-21 Thread Murali Krishna Kanagala
Try setting the --web.external-url in the container arguments. On Tue, Apr 21, 2020, 5:47 PM vikram yerneni wrote: > Fellas, I am using Helm/Stable/Prometheus Server for my Metrics datasource > and the Prometheus Server Dashboard is exposed using alb-ingress controller > in AWS. Somehow the pro

Re: [prometheus-users] inhibit all severity but how??

2020-04-16 Thread Murali Krishna Kanagala
Hi Danny, Did you try using a reflex for the target_match? And also adding another inhibitor rule with target_match as critical? On Wed, Apr 15, 2020, 7:17 AM Danny de Waard wrote: > I have a alrtmanger setup and prometheus setup that are working good. > I cab inhibit severity 'warning' when m

Re: [prometheus-users] Prometheus failing after few hours of restart.

2020-04-16 Thread Murali Krishna Kanagala
Looks like a memory issue. How much RAM you have on the server? https://github.com/prometheus/prometheus/issues/4945 On Wed, Apr 15, 2020, 11:09 AM Ben Kochie wrote: > That looks like an incomplete stack trace. Can you get the full stack > trace from your syslog? > > There's not enough informat

Re: [prometheus-users] Re: Pattern(s) for having a counter in short-lived applications (batch jobs)

2020-04-15 Thread Murali Krishna Kanagala
Thanks Brian! On Wed, Apr 15, 2020, 5:49 AM Brian Candler wrote: > pushgateway doesn't aggregate counters, but statsd_exporter does. This > sounds more like what you need. > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubsc

Re: [prometheus-users] Pattern(s) for having a counter in short-lived applications (batch jobs)

2020-04-14 Thread Murali Krishna Kanagala
Hi Sean, Prometheus does not keep track of the counters. Metric source or the exporter does it. As the metric goes from 0 to N on every run of the cronjob you can create a synthetic metric in Prometheus that tracks the increase. If the labels are persistent you can track the increase and probably

Re: [prometheus-users] Grafana support for Prometheus Webhook Alerts?

2020-04-07 Thread Murali Krishna Kanagala
It can take alertmanager as a data source. On Sun, Apr 5, 2020, 11:34 AM Steve wrote: > Hi > Does Grafana support HTTP/HTTPS webhook alerts originating from the > Prometheus alert manager(s)? > > Thanks > > -Steve > > -- > You received this message because you are subscribed to the Google Groups

Re: [prometheus-users] Re: Start recording rule until N mins of data is collected

2020-04-07 Thread Murali Krishna Kanagala
Every time the data is scrapped Prometheus does that calculation and record the custom metric. 5min here is a sliding window with latest metric's time stamp as the curren time and the previous one (if it exists within last 5min) as the previous metric. If there is no metric in last 5 min you'll not

Re: [prometheus-users] Removed target sends resolved alert

2020-03-28 Thread Murali Krishna Kanagala
If the target is removed from the scrape config the alert will resolve anyway. If you want to persist the alert set send_resolved to false in alertmanager. On Thu, Mar 26, 2020, 3:16 PM Stuart Clark wrote: > On 26/03/2020 19:09, Amanda H. L. de Andrade wrote: > > Hi, > > > > If a remove a target

Re: [prometheus-users] Postman Exporter

2020-03-28 Thread Murali Krishna Kanagala
Thank you Sebastian! On Fri, Mar 27, 2020, 5:51 PM Sebastian Ebling wrote: > Hi, > > I created an exporter that runs Postman collections and returns metrics on > those runs. > > You can find the project here: > https://github.com/hudeldudel/postman-exporter > > I would appreciate feedback. > > T

Re: [prometheus-users] Filter alertmanager notification like slack using templates

2020-03-24 Thread Murali Krishna Kanagala
. On Tue, Mar 24, 2020, 11:13 PM Rahul Hada wrote: > I am able to get json data, but was looking for more appropriate template > so that could send only required parameters to the api. Any suggestions for > that would be appreciated. > > On Wed, 25 Mar, 2020, 9:30 AM Murali K

Re: [prometheus-users] Filter alertmanager notification like slack using templates

2020-03-24 Thread Murali Krishna Kanagala
To send an alert to a web hook there is no need for a template. The receiver api receivs the alerts in as json payload. https://prometheus.io/docs/alerting/configuration/#webhook_config On Tue, Mar 24, 2020, 12:46 AM Rahul Hada wrote: > Hi All, > We are using templates to refine the parameters

Re: [prometheus-users] Working of Textfile Collector.

2020-03-08 Thread Murali Krishna Kanagala
Good to know that vmstat is for a different purpose. On Sun, Mar 8, 2020, 1:52 AM Brian Candler wrote: > The vmstat collector isn't anything to do with VMWare: it returns stats > about the Linux virtual memory subsystem from /proc/vmstat. > > $ curl /metrics | grep -i vmstat > ... > node_scr

Re: [prometheus-users] Working of Textfile Collector.

2020-03-07 Thread Murali Krishna Kanagala
Also, did you try using the vmstat command line flag on node exporter? On Sat, Mar 7, 2020, 12:31 PM Murali Krishna Kanagala < kanagalamur...@gmail.com> wrote: > Just drop a text file with data in Prometheus format and pass it as a > command line arguments on your node exporte

Re: [prometheus-users] Working of Textfile Collector.

2020-03-07 Thread Murali Krishna Kanagala
Just drop a text file with data in Prometheus format and pass it as a command line arguments on your node exporter. https://prometheus.io/docs/concepts/data_model/ https://github.com/prometheus/node_exporter/blob/master/README.md On Sat, Mar 7, 2020, 11:01 AM Yagyansh S. Kumar wrote: > Hi. I a

Re: [prometheus-users] How to expose a custom metrics from a pod/service in Kubernetes ?

2020-03-03 Thread Murali Krishna Kanagala
Sorry for the wrong link https://medium.com/@zhimin.wen/custom-prometheus-metrics-for-apps-running-in-kubernetes-498d69ada7aa On Tue, Mar 3, 2020, 10:04 PM Murali Krishna Kanagala < kanagalamur...@gmail.com> wrote: > I think this page can help you to write the sd config for kubern

Re: [prometheus-users] How to expose a custom metrics from a pod/service in Kubernetes ?

2020-03-03 Thread Murali Krishna Kanagala
I think this page can help you to write the sd config for kubernetes pods. https://github.com/nlighten/tomcat_exporter For scraping only master pod of a given service it should have a special label so that Prometheus can discover it. An alternative approach would be a stateful set where every pod

Re: [prometheus-users] Checking if NFS is hanged or not using node_exporter.

2020-03-03 Thread Murali Krishna Kanagala
to all my 2500+ servers. > > On Tuesday, March 3, 2020 at 8:46:27 PM UTC+5:30, Murali Krishna Kanagala > wrote: >> >> I would write a small shell script that tries to write to the nfs mount >> path and writes the status to a file which can be read by the text file >

Re: [prometheus-users] Checking if NFS is hanged or not using node_exporter.

2020-03-03 Thread Murali Krishna Kanagala
it out once. > > > On Tuesday, March 3, 2020 at 8:35:03 PM UTC+5:30, Murali Krishna Kanagala > wrote: >> >> Try enabling the nfs options in the node exporter config. It will spit >> out some metrics about the nfs status. >> >> Also look at the disk IO metric

Re: [prometheus-users] Checking if NFS is hanged or not using node_exporter.

2020-03-03 Thread Murali Krishna Kanagala
Try enabling the nfs options in the node exporter config. It will spit out some metrics about the nfs status. Also look at the disk IO metrics from node exporter and if you see no activity which indicates the nfs is not doing anything. On Tue, Mar 3, 2020, 7:10 AM Yagyansh S. Kumar wrote: > I w

Re: [prometheus-users] Re: Alert for high-frequency changes of a metric?

2020-03-02 Thread Murali Krishna Kanagala
Check if the absent function helps you. https://prometheus.io/docs/prometheus/latest/querying/functions/#absent On Mon, Mar 2, 2020, 9:49 AM Moses Moore wrote: > >Did you try something like "changes(foo[30m]) > 10" ? > > That would alert if the value changed 10 times in the last 30 minutes.

Re: [prometheus-users] 'insert metric at cursor' is showing all metrics

2020-03-02 Thread Murali Krishna Kanagala
Did you try a restart of Prometheus service? On Mon, Mar 2, 2020, 1:04 AM Ankita Khot wrote: > In Prometheus UI within the dropdown that says 'insert metric at cursor' > it shows the previous scraped metrics as well. How to remove those? I would > like it to show only the metrics which i am keep

Re: [prometheus-users] Read access log as metrics

2020-03-02 Thread Murali Krishna Kanagala
Prometheus is not designed to handle logs. Try Loki, or rsyslog along with grok exporter to translate logs into metrics. On Mon, Mar 2, 2020, 5:03 AM Stuart Clark wrote: > On 2020-03-02 11:00, Julian Ade Putra wrote: > > Hello Experts, > > > > I have access log that contains API list and status

Re: [prometheus-users] Prometheus metrics to a url

2020-03-02 Thread Murali Krishna Kanagala
Use remote write option. That remote service should be able to parse the metrics and forward them wherever you want them to be. On Mon, Mar 2, 2020, 4:42 AM adi garg wrote: > Hello experts, > > Is there a way to send Prometheus metrics to a URL, where there can be an > exporter to send metrics t

Re: [prometheus-users] Invalid token error while scraping a target

2020-03-01 Thread Murali Krishna Kanagala
Metric format is not as as exp6by Prometheus. Promtool can validate the metric format. Follow the steps in this link https://www.robustperception.io/invalid-is-not-a-valid-start-token-and-other-scrape-errors On Sun, Mar 1, 2020, 8:57 PM Tasleem Halai wrote: > I am using Prometheus : 2.8.1 > > pr

Re: [prometheus-users] Re: Process Monitoring

2020-03-01 Thread Murali Krishna Kanagala
Node exporter can monitor all the processes running on a machine. Just pass the commandline arguments as explained in the below link. https://github.com/prometheus/node_exporter On Sun, Mar 1, 2020, 3:21 AM Arun Raj wrote: > Can someone help me with details that which metrics provide me to mo

Re: [prometheus-users]

2020-03-01 Thread Murali Krishna Kanagala
It's not a good idea to keep prometheus.yml and the rest of the scrape config files in the same path. Create a sub directory under /etc/prometheus to keep the scrape config files. On Sun, Mar 1, 2020, 5:22 PM rs vas wrote: > Hi, I would like to include all the rule files in prometheus.yml > con

Re: [prometheus-users] Re: monitoring Openshift Cluster with External Promethus

2020-02-14 Thread Murali Krishna Kanagala
You can scrape the Prometheus on the openshift using federation configuration on the external Prometheus. With this approach you don't have to persist the metrics on the openshift cluster(no need of persist storage) Just expose the Prometheus on the open shift cluster using nodeport or something l