[prometheus-users] Setup Prometheus Cluster

I'm new on Prometheus : ) Here comes the following question: - To deploy a Prometheus cluster, is it better to use Prometheus Operator? Currently I used sperate yaml files(prometheus, alertmanager, nodeexporter). - There is a cluster with 3 nodes. prometheus on 3 PODs separately node expor

Re: [prometheus-users] Setup Prometheus Cluster

Because I would like to realize high availability. If one is done, the other can take over. On Friday, April 2, 2021 at 4:35:50 PM UTC+8 sup...@gmail.com wrote: > Why do you think you need a Prometheus cluster? What problem are you > trying to solve? > > On Fri, Apr 2, 2021 at 1

Re: [prometheus-users] Setup Prometheus Cluster

Because I would like to realize high availability. If one is down, the other can take over. On Friday, April 2, 2021 at 4:35:50 PM UTC+8 sup...@gmail.com wrote: > Why do you think you need a Prometheus cluster? What problem are you > trying to solve? > > On Fri, Apr 2, 2021 at 1

Re: [prometheus-users] Setup Prometheus Cluster

thub.com/prometheus-operator/kube-prometheus > > Or you can deploy it with Helm using the kube-prometheus-stack chart. > > > https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack > > On Fri, Apr 2, 2021 at 10:39 AM nina guo wrote: > >&g

[prometheus-users] how can I access Prometheus

I deployed a Prometheus cluster as below with 2 Prometheus PODs. prometheus-deployment-1 (master node) prometheus-deployment-2 The service is also deployed as below: NAME TYPECLUSTER-IP EXTERNAL-IP PORT(S) AGE alertmanager ClusterIP 1.2.3.4

[prometheus-users] backend storage for Prometheus

Hi We are struggling on which backend storage is best to use for Prometheus metrics remote storing, block storage or NFS? Any experience can be shared?f -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and sto

[prometheus-users] node exporter configuration

Hi, A Prometheus cluster has been setup with alert manager and node exporter. The corresponding PODs are running. Now I'm going to install node exporter on monitoring targets(virtual machine), can I use binary file to install? Is there any issue for the communication between node export POD on

[prometheus-users] how to take Configmap changes effectvie?

I made some changes on Prometheus Configmap, the only way is to restart the POD. But if we restart, the data will be lost totally. So is it other way to take Confimap changes effective? -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To un

Re: [prometheus-users] how to take Configmap changes effectvie?

Thank you. Could you give me an update? On Thursday, April 15, 2021 at 3:31:26 PM UTC+8 Stuart Clark wrote: > On 15/04/2021 08:28, nina guo wrote: > > I made some changes on Prometheus Configmap, the only way is to > > restart the POD. But if we restart, the data will be l

Re: [prometheus-users] how to take Configmap changes effectvie?

Thank you. Could you please give me an example? On Thursday, April 15, 2021 at 3:31:26 PM UTC+8 Stuart Clark wrote: > On 15/04/2021 08:28, nina guo wrote: > > I made some changes on Prometheus Configmap, the only way is to > > restart the POD. But if we restart, the data will

[prometheus-users] how to get the name of target/server on Prometheus GUI

Now I 'm using openstack service discovery to discover all the targets. On Prometheus GUI, it only shows Endpoint, so is there a way to know the server name? -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group a

Re: [prometheus-users] how to take Configmap changes effectvie?

Thank you very much. I will try. On Thursday, April 15, 2021 at 3:31:26 PM UTC+8 Stuart Clark wrote: > On 15/04/2021 08:28, nina guo wrote: > > I made some changes on Prometheus Configmap, the only way is to > > restart the POD. But if we restart, the data will be lost totally

Re: [prometheus-users] how to take Configmap changes effectvie?

/2021 08:28, nina guo wrote: > > I made some changes on Prometheus Configmap, the only way is to > > restart the POD. But if we restart, the data will be lost totally. So > > is it other way to take Confimap changes effective? -- > > If you have it enabled there i

[prometheus-users] how to evaluate if the current solution is suitable to afford the monitoring load

hi, A k8s cluster is setup for monitoring. But we would like to confirm if the current solution is OK to afford the monitoring load. Whether the CPU , MEM and local disk size settings are ok? Is there a load test between the number of series and CPU/MEM/local disk size? For example, if we will ha

[prometheus-users] OCI runtime create failed: container_linux.go:370: starting container process caused: exec: "/bin/prometheus": stat /bin/prometheus: no such file or directory: unknown

Trying to deploy prometheus in K8S with a image from private registry. But receive the above error. Anyone can help? -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an

[prometheus-users] show hostname rather than internal ip in alert message

Hi, Many I ask how to show hostname rather than internal ip in alert message. With the following settings, $labels.hostname still displays the internal ip. *alert rules definition:* - alert: HighCpuLoad expr: 100 - (avg by(instance) (irate(node_cpu_sec

[prometheus-users] load generator tool on Prometheus

Hi, Is there a load generator tool on Prometheus in k8s cluster recommend? We are going to generate metrics and series to forecast CPU/MEM consumption. -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop

[prometheus-users] is there a tool to generate sounds of metrics automatically

hi may I ask if there is a tool for metrics generation on Prometheus? -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegr

[prometheus-users] kubernetes sevice discovery

I'm going to use a tool Avalanche to generate metrics automatically. GitHub - open-fresh/avalanche: Prometheus/OpenMetrics endpoint series generator for load testing. prometheus configmap updated as below, but the pods are unable to be discovered curren

[prometheus-users] Re: kubernetes sevice discovery

Prometheus pods and Avalanche pods are reside in a same k8s cluster. On Monday, May 17, 2021 at 4:17:41 PM UTC+8 nina guo wrote: > I'm going to use a tool Avalanche to generate metrics automatically. > > GitHub - open-fresh/avalanche: Prometheus/OpenMetrics endpoint series > g

[prometheus-users] by default, prometheus pod is running inside k8s cluster with which user?

hi, may I ask prometheus pod is running inside k8s cluster with which user? how to check it? -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use

[prometheus-users] Where does prometheus tsdb stored

hi, Where does prometheus tsdb stored? Is it stored in memory? Yes if, when will it be flushed to local storage? -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an emai

[prometheus-users] level=error ts=2021-05-28T09:23:52.326Z caller=main.go:894 err="opening storage failed: lock DB directory: resource temporarily unavailable"

Hi, Got this error when I'm trying to start the Prometheus POD. The backend storage is NFS. -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-user

[prometheus-users] Re: level=error ts=2021-05-28T09:23:52.326Z caller=main.go:894 err="opening storage failed: lock DB directory: resource temporarily unavailable"

any one can help on this issue? I tried to deploy prometheus in k8s cluster with multiple replicas, and then met the issue. On Friday, May 28, 2021 at 5:27:11 PM UTC+8 nina guo wrote: > Hi, > > Got this error when I'm trying to start the Prometheus POD. > > The ba

Re: [prometheus-users] Re: level=error ts=2021-05-28T09:23:52.326Z caller=main.go:894 err="opening storage failed: lock DB directory: resource temporarily unavailable"

May I ask why multiple Prometheus cannot write to the same directory? On Friday, May 28, 2021 at 8:19:54 PM UTC+8 Julien Pivotto wrote: > On 28 May 03:40, nina guo wrote: > > any one can help on this issue? > > I tried to deploy prometheus in k8s cluster with multiple replicas

Re: [prometheus-users] Re: level=error ts=2021-05-28T09:23:52.326Z caller=main.go:894 err="opening storage failed: lock DB directory: resource temporarily unavailable"

Thank you very much Stuart. On Tuesday, June 1, 2021 at 6:14:45 PM UTC+8 Stuart Clark wrote: > On 31/05/2021 04:31, nina guo wrote: > > May I ask why multiple Prometheus cannot write to the same directory? > > Different instances of Prometheus are not aware of each other

Re: [prometheus-users] Re: level=error ts=2021-05-28T09:23:52.326Z caller=main.go:894 err="opening storage failed: lock DB directory: resource temporarily unavailable"

If Prometheus deploys in k8s with multiple Pods, the Prometheus Pods are running independently, am I right? On Wednesday, June 2, 2021 at 1:28:31 PM UTC+8 nina guo wrote: > Thank you very much Stuart. > > On Tuesday, June 1, 2021 at 6:14:45 PM UTC+8 Stuart Clark wrote: > >> O

Re: [prometheus-users] Re: level=error ts=2021-05-28T09:23:52.326Z caller=main.go:894 err="opening storage failed: lock DB directory: resource temporarily unavailable"

. Currently the data which was in memeroy will be lost. It will cause data inconsistency. Because the other running Pod probably already have written the data to persistent volume. On Wednesday, June 2, 2021 at 4:39:16 PM UTC+8 Stuart Clark wrote: > On 02/06/2021 09:22, nina guo wrote: >

Re: [prometheus-users] Re: level=error ts=2021-05-28T09:23:52.326Z caller=main.go:894 err="opening storage failed: lock DB directory: resource temporarily unavailable"

Can we solve this issue with load balancer? On Wednesday, June 2, 2021 at 6:01:38 PM UTC+8 nina guo wrote: > So the better solution would be mount another storage rather than NFS > separately to each Pod. > For example, 2 Prometheus Pods are running with 2 separate volumes, if one >

[prometheus-users] install Prometheus with Prometheus Operator

Hi, If using Prometheus Operator to install in k8s cluster, the data pv will be created automatically or not? -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email

Re: [prometheus-users] install Prometheus with Prometheus Operator

> This explains how to add persistent volumes: > https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/user-guides/storage.md > > Regards, > Julius > > On Thu, Jun 3, 2021 at 9:08 AM nina guo wrote: > >> Hi, >> >> If using P

Re: [prometheus-users] install Prometheus with Prometheus Operator

And also have a question that if Prometheus has any autoscaling solution? On Friday, June 4, 2021 at 3:24:58 PM UTC+8 nina guo wrote: > Thank you very much. > If I deploy multiple Prometheus Pods, and mount separate volumes to each > Pod: > 1. If one of the k8s nodes goes down,

Re: [prometheus-users] install Prometheus with Prometheus Operator

o do autoscaling based on > Prometheus metrics. > > Regards, > Julius > > On Fri, Jun 4, 2021 at 9:25 AM nina guo wrote: > >> Thank you very much. >> If I deploy multiple Prometheus Pods, and mount separate volumes to each >> Pod: >> 1. If one of the

Re: [prometheus-users] install Prometheus with Prometheus Operator

mulitpes pods are exactly scraping the same metrics, if then start autoscaling, it will break HA. On Monday, June 7, 2021 at 4:51:56 PM UTC+8 nina guo wrote: > Many thanks for your detailed answers Julius. > > On Friday, June 4, 2021 at 6:12:02 PM UTC+8 juliu...@promlabs.com wrote: >

Re: [prometheus-users] install Prometheus with Prometheus Operator

the duration makes sense), the load is 55m. -> a new Pod 2-1 will be created to afford the extra 5m load So if autoscaling happened on the 2 Pods at the same time feasible? On Monday, June 7, 2021 at 5:20:46 PM UTC+8 nina guo wrote: > I still have a question that - any conflict between au

Re: [prometheus-users] install Prometheus with Prometheus Operator

Autoscaling based on the prometheus metrics load. For example, if during the recent minutes, the metrics is more than 50m, a new prometheus pod will be started. On Monday, June 7, 2021 at 5:34:14 PM UTC+8 Stuart Clark wrote: > On 2021-06-07 10:20, nina guo wrote: > > I still have a

Re: [prometheus-users] install Prometheus with Prometheus Operator

one via things like pod distuption budgets. For a > pair of Prometheus servers I'd not want more than one change at once. For > other systems I might go as far as N-1 changes at once. > > On Monday, 7 June 2021 at 11:05:20 UTC+1 Stuart Clark wrote: > >> On 2021-06-07 10:

Re: [prometheus-users] install Prometheus with Prometheus Operator

re if it is the correct way. On Wednesday, June 9, 2021 at 2:30:30 PM UTC+8 Stuart Clark wrote: > On 09/06/2021 07:16, nina guo wrote: > > Thank you very much. > > May I ask if there is a way to make multiple Prometheus instances to > scrape different targets? > > Compared

Re: [prometheus-users] install Prometheus with Prometheus Operator

1 PM UTC+8 Stuart Clark wrote: > On 09/06/2021 09:46, nina guo wrote: > > Thank you very much Stuart : ) > > For implementing "split your list of targets across multiple servers", > > currently in our env, the mulitple jobs are sharing the same > > configmap.

Re: [prometheus-users] install Prometheus with Prometheus Operator

10/06/2021 06:36, nina guo wrote: > > Thank you Stuart. > > For my understanding, the concept of HA is there are 2 instances - > > master and slave. Master is running, and slave is standby. If master > > breakes down, slave will take over. > > But now for Prometheus

Re: [prometheus-users] install Prometheus with Prometheus Operator

Thank you very much Stuart. : ) We will improve our solution. On Thursday, June 10, 2021 at 2:57:04 PM UTC+8 Stuart Clark wrote: > On 10/06/2021 07:53, nina guo wrote: > > > > So for alert manager, the alerts will be sent to every member of an > > Alertmanager cluster

Re: [prometheus-users] install Prometheus with Prometheus Operator

So whether we need to do some configuration for alert manager or it is enabled by default? On Thursday, June 10, 2021 at 3:22:37 PM UTC+8 nina guo wrote: > Thank you very much Stuart. : ) > We will improve our solution. > > On Thursday, June 10, 2021 at 2:57:04 PM UTC+8 Stuart

[prometheus-users] alertmanager HA

Hi guys, I'm implementing alertmanager HA. alertmanager0: - containers: - name: alertmanager image: quay.io/prometheus/alertmanager:latest args: - "--config.file=/etc/alertmanager/config.yml" - "--storage.path=/aler

Re: [prometheus-users] alertmanager HA

May I ask what's the difference once I change to Statefulset? On Wednesday, June 23, 2021 at 2:42:56 PM UTC+8 sayf.eddi...@gmail.com wrote: > Hello, you can make use of Kubernetes Statefulset instead of deployment in > this case > > On Wed, Jun 23, 2021 at 8:20 AM nina guo wr

Re: [prometheus-users] alertmanager HA

stname every > time they die and come back, > ie: a statefulset named foobar with two pods will always have two pods > named foobar-0 and foobar-1 > combined with a headless service, the pods are reachable at > foobar-{0/1}...svc.cluster.local > > On Wed, Jun 23, 2021

Re: [prometheus-users] alertmanager HA

- alertmanager1-0.alertmanager1.monitoring.svc.cluster.local:9093 On Wednesday, June 23, 2021 at 3:34:02 PM UTC+8 nina guo wrote: > Thank you, great, I resolved the issue once I change to StatefulSet. > > > > > On Wednesday, June 23, 2021 at 3:21:10 PM UTC+8 sayf.eddi...@gmail.com

[prometheus-users] Prometheus Federation

Hi, May Prometheus Federation have the ability to store the data for a long period? Can the architecture be like this? Victoria metrics -> Prometheus Federation -> multi Prometheus instance? I'm thinking Victoria metrics may do the similiar thing with Prometheus Federation. -- You received

Re: [prometheus-users] Prometheus Federation

Prometheus > community / natively supported options. > > On Thu, Jun 24, 2021 at 11:27 AM nina guo wrote: > >> Hi, >> >> May Prometheus Federation have the ability to store the data for a long >> period? >> >> Can the architecture be like this? >&

[prometheus-users] Prometheus configmap

Hi guys, If statefulset strategy can share only 1 configmap for multiple replicas? -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr

[prometheus-users] prometheus statefulset confimap

Hi guy, If multiple statefulset replicas can share only 1 configmap? --- volumeMounts: # configutaion volumes on the container - name: prometheus-config-volume mountPath: /etc/prometheus - name:

[prometheus-users] alertmanager HA

Hi guys, I have a question regarding alertmanager HA. If I want to implement it in kubernetes cluster, I should deploy 2 alertmanager deployments or deploy 2 alertmanager with only 1 deployment? Actually currently I deployed with 2 seperate alertmanager deployments, each one is Statefulset, li

[prometheus-users] Re: alertmanager HA

what I mean "deploy 2 alertmanager with only 1 deployment" is deploying 2 alertmanager replicas but only 1 deployment yaml file. On Monday, July 26, 2021 at 1:07:43 PM UTC+8 nina guo wrote: > Hi guys, > > I have a question regarding alertmanager HA. > > If I want to i

[prometheus-users] how to judge CPU/mem allocation for a single Prometheus

hi guys, how to decide how many CPU/MEM is required for a single Prometheus? Can a Prometheus instance with 32GB monitor 2 servers? -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving email

[prometheus-users] how to implement alertmanager HA in k8s

Hi guys, How to implement alertmanager HA in k8s? Can someone give some help on this? -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsub

[prometheus-users] Show DNS name on Grafana Dashboard

Hi, Is there a way to show DNS name for a server on Grafana Dashboard? For example: (function_name) usrname@hostname:~> virtualip list Configured Virtual IPs: --- function_name Virtual IPs State: -- function_name: (info) Virtual IP is up and active on hos

[prometheus-users] Re: Show DNS name on Grafana Dashboard

C+8 nina guo wrote: > Hi, > > Is there a way to show DNS name for a server on Grafana Dashboard? > > For example: > (function_name) usrname@hostname:~> virtualip list > Configured Virtual IPs: > --- > function_name > > Virtual IPs State: >

[prometheus-users] Re: Show DNS name on Grafana Dashboard

Wednesday, 8 September 2021 at 09:19:29 UTC+1 ninag...@gmail.com wrote: > >> A correction: >> >> (function_name) usrname@hostname:~> virtualip list >> Configured Virtual IPs: >> --- >> function_name on eth0 >> >> Virtual IPs State: >> --

[prometheus-users] Re: Show DNS name on Grafana Dashboard

With node exporter, whether we can obtain the info of virtual IP? On Wednesday, September 8, 2021 at 5:00:42 PM UTC+8 nina guo wrote: > I'm using node exporter to get the metrics. I dont have the virtual IP > infos, I want to get the value first and display it on Grafana. > You k

[prometheus-users] Re: Show DNS name on Grafana Dashboard

One more question is with node_uname_info, we get the nodename, this nodename is the hostname of the server by default? On Wednesday, September 8, 2021 at 5:18:38 PM UTC+8 nina guo wrote: > With node exporter, whether we can obtain the info of virtual IP? > > On Wednesday, September 8,

[prometheus-users] Re: Show DNS name on Grafana Dashboard

Thank you very much for your detailed reply. We want to get this output "cat /etc/sysconfig/network/virtualip". So which collectors may contain this value? On Wednesday, September 8, 2021 at 6:26:45 PM UTC+8 Brian Candler wrote: > On Wednesday, 8 September 2021 at 10:00:42 UTC+1 ninag...@gmail.

[prometheus-users] Re: Show DNS name on Grafana Dashboard

th0:AAA* Link encap:Ethernet HWaddr FA: inet addr: Bcast: Mask: UP BROADCAST RUNNING MULTICAST Metric: On Thursday, September 9, 2021 at 10:53:46 AM UTC+8 nina guo wrote: > Thank you very much for your detailed reply. > > We want to get this output "cat /etc/sysco

[prometheus-users] Re: Show DNS name on Grafana Dashboard

So we only want to get a specific static value to be shown on Grafana, it may not be a generanl metric. On Thursday, September 9, 2021 at 12:02:01 PM UTC+8 nina guo wrote: > I checked again that the info we require can be shown with ifconfig > command. We want to get "eth0:AAA"

[prometheus-users] Re: Show DNS name on Grafana Dashboard

in your dashboard. I don't know how you'd do > that, and you'd have to ask elsewhere, because this is a mailing list for > prometheus, not grafana. (Grafana has its own discussion forum). > > On Thursday, 9 September 2021 at 05:30:00 UTC+1 ninag...@gmail.com wrote: &

[prometheus-users] Re: Show DNS name on Grafana Dashboard

be shown on Grafana, >>> it may not be a generanl metric. >>> >>> Everything in prometheus is a metric. Static values have to be labels >>> on metrics. Again, see how node_uname_info does this. >>> >>> Of course, since Grafana is a separate piece

[prometheus-users] Re: Show DNS name on Grafana Dashboard

For my understanding, the script is required to be copied to target systems. On Friday, September 10, 2021 at 2:23:59 PM UTC+8 nina guo wrote: > #!/bin/bash > # > # Description: Expose metrics from virtualip. > # > # > OUTPUT=$(cat "/etc/sysconfig/network/virtualip&

[prometheus-users] Re: Show DNS name on Grafana Dashboard

gt; <https://www.robustperception.io/network-interface-metrics-from-the-node-exporter> >>>>> >>>>> - but none giving layer 3 info (IP addresses). Either this is available >>>>> in >>>>> an optional node_exporter collector which i

[prometheus-users] Re: Show DNS name on Grafana Dashboard

label_A="label_a_value", then whether the value of the label can be got with node_uname_info? On Friday, September 10, 2021 at 4:03:37 PM UTC+8 nina guo wrote: > Thank you very much Brian > On Friday, September 10, 2021 at 3:49:07 PM UTC+8 Brian Candler wrote: > >> Th

[prometheus-users] Re: Show DNS name on Grafana Dashboard

t;> [image: Screenshot 2021-09-13 134538.png] >> >> eg.: >> If adding a new label label_A="label_a_value", then whether the value of >> the label can be got with node_uname_info? >> >> On Friday, September 10, 2021 at 4:03:37 PM UTC+8 nina guo wrote:

[prometheus-users] Re: Show DNS name on Grafana Dashboard

r/virtualip.prom *2. with the query (mymetric) * on(instance) group_left(hostname,domainname) node_uname_info to get the value on Grafana. * On Monday, September 13, 2021 at 3:26:37 PM UTC+8 nina guo wrote: > Got it. Thank you very much Brian. > > On Monday, September 13, 2021 at 3:03:24 PM

[prometheus-users] Re: Show DNS name on Grafana Dashboard

Could you please to review if the step 1 and step 2 are right? On Wednesday, September 15, 2021 at 6:11:12 PM UTC+8 nina guo wrote: > Sorry Brian disturb you again. > We still faced problems regarding this issue. > > With file discovery we defined as below: > Target:[] > Labels

[prometheus-users] Re: Show DNS name on Grafana Dashboard

The label "dnshost" is totally different pn Prometheus UI with the label which we want to be shown on Grafana. Is this understanding correct? On Wednesday, September 15, 2021 at 6:11:58 PM UTC+8 nina guo wrote: > Could you please to review if the step 1 and step 2 are right? >

[prometheus-users] Mutiple PromQLs for each expr in alert rules

Hello guys, Is there a way to report an alert like this: expr: if containerCPUusage>80 and serverCPUusge>80 Then trigger the alert. -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails f

[prometheus-users] Re: Mutiple PromQLs for each expr in alert rules

Thank you for your reply. Let me clarify what we are going to do more. We have the following alert rules for a container. But we still want to include the CPU usage of the corresponding host which host this container. That is: - when CPU usage for container is larger than 85%, and CPU usage f

Re: [prometheus-users] Re: Mutiple PromQLs for each expr in alert rules

May I ask how to check all the available labels for this? Can we define new labels? For instances, we use file discovery to get the monitored target list. On Monday, December 6, 2021 at 4:29:27 PM UTC+8 Stuart Clark wrote: > On 06/12/2021 08:19, nina guo wrote: > > Thank you for y

Re: [prometheus-users] Re: Mutiple PromQLs for each expr in alert rules

Thank you. in container_cpu_usage_seconds_total, there are following labels: instance name kubernetes_io_arch hostname job On Monday, December 6, 2021 at 5:54:08 PM UTC+8 Stuart Clark wrote: > On 06/12/2021 08:43, nina guo wrote: > > May I ask how to check all the available labels

Re: [prometheus-users] Re: Mutiple PromQLs for each expr in alert rules

in container_cpu_usage_seconds_total, there are following labels: instance name kubernetes_io_arch hostname job cpu (the value of this label is "total") On Tuesday, December 7, 2021 at 10:15:15 AM UTC+8 nina guo wrote: > Thank you. > > in container_cpu_usage_seconds_total,

[prometheus-users] Prometheus Talk with other web UI

Hi, Is there a way for Prometheus to talk with other web UI to get the real-time status of targets? -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometh

[prometheus-users] Re: Prometheus Talk with other web UI

Any one can give us some help on this? On Tuesday, December 7, 2021 at 1:28:38 PM UTC+8 nina guo wrote: > Hi, > > Is there a way for Prometheus to talk with other web UI to get the > real-time status of targets? > -- You received this message because you are subscribed to th

Re: [prometheus-users] Re: Mutiple PromQLs for each expr in alert rules

Could you provide some help on this topic? On Tuesday, December 7, 2021 at 10:17:59 AM UTC+8 nina guo wrote: > in container_cpu_usage_seconds_total, there are following labels: > instance > name > kubernetes_io_arch > hostname > job > cpu (the value of this label is &quo

[prometheus-users] Re: Prometheus Talk with other web UI

The web application is the application which manages all the targets. We want to check with the web application to get the real status of the target. Then if the target is with a "monitored" status, we expect there will be an alert triggered. So can Prometheus send this kind of check/command t

[prometheus-users] Re: Prometheus Talk with other web UI

Thank you for your detailed reply. Yes, currently we use file discovery. But one important thing maybe I misssed is that the status of the target might be changed. That is to say, now the status of the target is the status which we want to monitor, but after several minis, the target is down, w

[prometheus-users] Re: Prometheus Talk with other web UI

Thank you very much Brian. Our file discovery is happened every 5mins to get the latest target list. We are worry that during this 5 mins if there is one of the target state changes from monitored state to the state which we don't want to monitor, actually we don't want to receive the alert fro

[prometheus-users] Re: Prometheus Talk with other web UI

theus have this kind of checking with inventory management system? On Friday, December 10, 2021 at 9:56:03 AM UTC+8 nina guo wrote: > Thank you very much Brian. > > Our file discovery is happened every 5mins to get the latest target list. > We are worry that during this 5 mins if th

[prometheus-users] Re: Prometheus Talk with other web UI

Thank you Brian. Your suggestion is really good. May I have one more question? For example, if you read the inventory via Netbox every 5mins, let's assume, if during the 5mins, there is a state of the target is removed from the inventory, at this time point Prometheus still triggers an fake a

[prometheus-users] Re: Prometheus Talk with other web UI

Correct the words * there is a target is removed from the inventory On Monday, December 13, 2021 at 4:31:09 PM UTC+8 nina guo wrote: > Thank you Brian. > > Your suggestion is really good. May I have one more question? > > For example, if you read the inventory via Netbox eve

[prometheus-users] Re: Prometheus Talk with other web UI

Thank you Brian. Let's assume the refresh interval is only 5 secs, but during 5 secs the state might be changed. For exmaple, the target is changed to maintence(which is not the state we want to monitor) and start a reboot, but at this time point, prometheus doesn't know the state change, a fa

[prometheus-users] Re: Prometheus Talk with other web UI

if some one changes the status to in maintenance and performs a reboot ... prometheus will notice this status update after 3 min and during this time it could generates fake alerts On Monday, December 13, 2021 at 5:10:51 PM UTC+8 nina guo wrote: > Thank you Brian. > > Let's assu

[prometheus-users] Re: Prometheus Talk with other web UI

if some one changes the status to in maintenance and performs a reboot ... prometheus will notice this status update after 5 secs and during this time it could generates fake alerts On Monday, December 13, 2021 at 5:10:51 PM UTC+8 nina guo wrote: > Thank you Brian. > > Let's assu

[prometheus-users] Re: Prometheus Talk with other web UI

Thank you very much Brian. On Monday, December 13, 2021 at 7:04:33 PM UTC+8 Brian Candler wrote: > I think there's not much point continuing this discussion. Prometheus > does not have exactly what you keep asking for, which is "to check" > something externally before sending an alert to alert

[prometheus-users] How can we know which metrics does bind exporter expose

Hi guys, We would like to use bind exporter to monitor DNS server with bind exporter. But which metrics does bind exporter expose exactly? -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving em

Re: [prometheus-users] How can we know which metrics does bind exporter expose

i, Dec 31, 2021 at 1:54 PM nina guo wrote: > >> Hi guys, >> >> We would like to use bind exporter to monitor DNS server with bind >> exporter. >> >> But which metrics does bind exporter expose exactly? >> >> -- >> You received this messa

[prometheus-users] bind exporter installation

Hi guys, We are configuring bind exporter in k8s cluster. Here comes several questions: - should bind exporter and bind server on a same server? Can we install bind exporter on a different server? if we install on a different server, which parameters should be specified? Whether the following p

[prometheus-users] Alerts cannot be forwarded to email address, alway with Active state

Hi, kubectl logs alertmanager-1 -n monitoring level=debug ts=2022-03-21T06:41:59.362Z caller=dispatch.go:473 component=dispatcher aggrGroup="{}:{alertname=\"KubernetesStatefulsetDown\", instance=\"kube-state-metrics.kube-system.svc.cluster.local\", job=\"kube-state-metrics\"}" msg=flushin

[prometheus-users] Re: Alerts cannot be forwarded to email address, alway with Active state

group_by: [instance, alertname, job] group_wait: 45s group_interval: 5m repeat_interval: 24h receivers: - name: email-me email_configs: - to: alert@address send_resolved: true On Monday, March 21, 2022 at 2:57:43 PM UTC+8 nina guo wrote: >

[prometheus-users] How to implement Alertmanager High Availability in K8s cluster

Hi Understand that there is a guide Alertmanager | Prometheus for High Availability. But if we want to implement High Availability in k8s cluster, can we deploy 2 deployment/statefulset replicas to k8s cluster? -- You re

[prometheus-users] alertmanager HA error "no such host"

We are deploying alertmanager HA solution, but got the below error. level=debug ts=2022-03-24T09:35:50.731Z caller=cluster.go:329 component=cluster memberlist="2022/03/24 09:35:50 [WARN] memberlist: Failed to resolve alertmanager-1.alertmanager.monitoring.svc.cluster.local:9094: lookup alertman

[prometheus-users] black box exporter monitoring SSH and PING

Hi guys, We are using black box exporter to monitor ssh and ping. For ssh, (we monitor the port 22) if we stop sshd service, actually the service will be auto-recovered, but black box exporter detect the recover behavior after about 5mins. For ping, we use icmp module to monitor system ping, w

[prometheus-users] Re: black box exporter monitoring SSH and PING

*blackbox exporter config:* icmp: prober: icmp icmp: preferred_ip_protocol: "ip4" tcp: prober: tcp timeout: 5s tcp: preferred_ip_protocol: "ip4" *Prometheus scrape config:* global: scrape_interval: 60s evaluation_interval: 60s

[prometheus-users] Prometheus Memory Usage

Hi, I'm using this formula to calculator the memory usage. How much RAM does Prometheus 2.x need for cardinality and ingestion? – Robust Perception | Prometheus Monitoring Experts I have severa

[prometheus-users] Alert manager cannot be connect

Hi, I used the following files to deploy alertmanager, but it cannot be communicated. apiVersion: apps/v1 kind: Deployment metadata: name: alertmanager1 namespace: monitoring spec: replicas: 1 selector: matchLabels: app: alertmanager1 template: metadata: name: alert

  1   2   >