Re: [prometheus-users] install Prometheus with Prometheus Operator

2021-06-10 Thread Stuart Clark
On 10/06/2021 09:51, nina guo wrote: So whether we need to do some configuration for alert manager or it is enabled by default? Take a look at the documentation: https://github.com/prometheus/alertmanager#high-availability -- Stuart Clark -- You received this message because you are subscri

Re: [prometheus-users] install Prometheus with Prometheus Operator

2021-06-10 Thread nina guo
So whether we need to do some configuration for alert manager or it is enabled by default? On Thursday, June 10, 2021 at 3:22:37 PM UTC+8 nina guo wrote: > Thank you very much Stuart. : ) > We will improve our solution. > > On Thursday, June 10, 2021 at 2:57:04 PM UTC+8 Stuart Clark wrote: > >> O

Re: [prometheus-users] install Prometheus with Prometheus Operator

2021-06-10 Thread nina guo
Thank you very much Stuart. : ) We will improve our solution. On Thursday, June 10, 2021 at 2:57:04 PM UTC+8 Stuart Clark wrote: > On 10/06/2021 07:53, nina guo wrote: > > > > So for alert manager, the alerts will be sent to every member of an > > Alertmanager cluster before firing. Is it correc

Re: [prometheus-users] install Prometheus with Prometheus Operator

2021-06-09 Thread Stuart Clark
On 10/06/2021 07:53, nina guo wrote: So for alert manager, the alerts will be sent to every member of an Alertmanager cluster before firing. Is it correct? But when the alerts are fired, it will only have 1 notification eg. inĀ  destination Email or Slack? That is correct. -- Stuart Clark -

Re: [prometheus-users] install Prometheus with Prometheus Operator

2021-06-09 Thread nina guo
So for alert manager, the alerts will be sent to every member of an Alertmanager cluster before firing. Is it correct? But when the alerts are fired, it will only have 1 notification eg. in destination Email or Slack? On Thursday, June 10, 2021 at 2:09:59 PM UTC+8 Stuart Clark wrote: > On 10/0

Re: [prometheus-users] install Prometheus with Prometheus Operator

2021-06-09 Thread Stuart Clark
On 10/06/2021 06:36, nina guo wrote: Thank you Stuart. For my understanding, the concept of HA is there are 2 instances - master and slave. Master is running, and slave is standby. If master breakes down, slave will take over. But now for Prometheus, the 2 promethues instanaces are running at t

Re: [prometheus-users] install Prometheus with Prometheus Operator

2021-06-09 Thread nina guo
Thank you Stuart. For my understanding, the concept of HA is there are 2 instances - master and slave. Master is running, and slave is standby. If master breakes down, slave will take over. But now for Prometheus, the 2 promethues instanaces are running at the same time with scraping the same me

Re: [prometheus-users] install Prometheus with Prometheus Operator

2021-06-09 Thread Stuart Clark
On 09/06/2021 09:46, nina guo wrote: Thank you very much Stuart : ) For implementing "split your list of targets across multiple servers", currently in our env, the mulitple jobs are sharing the same configmap. So in order to split the target list, should I create separateĀ  configmap for each

Re: [prometheus-users] install Prometheus with Prometheus Operator

2021-06-09 Thread nina guo
Thank you very much Stuart : ) For implementing "split your list of targets across multiple servers", currently in our env, the mulitple jobs are sharing the same configmap. So in order to split the target list, should I create separate configmap for each instance? I'm not sure if it is the cor

Re: [prometheus-users] install Prometheus with Prometheus Operator

2021-06-08 Thread Stuart Clark
On 09/06/2021 07:16, nina guo wrote: Thank you very much. May I ask if there is a way to make multiple Prometheus instances to scrape different targets? Compared the 2 solution, scraping same targerts vs scraping different targets, which is more better? On Monday, June 7, 2021 at 6:08:47 P

Re: [prometheus-users] install Prometheus with Prometheus Operator

2021-06-08 Thread nina guo
Thank you very much. May I ask if there is a way to make multiple Prometheus instances to scrape different targets? Compared the 2 solution, scraping same targerts vs scraping different targets, which is more better? On Monday, June 7, 2021 at 6:08:47 PM UTC+8 Stuart Clark wrote: > When doing

Re: [prometheus-users] install Prometheus with Prometheus Operator

2021-06-07 Thread Stuart Clark
When doing autoscaling (not specifically with Prometheus but everything) you need to ensure that you don't have too many changes happening at once, otherwise you might start rejecting requests (if all instances are restarting at the same time). This would generally be done via things like pod d

Re: [prometheus-users] install Prometheus with Prometheus Operator

2021-06-07 Thread Stuart Clark
On 2021-06-07 10:36, nina guo wrote: Autoscaling based on the prometheus metrics load. For example, if during the recent minutes, the metrics is more than 50m, a new prometheus pod will be started. So you are wanting to autoscale Prometheus itself? That's not normally needed, as the load shou

Re: [prometheus-users] install Prometheus with Prometheus Operator

2021-06-07 Thread nina guo
Autoscaling based on the prometheus metrics load. For example, if during the recent minutes, the metrics is more than 50m, a new prometheus pod will be started. On Monday, June 7, 2021 at 5:34:14 PM UTC+8 Stuart Clark wrote: > On 2021-06-07 10:20, nina guo wrote: > > I still have a question that

Re: [prometheus-users] install Prometheus with Prometheus Operator

2021-06-07 Thread Stuart Clark
On 2021-06-07 10:20, nina guo wrote: I still have a question that - any conflict between autoscaling and High Availability? Because for my understanding, if currently the solution is with autoscaling, the multiple pods may scrape different metrics, then this siutation will not have HA. but if no

Re: [prometheus-users] install Prometheus with Prometheus Operator

2021-06-07 Thread nina guo
I came out an idea, the following 2 pods are scraping the same metrics. Pod 1 - current load 50m - during the recent 1m(not sure if the duration makes sense), the load is 55m. Pod 2 - current load 50m - during the recent 1m(not sure if the duration makes sense), the load is 55m. The rule is if

Re: [prometheus-users] install Prometheus with Prometheus Operator

2021-06-07 Thread nina guo
I still have a question that - any conflict between autoscaling and High Availability? Because for my understanding, if currently the solution is with autoscaling, the multiple pods may scrape different metrics, then this siutation will not have HA. but if now the solution is with HA, the muli

Re: [prometheus-users] install Prometheus with Prometheus Operator

2021-06-07 Thread nina guo
Many thanks for your detailed answers Julius. On Friday, June 4, 2021 at 6:12:02 PM UTC+8 juliu...@promlabs.com wrote: > Hi Nina, > > if you run multiple HA replicas of Prometheus and one of them becomes > unavailable for some reason and you query that broken replica, the queries > will indeed

Re: [prometheus-users] install Prometheus with Prometheus Operator

2021-06-04 Thread Julius Volz
Hi Nina, if you run multiple HA replicas of Prometheus and one of them becomes unavailable for some reason and you query that broken replica, the queries will indeed fail. You could either load-balance (with dead backend detection) between the replicas to avoid this, or use something like Thanos (

Re: [prometheus-users] install Prometheus with Prometheus Operator

2021-06-04 Thread nina guo
And also have a question that if Prometheus has any autoscaling solution? On Friday, June 4, 2021 at 3:24:58 PM UTC+8 nina guo wrote: > Thank you very much. > If I deploy multiple Prometheus Pods, and mount separate volumes to each > Pod: > 1. If one of the k8s nodes goes down, is there any chan

Re: [prometheus-users] install Prometheus with Prometheus Operator

2021-06-04 Thread nina guo
Thank you very much. If I deploy multiple Prometheus Pods, and mount separate volumes to each Pod: 1. If one of the k8s nodes goes down, is there any chance the access is currently on the crashed nodes, then the query will be failed? 2. If multiple Pods are running in k8s cluster, is there any da

Re: [prometheus-users] install Prometheus with Prometheus Operator

2021-06-03 Thread Julius Volz
Hi Nina, No, by default, the Prometheus Operator uses an emptyDir for the Prometheus storage, which gets lost when the pod is rescheduled. This explains how to add persistent volumes: https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/user-guides/storage.md Rega

[prometheus-users] install Prometheus with Prometheus Operator

2021-06-03 Thread nina guo
Hi, If using Prometheus Operator to install in k8s cluster, the data pv will be created automatically or not? -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email