Hi Ben We will have multiple K8s clusters (see future), various workloads on mostly production networks, happy that each of these run a local prometheus store and local Grafana dashboard for those users, but then additionally have a couple of hundred linux and windows VM's.
I want to get sidecar deployed on all prometheus deployments, be that the K8S hosted or a central hosted Prometheus that looks after my Linux and Windows hosts / DC. currently we don't have those K8S clusters yet, but we do have all the linux and windows... so want to deploy a stack now, Linux installed Prometheus server with Thanos sidecar. Dedicated Grafana server + Thanus Query, environment can then be expanded, with all future prometheus servers also having a side car, all talking to a single Thanus Store with a shared S3 bucket. Once we have a management K8S cluster, the linux and windows monitoring prometheus can be migrated. G On Thu, Aug 27, 2020 at 5:32 PM Ben Kochie <[email protected]> wrote: > On Thu, Aug 27, 2020 at 4:59 PM George <[email protected]> wrote: > >> Stretching my question a bit here... >> >> Most of the examples of deploying thanos assume a Prometheus on K8S >> (which would be great option down the road once we have a management K8s >> cluster), as I don't want to run my management environment on the K8s >> cluster that I also want to monitor. >> > > There is nothing wrong with running monitoring inside the Kubernetes > cluster. This is very common and normal. > > >> >> Don't see allot for deploying Thanos on a Linux host for Prometheus, >> although there is the one example of simply building the Thanos executable >> from the Go code. >> ... or downloading the tar ball, looking inside this is a single >> executable. >> >> ... this example also does not mention if you are building the Thanos >> sidecar, Thanos Query or the Thanos Store component... assuming it builds >> the one executable that can do all 3... and is then configured. What am I >> missing here. >> > > Lots of people have built Chef, Ansible, automation for doing non-K8s > deploys. We built our own for Chef, but are planning to move away from that > and move it to Kubernetes. > > >> >> G >> >> On Thu, Aug 27, 2020 at 4:17 PM George <[email protected]> wrote: >> >>> Doing some more googling as we're speaking. >>> >>> Looks like each Prometheus server gets the Thanos side car, and then >>> Thanos Query is deployed onto my Grafana server/s, >>> >>> Grafana talks to Thanos Query (locally), thats configured to know about >>> all my Thanos side cars + Thanos Store for aging out older metrics to S3 >>> buckets. >>> >>> G >>> >>> On Thu, Aug 27, 2020 at 4:11 PM Ben Kochie <[email protected]> wrote: >>> >>>> There is no "Thanos Server". There are a number of components that are >>>> deployed in different places. I suggest you review the official Thanos >>>> docs. >>>> >>>> You only need Thanos Query deployed where Grafana is. Thanos Query >>>> talks directly to the sidecars and optionally store servers. >>>> >>>> On Thu, Aug 27, 2020 at 4:07 PM George <[email protected]> wrote: >>>> >>>>> Hi Ben >>>>> >>>>> So would you recommend we deploy one Thanos server/deployment per DC. >>>>> to which the local Grafana server then connects. >>>>> >>>>> That Thanos then talks to all local and remote (2nd DC) prometheus >>>>> servers. >>>>> >>>>> G >>>>> >>>>> On Thu, Aug 27, 2020 at 4:04 PM Ben Kochie <[email protected]> wrote: >>>>> >>>>>> This sounds very much like a standard multi-DC / multi-subnet >>>>>> Prometheus+Thanos setup. >>>>>> >>>>>> Basically what you have is a Thanos Query front-end, that is >>>>>> configured to connect to all of the sidecars. It can query data from all >>>>>> Prometheus servers simultaneously. >>>>>> >>>>>> Grafana is configured to talk to Thanos Query, so you can get all >>>>>> data in one view. >>>>>> >>>>>> On Thu, Aug 27, 2020 at 3:44 PM George <[email protected]> wrote: >>>>>> >>>>>>> Hi all >>>>>>> >>>>>>> Question... in our current management tool set we have a proxy >>>>>>> server on each of the subnets that is responsible for collecting the >>>>>>> performance metrics of all servers on that subnet, >>>>>>> >>>>>>> those values are then send to a large management server where >>>>>>> everything is submitted. >>>>>>> >>>>>>> in the Prometheus architecture the server itself collects the data >>>>>>> and stores it in the local TSB. >>>>>>> >>>>>>> We also have 2 DC's, thinking is to deploy prometheus servers in >>>>>>> each of the DC's, responsible for the local servers. >>>>>>> >>>>>>> My thinking is to deploy a prometheus server on each of our subnets, >>>>>>> and then a Thanos side car on the prometheus server. >>>>>>> >>>>>>> In this configuration... does the various Thanos deployed get told >>>>>>> about each other, so that they can display all the metrics of all the >>>>>>> prometheus servers as one large deployment ? >>>>>>> >>>>>>> For Grafana, at the moment it's pointed at my prometheus server ip >>>>>>> address and a port, how does this change, or does it not, and Thanos >>>>>>> inserts itself in the middle of that path. >>>>>>> >>>>>>> please comment. >>>>>>> >>>>>>> G >>>>>>> >>>>>>> -- >>>>>>> You have the obligation to inform one honestly of the risk, and as a >>>>>>> person >>>>>>> you are committed to educate yourself to the total risk in any >>>>>>> activity! >>>>>>> >>>>>>> Once informed & totally aware of the risk, >>>>>>> every fool has the right to kill or injure themselves as they see >>>>>>> fit! >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "Prometheus Users" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/prometheus-users/CALw5Ujv24MAYfXFojRXd%2BSDd8LS%3D68GbsPQb4d-NmE3pnWLv_A%40mail.gmail.com >>>>>>> <https://groups.google.com/d/msgid/prometheus-users/CALw5Ujv24MAYfXFojRXd%2BSDd8LS%3D68GbsPQb4d-NmE3pnWLv_A%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> >>>>>> >>>>> >>>>> -- >>>>> You have the obligation to inform one honestly of the risk, and as a >>>>> person >>>>> you are committed to educate yourself to the total risk in any >>>>> activity! >>>>> >>>>> Once informed & totally aware of the risk, >>>>> every fool has the right to kill or injure themselves as they see fit! >>>>> >>>> >>> >>> -- >>> You have the obligation to inform one honestly of the risk, and as a >>> person >>> you are committed to educate yourself to the total risk in any activity! >>> >>> Once informed & totally aware of the risk, >>> every fool has the right to kill or injure themselves as they see fit! >>> >> >> >> -- >> You have the obligation to inform one honestly of the risk, and as a >> person >> you are committed to educate yourself to the total risk in any activity! >> >> Once informed & totally aware of the risk, >> every fool has the right to kill or injure themselves as they see fit! >> > -- You have the obligation to inform one honestly of the risk, and as a person you are committed to educate yourself to the total risk in any activity! Once informed & totally aware of the risk, every fool has the right to kill or injure themselves as they see fit! -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CALw5UjvgARfzZ40kkt3kROdXmYDEF8gDqXF11yG5XkBLj0FZ_Q%40mail.gmail.com.
