Hi Ben, Does latest version of Cortex /Thanos supports the alerting with multiple shards of prometheus ? Thanos Ruler wasn't ready for production to evalute the expression across the prometheus instances .. Do we have any docuemnet or blog about this ?
Thanks, Rajesh On Tue, May 26, 2020 at 11:37 AM Ben Kochie <[email protected]> wrote: > This is probably a case where you would want to look into Thanos or Cortex > to provide a larger aggregation layer on top of multiple Prometheus servers. > > On Sun, May 17, 2020 at 11:53 AM Rajesh Reddy Nachireddi < > [email protected]> wrote: > >> Hi, >> >> Basically, we have large networking setup with 10k devices. we are >> hitting 1M metrics every second from 20 % of devices itself, so we have 5 >> prom instances and one global proemtheus which uses remote read to handle >> alert rule evaluations and thanos querier for visualisation on grafana. >> >> We have segregated devices with specific device ip ranges to each >> Prometheus instances. >> >> So, we have one aggregator which is using remote read from all the >> individual prom instances through remote read >> >> 1. will the remote read cause an issue w.r.t loading the large time >> series over wire every 1 min ? >> 2. Is it CPU or memory intensive ? >> >> What is best design strategy to handle these scale and alerting across >> the devices or metrics ? >> >> Regards, >> >> Rajesh >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/CAEyhnp%2BfG8YvciR4-30D%2BzsDzg_kF%2BKkJUavdbyGCxoz-97q_A%40mail.gmail.com >> <https://groups.google.com/d/msgid/prometheus-users/CAEyhnp%2BfG8YvciR4-30D%2BzsDzg_kF%2BKkJUavdbyGCxoz-97q_A%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAEyhnpJt4QoMxzcMPvMa8qyDra8LLR9Je4nJqPZek8jSGYPbwA%40mail.gmail.com.
