It is used all over the place. AWS managed prometheus is basically just Cortex[0], which is what Grafana Mimir[1] is based on. The solution has been around for quite a while.
[0]: https://github.com/cortexproject/cortex [1]: https://github.com/grafana/mimir On Fri, Nov 4, 2022 at 4:40 PM Navid Shakibapour < navid.shakibap...@gmail.com> wrote: > > You would need additional tools like corosync or keepalived to achieve > this. > Thanks for pointing me to those tools. I did a quick search and didn't see > much info on setting up Prometheus HA with corosync/keepalived. Except that > it looks like AWS managed Prometheus uses that approach [1]: > *When you set up deduplication, Amazon Managed Service for Prometheus > makes one Prometheus instance a leader replica and ingests data samples > only from that replica. If the leader replica stops sending data samples to > Amazon Managed Service for Prometheus for 30 seconds, Amazon Managed > Service for Prometheus automatically makes another Prometheus instance a > leader replica and ingests data from the new leader.* > > I'm a bit surprised that this approach isn't used more. > > Re using multiple replicas, my concern is the cost in bigger clusters. The > cost will add up to having multiple replicas per shard knowing the data > would be deduplicated downstream. > On Friday, November 4, 2022 at 10:58:37 AM UTC-4 sayf.eddi...@gmail.com > wrote: > >> I faced this issue in the design phase too, found this >> https://github.com/beorn-/highlander but it looks a bit dead now... >> else you can push duplicated metrics to thanos (while setting a unique >> label for every prometheus instance) and then configure thanos querier to >> deduplicate when serving metrics >> On Friday, November 4, 2022 at 3:40:29 PM UTC+1 Julien Pivotto wrote: >> >>> You would need additional tools like corosync or keepalived to achieve >>> this. In prometheus terms, all the prometheis servers are active and >>> deduplication is done at the remote write end if needed. >>> >>> On 04 Nov 07:24, Navid Shakibapour wrote: >>> > Is there a way to set up multiple Prometheus servers in an >>> active/passive >>> > way? Meaning only 1 server scrapes targets at a time and other ones >>> only >>> > scrape if the active one doesn't "work" after x seconds/minutes. >>> > >>> > On Friday, November 4, 2022 at 3:00:15 AM UTC-4 sup...@gmail.com >>> wrote: >>> > >>> > > There is no option without using a single instance. If your 3rd >>> party >>> > > solution does not handle HA, you can not use HA. >>> > > >>> > > On Fri, Nov 4, 2022 at 7:57 AM Navid Shakibapour < >>> navid.sh...@gmail.com> >>> > > wrote: >>> > > >>> > >> We are evaluating using Prometheus agent mode in a highly available >>> > >> manner to forward metrics to a 3rd party remote storage using >>> remote write >>> > >> API. >>> > >> We need to have multiple shards and multiple replicas per shard. >>> Our 3rd >>> > >> party remote storage does not handle deduplication on their side. >>> > >> >>> > >> Could someone point me to different options for metrics >>> deduplication on >>> > >> ingestion? >>> > >> >>> > >> >>> > >> -- >>> > >> You received this message because you are subscribed to the Google >>> Groups >>> > >> "Prometheus Users" group. >>> > >> To unsubscribe from this group and stop receiving emails from it, >>> send an >>> > >> email to prometheus-use...@googlegroups.com. >>> > >> To view this discussion on the web visit >>> > >> >>> https://groups.google.com/d/msgid/prometheus-users/1b274d3d-77a9-4de3-9b63-61a533b84140n%40googlegroups.com >>> > >> < >>> https://groups.google.com/d/msgid/prometheus-users/1b274d3d-77a9-4de3-9b63-61a533b84140n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> >>> > >> . >>> > >> >>> > > >>> > >>> > -- >>> > You received this message because you are subscribed to the Google >>> Groups "Prometheus Users" group. >>> > To unsubscribe from this group and stop receiving emails from it, send >>> an email to prometheus-use...@googlegroups.com. >>> > To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-users/91c8b0d4-67e2-40d3-8314-bd58409cf937n%40googlegroups.com. >>> >>> >>> >>> -- >>> Julien Pivotto >>> @roidelapluie >>> >> -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to prometheus-users+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/01470303-ceac-4b6a-b837-463fd5440db5n%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/01470303-ceac-4b6a-b837-463fd5440db5n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmqV3RY5iAVgcsngOSApVQJqA2MtwkAe%2BnTX1pi0VCs%3DrQ%40mail.gmail.com.
