I want to consult what is the right approach to break down a "big" prometheus (that scrap too many targets\metrics) into small prometheus instances(each scrap a range of targets) but still get global view via Grafana?
Which approach below is better? Approach1) to use prometheus instance that remote_read <https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_read> from all the small prometheus instances. The prometheus remote_read can be installed as k8s deployment with loadbalancer above to allow scale out depend on network traffic. So this remote_read will be used as datasource in Grafana (so all dashboards and queries will still use single datasource). Approach2) to use Thanos Querier <https://github.com/thanos-io/thanos/blob/master/docs/components/query.md#querier-use-cases-why-do-i-need-this-component> that read from all the small prometheus instances. The Thanos querier is stateless by design so could be deploy in k8s as deployment with loadbalancer. The Thanos querier will be used as datasource in Grafana. I believe that Thanos querier component has small footprint (compare to prometheus remote_read). Thanks Disclaimer The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful. This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast Ltd, an innovator in Software as a Service (SaaS) for business. Providing a safer and more useful place for your human generated data. Specializing in; Security, archiving and compliance. To find out more visit the Mimecast website. -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/8f6ca977-6f61-4276-995d-7ba30c71b3d9%40googlegroups.com.
