> grafana has the datasource, which points to the k8s service, which is a "loadbalancer", which has two endpoints; one for each prometheus pod.
Then each query from grafana goes to *either* prometheus A *or* prometheus B. If both prometheus servers are scraping the same endpoints then they'll have roughly the same data in them, so you'll see roughly the same graphs. However, using a loadbalancer in this way is not recommended, as it's not deterministic which prometheus server your query will hit. I suggest you look at promxy <https://github.com/jacksontj/promxy>, which is designed for this use case and can do useful things like filling any gaps from data in prometheus A with data from prometheus B, or vice versa. > take it that my two prometheus do not sync timeseries; That is correct. They'll both independently scrape and store. I say they'll have "roughly" the same data because they won't scrape at the same points in time. > but with grafana i see no gaps in my graphs. Firstly, restarting prometheus is pretty quick. Secondly, PromQL queries will look back in time up to 5 minutes (by default) to find a previous data point. Up to that interval, your grafana graphs will look flat but without gaps. On Thursday, 23 February 2023 at 22:13:46 UTC+8 Mario Cornaccini wrote: > hi, > > i just updated our kube-prometheus custom ressource, which manages a > deployment of two prometheus pods, which flawlessly upgraded the pods one > after each other..(deployment strategy) > > but with grafana i see no gaps in my graphs. > grafana has the datasource, which points to the k8s service, which is a > "loadbalancer", which has two endpoints; one for each prometheus pod. > (maybe thats a question for the grafana mailinglist then) > > i take it that my two prometheus do not sync timeseries; > (as opposed to alertmanager which does gossip) > > das anybody know why there seem to be no gaps in prometheus scraping > timeseries ? > although i do see a gap when i look at each pods' cpu-usage. of course, > it's because they got restarted. > > just curious.. > cheers, > fil > > > > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/ed619ddd-f48e-4fc3-83b9-c9d85f231706n%40googlegroups.com.
