I have a job on the Prometheus which gathers metrics from 4 netdata master servers. Here is the scenario I had: - on netdata master number 1, I gather metrics of about 200 slaves - For some reason, I decided to move 12 slaves (a1,a2,a3,a4,b1,b2,b3,b4,c1,c2,c3,c4) from the first netdata master to the second netdata master - Now I only see metrics from 8 servers on the Prometheus server a1,a2,a3,a4,b1,b2,b3,b4) coming from the second master - I check the job status in the targets page and I see all 4 masters are up and metrics are gathered successfully - Here is the URL which Prometheus uses to read the metrics from the netdata master number 2: http://172.16.76.152:19999/api/v1/allmetrics?format=prometheus_all_hosts - I grep the downloaded file with hosts metrics for the c1,c2,c3,c4 hosts and I see netdata is sending all the metrics relevant to these slaves - But when I search for the metric in the Graph page, I don't see any results:
[image: Screenshot from 2020-05-10 20-58-27.png] all the servers' time is synced and are correct. here is the output of systemctl status prometheus: May 10 19:35:07 devops-mon-01 systemd[1]: Reloading Prometheus. May 10 19:35:07 devops-mon-01 prometheus[6076]: level=info ts=2020-05-10T15:05:07.407Z caller=main.go:734 msg="Loading configuration file" filename=/e tc/prometheus/prometheus.yml May 10 19:35:07 devops-mon-01 prometheus[6076]: level=info ts=2020-05-10T15:05:07.416Z caller=main.go:762 msg="Completed loading of configuration file " filename=/etc/prometheus/prometheus.yml May 10 19:35:07 devops-mon-01 systemd[1]: Reloaded Prometheus. May 10 19:53:22 devops-mon-01 prometheus[6076]: level=error ts=2020-05-10T15:23:22.621Z caller=api.go:1347 component=web msg="error writing response" bytesWritten=0 err="write tcp 172.16.77.50:9090->172.16.76.168:56778: write: broken pipe" May 10 20:25:53 devops-mon-01 prometheus[6076]: level=error ts=2020-05-10T15:55:53.058Z caller=api.go:1347 component=web msg="error writing response" bytesWritten=0 err="write tcp 172.16.77.50:9090->172.16.76.168:41728: write: broken pipe" 172.16.77.50 is our Prometheus server and 172.16.76.168 is our grafana server so I think the last error is not related to my problem -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/156d8c36-c1de-4ca3-8b2a-2cfbcb5895fc%40googlegroups.com.
