Hello guys. I am scraping some metrics wich ar ethen shipped of to an Elastic agent to be ingested into Elasticsearch. All seems fine to start with, but after some time, metrics stop comming in, and the prometheus logs shows a lot of entries like this:
ts=2023-01-18T10:51:46.125Z caller=dedupe.go:112 component=remote level=warn remote_name=010ca8 url=http://agent-svc.observability.svc.cluster.local:9201/write msg="Failed to send batch, retrying" err="Post \"http://agent-svc.observability.svc.cluster.local:9201/write\": context deadline exceeded" ts=2023-01-18T10:51:20.364Z caller=dedupe.go:112 component=remote level=debug remote_name=010ca8 url=http://agent-svc.observability.svc.cluster.local:9201/write msg="Not downsharding due to being too far behind" I am guessing that Prometheus is trying to tell me something, but i just dont know what. Checking the TSDB status on the prom UI it tells me that Number of series is 8439 wich does not sound like a lot. Any help would be very appreciated. Best regards Christian Oelsner -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/776fa548-b38e-47dc-bd54-b08cef7be93cn%40googlegroups.com.
