Looks to me like a problem at the receiver end (i.e. the middleware Elastic agent, or Elasticsearch itself), i.e. that side has stopped accepting data.
Try looking at logs of these to determine why they are no longer accepting data. On Wednesday, 18 January 2023 at 11:29:05 UTC christia...@gmail.com wrote: > Hello guys. > > I am scraping some metrics wich ar ethen shipped of to an Elastic agent to > be ingested into Elasticsearch. All seems fine to start with, but after > some time, metrics stop comming in, and the prometheus logs shows a lot of > entries like this: > > ts=2023-01-18T10:51:46.125Z caller=dedupe.go:112 component=remote > level=warn remote_name=010ca8 url= > http://agent-svc.observability.svc.cluster.local:9201/write msg="Failed > to send batch, retrying" err="Post \" > http://agent-svc.observability.svc.cluster.local:9201/write\": context > deadline exceeded" > > ts=2023-01-18T10:51:20.364Z caller=dedupe.go:112 component=remote > level=debug remote_name=010ca8 url= > http://agent-svc.observability.svc.cluster.local:9201/write msg="Not > downsharding due to being too far behind" > > I am guessing that Prometheus is trying to tell me something, but i just > dont know what. > > Checking the TSDB status on the prom UI it tells me that Number of series > is 8439 wich does not sound like a lot. > Any help would be very appreciated. > > Best regards > Christian Oelsner > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/16e03bd3-eefb-4c68-87d4-a3d181315ab5n%40googlegroups.com.
