[prometheus-users] Re: Remote write dying after some time.
Hi Brian, Sorry about the latency. It was indeed an issue with the receiver, which is sorted now by the team that manages the Elastic agent. Best regards Christian Oelsner torsdag den 19. januar 2023 kl. 07.05.17 UTC+1 skrev Christian Oelsner: > Hi Brian, > Thanks for your input, it as always apreaciated. > I will try to have the observability team enable some debug logging on the > agent, and see if i can spot something. > > Regards > > Christian Oelsner > > onsdag den 18. januar 2023 kl. 14.14.23 UTC+1 skrev Brian Candler: > >> Looks to me like a problem at the receiver end (i.e. the middleware >> Elastic agent, or Elasticsearch itself), i.e. that side has stopped >> accepting data. >> >> Try looking at logs of these to determine why they are no longer >> accepting data. >> >> On Wednesday, 18 January 2023 at 11:29:05 UTC christia...@gmail.com >> wrote: >> >>> Hello guys. >>> >>> I am scraping some metrics wich ar ethen shipped of to an Elastic agent >>> to be ingested into Elasticsearch. All seems fine to start with, but after >>> some time, metrics stop comming in, and the prometheus logs shows a lot of >>> entries like this: >>> >>> ts=2023-01-18T10:51:46.125Z caller=dedupe.go:112 component=remote >>> level=warn remote_name=010ca8 url= >>> http://agent-svc.observability.svc.cluster.local:9201/write msg="Failed >>> to send batch, retrying" err="Post \" >>> http://agent-svc.observability.svc.cluster.local:9201/write\": context >>> deadline exceeded" >>> >>> ts=2023-01-18T10:51:20.364Z caller=dedupe.go:112 component=remote >>> level=debug remote_name=010ca8 url= >>> http://agent-svc.observability.svc.cluster.local:9201/write msg="Not >>> downsharding due to being too far behind" >>> >>> I am guessing that Prometheus is trying to tell me something, but i just >>> dont know what. >>> >>> Checking the TSDB status on the prom UI it tells me that Number of >>> series is 8439 wich does not sound like a lot. >>> Any help would be very appreciated. >>> >>> Best regards >>> Christian Oelsner >>> >> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/5c17b156-87e4-48d2-a5b7-0084305897ean%40googlegroups.com.
[prometheus-users] Re: Remote write dying after some time.
Hi Brian, Thanks for your input, it as always apreaciated. I will try to have the observability team enable some debug logging on the agent, and see if i can spot something. Regards Christian Oelsner onsdag den 18. januar 2023 kl. 14.14.23 UTC+1 skrev Brian Candler: > Looks to me like a problem at the receiver end (i.e. the middleware > Elastic agent, or Elasticsearch itself), i.e. that side has stopped > accepting data. > > Try looking at logs of these to determine why they are no longer accepting > data. > > On Wednesday, 18 January 2023 at 11:29:05 UTC christia...@gmail.com wrote: > >> Hello guys. >> >> I am scraping some metrics wich ar ethen shipped of to an Elastic agent >> to be ingested into Elasticsearch. All seems fine to start with, but after >> some time, metrics stop comming in, and the prometheus logs shows a lot of >> entries like this: >> >> ts=2023-01-18T10:51:46.125Z caller=dedupe.go:112 component=remote >> level=warn remote_name=010ca8 url= >> http://agent-svc.observability.svc.cluster.local:9201/write msg="Failed >> to send batch, retrying" err="Post \" >> http://agent-svc.observability.svc.cluster.local:9201/write\": context >> deadline exceeded" >> >> ts=2023-01-18T10:51:20.364Z caller=dedupe.go:112 component=remote >> level=debug remote_name=010ca8 url= >> http://agent-svc.observability.svc.cluster.local:9201/write msg="Not >> downsharding due to being too far behind" >> >> I am guessing that Prometheus is trying to tell me something, but i just >> dont know what. >> >> Checking the TSDB status on the prom UI it tells me that Number of series >> is 8439 wich does not sound like a lot. >> Any help would be very appreciated. >> >> Best regards >> Christian Oelsner >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/4cb58947-e401-4213-bd34-8e00096f3807n%40googlegroups.com.
[prometheus-users] Re: Remote write dying after some time.
Looks to me like a problem at the receiver end (i.e. the middleware Elastic agent, or Elasticsearch itself), i.e. that side has stopped accepting data. Try looking at logs of these to determine why they are no longer accepting data. On Wednesday, 18 January 2023 at 11:29:05 UTC christia...@gmail.com wrote: > Hello guys. > > I am scraping some metrics wich ar ethen shipped of to an Elastic agent to > be ingested into Elasticsearch. All seems fine to start with, but after > some time, metrics stop comming in, and the prometheus logs shows a lot of > entries like this: > > ts=2023-01-18T10:51:46.125Z caller=dedupe.go:112 component=remote > level=warn remote_name=010ca8 url= > http://agent-svc.observability.svc.cluster.local:9201/write msg="Failed > to send batch, retrying" err="Post \" > http://agent-svc.observability.svc.cluster.local:9201/write\": context > deadline exceeded" > > ts=2023-01-18T10:51:20.364Z caller=dedupe.go:112 component=remote > level=debug remote_name=010ca8 url= > http://agent-svc.observability.svc.cluster.local:9201/write msg="Not > downsharding due to being too far behind" > > I am guessing that Prometheus is trying to tell me something, but i just > dont know what. > > Checking the TSDB status on the prom UI it tells me that Number of series > is 8439 wich does not sound like a lot. > Any help would be very appreciated. > > Best regards > Christian Oelsner > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/16e03bd3-eefb-4c68-87d4-a3d181315ab5n%40googlegroups.com.
