Could someone please help me with the above query ? whenever time goes forward by 10 mins or so prometheus should be able to scrape the next samples right ? or in my case samples were dropped continuously? How to recover from that error. my scraping interval is 15 seconds.
Thanks, Rajesh Dasari. On Fri, Apr 9, 2021 at 8:06 AM RAJESH DASARI <[email protected]> wrote: > > Hi , > > when our system is booted up , system time was in sync with ntpd > service, and prometheus also started running at that time(08:51 AM), > after some 5 minutes, system time got in sync with GPS receiver and > time was advanced by 10 mins(09:16 AM). > > we have noticed the below error logs from prometheus. > > > 14:24:41.837951 info CONTROLLER prometheus[1343]: level=warn > ts=2021-03-31T08:54:41.837Z caller=scrape.go:1372 component="scrape > manager" scrape_pool=prometheus target=http://pm:9090/metrics > msg="Error on ingesting out-of-order samples" num_dropped=398 > > we are noticing the Error on ingesting out-of-order samples continuously. > > is there any way to recover from this issue, without any manual action > (like restarting prometheus). Could some please help me with this ? > > prometheus version: 2.23.0 > we don't have any rules configured and also we don't have multiple targets. > > Regards, > Rajesh. -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAPXMrf8LAW55dZmqS3N5D%3DU-UHJKaU%2B1Wk86ZQ1ESbWXWQC0Gg%40mail.gmail.com.
