Hi , when our system is booted up , system time was in sync with ntpd service, and prometheus also started running at that time(08:51 AM), after some 5 minutes, system time got in sync with GPS receiver and time was advanced by 10 mins(09:16 AM).
we have noticed the below error logs from prometheus. 14:24:41.837951 info CONTROLLER prometheus[1343]: level=warn ts=2021-03-31T08:54:41.837Z caller=scrape.go:1372 component="scrape manager" scrape_pool=prometheus target=http://pm:9090/metrics msg="Error on ingesting out-of-order samples" num_dropped=398 we are noticing the Error on ingesting out-of-order samples continuously. is there any way to recover from this issue, without any manual action (like restarting prometheus). Could some please help me with this ? prometheus version: 2.23.0 we don't have any rules configured and also we don't have multiple targets. Regards, Rajesh. -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAPXMrf9%2Bv1Q3Us3S6Z7WRyLEr3GG0Jbfxg9F_SDoFtVuNJSByg%40mail.gmail.com.
