I'm also observing the same situation and was wondering if any additional clarity could be given on this subject by some of the core prometheus developers?
I've also come across Github issues #11589 <https://github.com/prometheus/prometheus/issues/11589> which seems to be related to this same issue. On Friday, June 30, 2023 at 3:48:42 PM UTC-4 Julio Leal wrote: > Hi folks. > We have configured a remote write to Victoria Metrics cluster and for some > reason that we do not know we started get this error message: *Dropped > sample for series that was not explicitly dropped via relabelling* > > Our Prometheus instances are running in the kubernetes cluster > > Before we guessed was because cpu throttling and adjusted the GOMAXPROCS > to the same value that cpu limit (in our configuration cpu limit and > request have the same value) > > The CPU throttling decrease and so, we enabled auto-gomaxprocs feature, > but in both cases did not resolve. > > The next step was delete wal files from prometheus instances and some > curious happened. > When I delete the wal file and delete pod in sequence the drop samples > disappear, but if I delete the wal file and run a rollout processs (delete > the first pod, wait this pod get ready) and so the another pod is deleted, > the last pod deleted (that was working until the pod 1 to be deleted) do > not disappear the drop sample message. I've tried delete the wal again and > delete manually in sequence, the drop message do not appear after start > process. > > Looking for the source code > https://github.com/prometheus/prometheus/blob/main/storage/remote/queue_manager.go#L579 > > this message can happen in two situations: > 1. When it is not possible recover the labels and > 2. When it is not found the reference id to sample. > > My question is: > what causes these two situations? > How to avoid this? > How to resolve this? > > > > -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/ad6998ab-b771-417e-92d4-70cc310c8029n%40googlegroups.com.
