It is. You can clearly see in your graphs that the Go memstats goes back down to the prior level.
Go is a garbage collected language, memory use is going to fluctuate over time as Prometheus operates and GC happens. It's not an exact value and never will be. Memory use also depends on queries that are run, since each query requires some memory allocation. If your users have dashboards open with refresh, this will increase the amount of memory use. There are GC options like `GOGC` that you can tune, but of course this will impact how often GC is run, which could negatively impact performance. On Thu, Nov 30, 2023 at 2:30 PM Vu Nguyen <win...@gmail.com> wrote: > I could see the number of time series are truncated to a half, from 1M to > 500K but the memory does not go back to the point before target got > restarted (1) > > If I keep restarting the target the second time, the number of time series > will jump up to 1M and will eventually drop to 500M after few HEAD > truncation cycles but the memory now could be even higher than the case 1 > above. > > What confusing me here is that Prometheus memory does not return to the > point before restarting the target even the time series in HEAD goes back > to 500K. > > On Thu, Nov 30, 2023 at 6:16 PM Bryan Boreham <bjbore...@gmail.com> wrote: > >> >> >> > On 30 Nov 2023, at 07:20, Vu Nguyen <win...@gmail.com> wrote: >> > >> > >> > Thank you very much for your supports. >> > >> > > are you still deleting the WAL? >> > >> > No, I did not delete WAL at all. What I did was restarting a pod that >> has 500K time series exposed. >> > >> >> Ah, if there is any label different (eg the pod name) then that creates >> new series. >> So this would be consistent with the number of series bouncing up to >> 1,000,000. >> Prometheus only clears stale series out of the head after compaction, so >> you see series and memory go up for a couple of hours then come down again. >> >> In other words this is standard behaviour. >> >> Bryan >> >> >> -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to prometheus-users+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/CAF7KRaxGCRsfoW7Z7hTKwwx_vZzx9UddS9cKm-OztZG%3Duz90vw%40mail.gmail.com > <https://groups.google.com/d/msgid/prometheus-users/CAF7KRaxGCRsfoW7Z7hTKwwx_vZzx9UddS9cKm-OztZG%3Duz90vw%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmqL%2BCNAmjNafHm5cE3TVOAP_OFcXK7SoNxkqBYOOywoLg%40mail.gmail.com.
