If we go clean all data under /wal then restart Prometheus, then the memory comes back to the low point as it was before triggering the restart. But we don't want to apply that trick as we could lose 3h time span of data.
On Wednesday, November 1, 2023 at 11:24:23 AM UTC+7 Vu Nguyen wrote: > Leaving the deployment running for a while after the 3rd restart of the > target - 6 rounds of WAL truncation, the memory goes up to 3.7Gi comparing > to 2.5Gi before doing the restart. There must be something that Prometheus > holds back for this upgrade/restart scenario I guess. > > [image: > ask_prometheus_user_memory_increase_after_target_restart_upgrade_v2.jpg] > > On Tuesday, October 31, 2023 at 10:07:24 PM UTC+7 Vu Nguyen wrote: > >> We have Prometheus v2.47.1 deployed on k8s; scraping 500k time series >> from a single target (*) >> >> When restart the target, the number of time series in HEAD block jump to >> 1M (1), and Prometheus memory increases from the average of 2.5Gi to 3Gi. >> Leave Prometheus running for few WAL truncation cycles, the memory still >> not go back to the point before restarting the target even the number of >> time series in HEAD block back to 500K. >> >> If I trigger another target restarts, that memory keeps going up. Here is >> the graph: >> >> Could you please help us understand why the memory does not fallback to >> the initial point (*) before we restart/upgrade target? >> >> [1] k8s pod restart will come up with a new IP - new instance label >> value; therefore, a new set of of 500K time series is generated. >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/80c7b428-125f-4f7b-8e79-5c5a382439cbn%40googlegroups.com.
