Re: [prometheus-users] Sudden & Permanent increase in Memory consumption.

RSS is not a true number. It's more of an estimate than anything. The Linux kernel tracking of process memory is fairly lazy. Especially when you get into more complex things like MADV_DONTNEED vs MADV_FREE. If you really want to get a more realistic number, you can look at PSS, but it's an expens

Re: [prometheus-users] Sudden & Permanent increase in Memory consumption.

Also, I had a look at Go memory utilization. I see that Go memory utilization(go_memstats_alloc_bytes) is around 50% of the total memory used by Prometheus(process_resident_memory_bytes) On Tuesday, February 9, 2021 at 2:39:42 AM UTC+5:30 yagyans...@gmail.com wrote: > Thanks, Ben. I'll upgra

Re: [prometheus-users] Sudden & Permanent increase in Memory consumption.

Thanks, Ben. I'll upgrade to a newer Prometheus version and check if the issue still persists. But I still have one doubt here, I am running this Prometheus instance for almost an year now, but I have noticed this memory increase recently only. First time on 31st Jan and 2nd time on Feb 8. If it r

Re: [prometheus-users] Sudden & Permanent increase in Memory consumption.

Prometheus performs compactions at regular intervals. This is likely what generated some IO. Note, if you're just looking at RSS, this is not going to tell the whole story. Depending on which version of Go you built with, it may not be fast at reclaiming RSS memory. Look at the go_memstats_alloc_

[prometheus-users] Sudden & Permanent increase in Memory consumption.

Hi. I am using Prometheus version 2.12.0. I am running Alertmanager 0.21.0 in cluster mode. Since, last 9 days, I have observed twice that the memory consumption by Prometheus increased by 10-12% and it remained to the increased value there after. Interesting thing to note here is that both th