Your *8257* metrics per node means you have *825,700* metrics on the server. The typical usage of Prometheus is around 8KiB per active series. This is expected to need *7GiB* of memory.
The problem is you have posted pprof and memory usage that do not match what your claims are. Without *data while under your real load*, it's impossible to tell you what is wrong. It's also useful to include the *prometheus_tsdb_head_series* metric. But again, while you have your 100 servers configured. You need to post more information about your actual setup. Please include your full configuration file and version information. On Thu, Feb 10, 2022 at 3:45 AM Shubham Shrivastav < [email protected]> wrote: > Hi all, > > I've been investigating Prometheus memory utilization over the last couple > of days. > > Based on *pprof* command outputs, I do see a lot of memory utilized by > *getOrSet* function, but according to docs, it's just for creating new > series, so not sure what I can do about it. > > > Pprof "top" output: > https://pastebin.com/bAF3fGpN > > Also, to figure out if I have any metrics that I can remove I ran ./tsdb > analyze on memory *(output here: https://pastebin.com/twsFiuRk > <https://pastebin.com/twsFiuRk>)* > > I did find some metrics having more cardinality than others but the > difference was not very massive. > > With ~100 nodes our RAM takes around 15 Gigs. > > We're getting* average Metrics Per node: 8257* > Our estimation is around 200 nodes, which will make our RAM go through the > roof. > > Present Situation: > Prometheus Containers got restarted due to OOM and I have fewer targets > now (~6). That's probably why numbers seem low, but the metrics pulled will > be the same. > I was trying to recognize the pattern > > Some metrics: > > *process_resident_memory_bytes{instance="localhost:9090", job="prometheus"} > 1536786432* > *go_memstats_alloc_bytes{instance="localhost:9090", job="prometheus"} > 908149496* > > Apart from distributing our load over multiple Prometheus nodes, are there > any alternatives? > > > > TIA, > Shubham > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/a74b38ab-ee70-46c6-bd5c-563aede095f4n%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/a74b38ab-ee70-46c6-bd5c-563aede095f4n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmpkgYNCP_8E9XV5SATm_uZHZ_P9-KVmBigukzXc8pEbNQ%40mail.gmail.com.
