How are you getting the value of 15GiB? What do you get for process_resident_memory_bytes and go_memstats_alloc_bytes for Prometheus?
On Wed, Jan 26, 2022 at 6:58 AM Shubham Shrivastav < [email protected]> wrote: > Hi all, > > I've been investigating Prometheus memory utilization over the last couple > of days. > > Based on *pprof* command outputs, I do see a lot of memory utilized by > *getOrSet* function, but according to docs, it's just for creating new > series, so not sure what I can do about it. > > > Pprof "top" output: > https://pastebin.com/bAF3fGpN > > Also, to figure out if I have any metrics that I can remove I ran ./tsdb > analyze on memory *(output here: https://pastebin.com/twsFiuRk > <https://pastebin.com/twsFiuRk>)* > > I did find some metrics having more cardinality than others but the > difference was not very massive. > > With ~100 nodes our RAM takes around 15 Gigs. > > We're getting* average Metrics Per node: 8257* > > Our estimation is around 200 nodes, which will make our RAM go through the > roof. > > Apart from distributing our load over multiple Prometheus nodes, are there > any alertnatives? > > TIA, > Shubham > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/201d2d39-f5ca-4dfa-948b-5c6b54aa4dc4n%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/201d2d39-f5ca-4dfa-948b-5c6b54aa4dc4n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmqNOyu2O_fuErJxT_94B_oDN%3DAEsZ5cB9X3Pavxf6gLqw%40mail.gmail.com.
