Hi Ben,
Prometheus Containers got restarted due to OOM, and has fewer targets now
(~6). That's probably why numbers seem low, but the metrics pulled will be
same.
I was trying to recognize the pattern
Below are the metrics you requested,
*process_resident_memory_bytes{instance="localhost:9090", job="prometheus"}
1536786432*
*go_memstats_alloc_bytes{instance="localhost:9090", job="prometheus"}
908149496*
On Tuesday, 25 January 2022 at 22:55:11 UTC-8 [email protected] wrote:
> How are you getting the value of 15GiB?
>
> What do you get for process_resident_memory_bytes
> and go_memstats_alloc_bytes for Prometheus?
>
> On Wed, Jan 26, 2022 at 6:58 AM Shubham Shrivastav <[email protected]>
> wrote:
>
>> Hi all,
>>
>> I've been investigating Prometheus memory utilization over the last
>> couple of days.
>>
>> Based on *pprof* command outputs, I do see a lot of memory utilized by
>> *getOrSet* function, but according to docs, it's just for creating new
>> series, so not sure what I can do about it.
>>
>>
>> Pprof "top" output:
>> https://pastebin.com/bAF3fGpN
>>
>> Also, to figure out if I have any metrics that I can remove I ran ./tsdb
>> analyze on memory *(output here: https://pastebin.com/twsFiuRk
>> <https://pastebin.com/twsFiuRk>)*
>>
>> I did find some metrics having more cardinality than others but the
>> difference was not very massive.
>>
>> With ~100 nodes our RAM takes around 15 Gigs.
>>
>> We're getting* average Metrics Per node: 8257*
>>
>> Our estimation is around 200 nodes, which will make our RAM go through
>> the roof.
>>
>> Apart from distributing our load over multiple Prometheus nodes, are
>> there any alertnatives?
>>
>> TIA,
>> Shubham
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Prometheus Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected].
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/prometheus-users/201d2d39-f5ca-4dfa-948b-5c6b54aa4dc4n%40googlegroups.com
>>
>> <https://groups.google.com/d/msgid/prometheus-users/201d2d39-f5ca-4dfa-948b-5c6b54aa4dc4n%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/9130dd5a-592d-4df6-a3ae-dac8c51c85afn%40googlegroups.com.
