that makes sense but if I look at the numbers in the url you gave me: Number of Series 2514033 Number of Chunks 3098707 Number of Label Pairs 1088507 and use them in memory calculator I found, it shows me much less ram than what I am using now.
do you see any number here that should be a red light for me? something that is not right? ב-יום שני, 23 באוגוסט 2021 בשעה 14:58:36 UTC+3, [email protected] כתב/ה: > Prometheus needs memory to buffer incoming data before writing it to disk. > The more you scrape, the more it needs. > > You can see a summary of this information on prometheus:9090/tsdb-status > > On Mon, Aug 23, 2021 at 1:55 PM Yaron B <[email protected]> wrote: > >> can anyone understand from this image why is the server is using so much ? >> production-prometheus-server-869bffc459-r92nh 1186m >> 54937Mi >> thats crazy! >> ב-יום שני, 23 באוגוסט 2021 בשעה 13:35:18 UTC+3, Yaron B כתב/ה: >> >>> at the moment we did add some scrape jobs that bumped the memory usage >>> from around 30gb to 40gb but we are not sure why the self scraping takes so >>> much ram. >>> its not a new implementation, we did notice it is using a lot of memory >>> but it didn't crash on us so we let it run. today >>> as you can see in the attached image, it crashed, skyrocket the memory >>> usage to 60gb ,then we started to disable jobs until the server didn't >>> crash anymore but it is using more than it used in the last 15 days >>> >>> >>> ב-יום שני, 23 באוגוסט 2021 בשעה 13:29:59 UTC+3, Stuart Clark כתב/ה: >>> >>>> On 23/08/2021 11:23, Yaron B wrote: >>>> >>>> I am attaching the heap.svg if someone can help me figure out what is >>>> using the memory >>>> ב-יום שני, 23 באוגוסט 2021 בשעה 12:23:33 UTC+3, Yaron B כתב/ה: >>>> >>>>> Hi, >>>>> >>>>> we are facing an issue with the prometheus server memory usage. >>>>> when starting the server it starts with around 30GB of ram , even >>>>> without any jobs configured other than the self one. >>>>> in the image attached you can see the heap size usage for the >>>>> prometheus job. >>>>> is there a way to reduce this size? when we add our kubernetes scrape >>>>> job we reach our node limit and get OOMKilled. >>>>> >>>> So at the moment it isn't scraping anything other than itself via the >>>> /metrics endpoint? >>>> >>>> Is this a brand new service (i.e. no existing data stored on disk)? >>>> >>>> Is there anything querying the server (e.g. Grafana dashboards, etc.)? >>>> >>>> -- >>>> Stuart Clark >>>> >>>> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/0659c262-daeb-452e-8dc4-4df8df22021dn%40googlegroups.com >> >> <https://groups.google.com/d/msgid/prometheus-users/0659c262-daeb-452e-8dc4-4df8df22021dn%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/b17b8d09-fe23-4c43-b85e-c2f4d7a87539n%40googlegroups.com.
