Re: [prometheus-users] Prometheus Growth
Hi Stuart Thank you so much! It was very useful! I got a lot of insights! On Monday, November 21, 2022 at 5:25:52 AM UTC-3 Stuart Clark wrote: > On 20/11/2022 22:53, Julio Leal wrote: > > Hi everyone! > > I'm doing a study about how much time we have in our prometheus > instances. > > First of all, I thought that prometheus memory grew because number > > timeseries ingested. I thought like this because timeseries stay in > > memory to recover informations more faster. > > But I got timeseries from last 3 months and I found that my timeseries > > ingested was grew little, like this graph and in the same time, I > > needed to increase the memory ram and CPU of the our instances sometimes. > > There was an interesting talk at the recent PromCon EU > (https://promcon.io/2022-munich/talks/why-is-it-so-big-analysing-the-m/) > which is available on YouTube, but basically the main memory usage is > generally due to the number of time series, with only a very minor usage > due to queries. > > -- > Stuart Clark > > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/1ee95e5e-6f4b-4a0c-8f73-4f8191209a4dn%40googlegroups.com.
Re: [prometheus-users] Prometheus Growth
On 20/11/2022 22:53, Julio Leal wrote: Hi everyone! I'm doing a study about how much time we have in our prometheus instances. First of all, I thought that prometheus memory grew because number timeseries ingested. I thought like this because timeseries stay in memory to recover informations more faster. But I got timeseries from last 3 months and I found that my timeseries ingested was grew little, like this graph and in the same time, I needed to increase the memory ram and CPU of the our instances sometimes. There was an interesting talk at the recent PromCon EU (https://promcon.io/2022-munich/talks/why-is-it-so-big-analysing-the-m/) which is available on YouTube, but basically the main memory usage is generally due to the number of time series, with only a very minor usage due to queries. -- Stuart Clark -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/1ec0a449-2a46-3c3d-31aa-79c0e0280bd3%40Jahingo.com.
[prometheus-users] Prometheus Growth
Hi everyone! I'm doing a study about how much time we have in our prometheus instances. First of all, I thought that prometheus memory grew because number timeseries ingested. I thought like this because timeseries stay in memory to recover informations more faster. But I got timeseries from last 3 months and I found that my timeseries ingested was grew little, like this graph and in the same time, I needed to increase the memory ram and CPU of the our instances sometimes. [image: Screen Shot 2022-11-20 at 7.20.07 PM.png] After that I used query log to get queries on prometheus server and I got execTotalTime and timestamp and got amount queries in between two dates (10 minutes, 5 minutes, 1 hour). I also got sum and average of the queries spend time between of the same dates, like this graph: [image: Screen Shot 2022-11-20 at 7.43.52 PM.png] With that I discovered that my record rules did my amount queries stay almost the same. I isolated queries of the record rules from web view the difference. [image: Screen Shot 2022-11-20 at 7.45.15 PM.png] When I did this, the amount of metrics changed, but the memory usage doesn't kept up with growth of the queries. My question here is. What can I consider to mensure memory and/or CPU usage growth on prometheus server? Timeseries, queries or another factor that I am not considering? Thank you! -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/b7e0be98-0507-4faf-99f9-1fab7995bccbn%40googlegroups.com.
