Re: [prometheus-developers] RFC: The implementation of labels.Labels
Update: I reimplemented labels.Labels with a different data structure at https://github.com/prometheus/prometheus/pull/10991. This is reported by one person as working and saving 15-18% heap memory. Prombench had about 10% saving. Next step (maybe in a week or two) I will port Mimir to this fork. Bryan On Sunday, 19 June 2022 at 17:32:16 UTC+1 Bryan Boreham wrote: > https://github.com/prometheus/prometheus/pull/10887 > > On Wednesday, 25 May 2022 at 16:40:41 UTC+1 bjo...@rabenste.in wrote: > >> On 12.05.22 17:07, Bryan Boreham wrote: >> > >> > If there is interest I will make a PR containing just the above change, >> no >> > change to the structure labels.Labels itself as yet. >> >> I'm definitely interested. ;o) >> >> -- >> Björn Rabenstein >> [PGP-ID] 0x851C3DA17D748D03 >> [email] bjo...@rabenste.in >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/dad9f27a-b015-4913-a59a-cd7fd9344a99n%40googlegroups.com.
Re: [prometheus-developers] RFC: The implementation of labels.Labels
https://github.com/prometheus/prometheus/pull/10887 On Wednesday, 25 May 2022 at 16:40:41 UTC+1 bjo...@rabenste.in wrote: > On 12.05.22 17:07, Bryan Boreham wrote: > > > > If there is interest I will make a PR containing just the above change, > no > > change to the structure labels.Labels itself as yet. > > I'm definitely interested. ;o) > > -- > Björn Rabenstein > [PGP-ID] 0x851C3DA17D748D03 > [email] bjo...@rabenste.in > -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/c8b328b0-ab12-4b7e-b648-db74ef734a5an%40googlegroups.com.
Re: [prometheus-developers] RFC: The implementation of labels.Labels
On 12.05.22 17:07, Bryan Boreham wrote: > > If there is interest I will make a PR containing just the above change, no > change to the structure labels.Labels itself as yet. I'm definitely interested. ;o) -- Björn Rabenstein [PGP-ID] 0x851C3DA17D748D03 [email] bjo...@rabenste.in -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/Yo5N9oaGkUTL4v1I%40mail.rabenste.in.
[prometheus-developers] RFC: The implementation of labels.Labels
Hi, as you might infer from recent PRs I posted, I have been looking at the implementation of labels.Labels. I wanted to post a comprehensive design document, but it's been a few months and I didn't finish it, so I thought I would briefly describe the work here. Motivation: when I look at profiles of Prometheus and Cortex(ish) systems, labels account for 30-40% of Go heap memory. This translates directly into cloud costs, or if you have a machine of a certain size how many series it can handle. Brief accounting: labels.Labels is a slice, so takes up a 24-byte slice header plus 16-byte string header per name and value, plus the contents of the strings. Let's take as an example a series with 10 name/value pairs, where each name and value string averages 10 bytes long. Fundamentally the strings need 200 bytes, but labels.Labels takes up 24 + 10 * 2 * (16 + 10) = 544 bytes. That's 170% overhead. I have some ideas what to replace this with, but don't want to get into that detail right now. I will observe that the slice nature of labels.Labels is hard-coded in about 1,000 places in the code of Prometheus. I have done all the work to replace those with an abstraction: all tests pass, and benchmarks go broadly at the same speed. If there is interest I will make a PR containing just the above change, no change to the structure labels.Labels itself as yet. Regards, Bryan Boreham -- You received this message because you are subscribed to the Google Groups "Prometheus Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/6d396a59-2155-4438-bd0e-2ed70d6e7861n%40googlegroups.com.
