On 18 Dec 16:40, Brett Larson wrote: > Here is a link to the "snapshot" of the dashboard. > https://snapshot.raintank.io/dashboard/snapshot/crxdjU7fhzAhl0x0KWiH1ZHGZXKyhqmF
Thanks, however this does not seem to show any memory issue. There might be an issue with your configuration, where you could take advantage of some tweaks , like reusing the same sd configs + relabeling or using selectors: in your kubernetes config. I would expect that if that is still an issue, a memory profile of Prometheus when memory is high would help. > On Friday, December 18, 2020 at 6:09:13 PM UTC-6 Julien Pivotto wrote: > > > On 18 Dec 16:03, Brett Larson wrote: > > > Hello, > > > I am using a Prometheus 2.21 server and I am seeing that the memory is > > > growing at an untenable rate. I will start the pod at 4GB and eventually > > it > > > will move to 10GB after a few days and go into a crashloop back off > > state. > > > > > > The pod is configured to only keep around 8 hours of data, and this data > > is > > > stored to empyDir, not a persistent file system. We are doing remote > > write > > > to postgres for only about 4 metrics. > > > > > > We do have some no-nos (high cardinality labels & pod names) but > > > unfortunately these are needed. > > > > > > I don't understand why after a retention of 8 hours we would still cause > > > memory like this to grow and i'm looking for some guidance on how I can > > > troubleshoot this. > > > > > > Please let me know, > > > Thank you! > > > > Can you share screenshots of the > > https://grafana.com/grafana/dashboards/12054 dashboard ? > > > > -- > > Julien Pivotto > > @roidelapluie > > > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to prometheus-users+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/98ef5eea-5552-42c8-9798-8c005cd01eaen%40googlegroups.com. -- Julien Pivotto @roidelapluie -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/20201219005000.GA410997%40oxygen.
