Hi experts, Could you please help me with this issue, i am trying to understand what would have caused this issue.. for two weeks of time there is no data logged in prometheus by its scraping rules. it started working again after 2 weeks, not sure what went wrong. am i right in assuming that it could have been memory , disk issue and just need to increase it to avoid such issues again?
[prometheus-users] component=tsdb msg="compaction failed" err="persist head block: mkdir /prometheus/aaaa.tmp: permission denied" On Thu, Apr 15, 2021 at 3:17 PM Bharanidharan M < [email protected]> wrote: > Hi, > > I got this error (mentioned in subject) for a period of 2 weeks in our > Kubernetes cluster in Prometheus, this caused all our scraping rules to > fail and grafana was not showing any data. After 2 weeks, everything > started to work again. > > I am trying to understand what could have happened whether I need to > increase disk size? It is already 10gi or whether i have to increase > memory.. currently it is 2 gi limit and 1 gi request limit. > > please help. > > thanks in advance. > > Regards, > Bharani > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/7d21f32e-2e69-41c0-9ccd-710c616a500cn%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/7d21f32e-2e69-41c0-9ccd-710c616a500cn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAK8hdH%3DQDbdo0PMBm-8vYaF0RA7P548hC9_QRTR9dgb4RoPfoA%40mail.gmail.com.
