thanks a lot Stuart for the reply. We have not done any change to the environment and it started working again by itself. That is something puzzling and i am not sure how to time travel to check the logs in prometheus, we have configured 30 days retention. i doubt it would be stlll there, or is there a way to get the logs at kubernetes level ?
Best regards, Bharani On Fri, Apr 16, 2021 at 12:27 PM Stuart Clark <[email protected]> wrote: > On 2021-04-16 11:08, Bharanidharan M wrote: > > Hi experts, > > > > Could you please help me with this issue, i am trying to understand > > what would have caused this issue.. for two weeks of time there is no > > data logged in prometheus by its scraping rules. > > it started working again after 2 weeks, not sure what went wrong. am i > > right in assuming that it could have been memory , disk issue and just > > need to increase it to avoid such issues again? > > > > [prometheus-users] component=tsdb msg="compaction failed" err="persist > > head block: mkdir /prometheus/aaaa.tmp: permission denied" > > > > That error suggests the user/group Prometheus is running as does not > have the correct permissions for the /prometheus directory. > > -- > Stuart Clark > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAK8hdHn8HTabCL9fwgo%2BPrRm%2BKkABXX1b2a-pttdmZgBxEGJKw%40mail.gmail.com.
