We are using prometheus version 2.15.2 and started seeing compaction failures that are causing all database files after the initial issue to fail to load.
In the graph this looks like a large gap from the start of failure to where the in-memory database begins. I've ******** out exact names, in console directory names and timestamps are displayed. level=info ts=2020-04-02T19:00:06.111Z caller=compact.go:496 component=tsdb msg="write block" mint=******** maxt=******** ulid=************ duration= 5.648508245s level=info ts=2020-04-02T19:00:07.180Z caller=head.go:668 component=tsdb msg ="head GC completed" duration=115.070561ms level=error ts=2020-04-02T19:00:13.304Z caller=db.go:609 component=tsdb msg="compaction failed" err="delete persisted head block after failed db reload:*****************: unlinkat /prometheus/****************/chunks: directory not empty" level=error ts=2020-04-02T19:00:14.572Z caller=db.go:609 component=tsdb msg="compaction failed" err="plan compaction: open /prometheus/**************************/meta.json: no such file or directory" level=error ts=2020-04-02T19:01:18.920Z caller=db.go:609 component=tsdb msg="compaction failed" err="plan compaction: open /prometheus/**************************/meta.json: no such file or directory" level=error ts=2020-04-02T19:02:27.367Z caller=db.go:609 component=tsdb msg="compaction failed" err="plan compaction: open /prometheus/**************************/meta.json: no such file or directory" level=error ts=2020-04-02T19:03:45.459Z caller=db.go:609 component=tsdb msg="compaction failed" err="plan compaction: open /prometheus/**************************/meta.json: no such file or directory" Is anyone familiar with this error? -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/b087ed1d-33fe-4245-b3f5-2f356c8a3d0a%40googlegroups.com.
