Issue : Prometheus Restarting with below error
``` level=error ts=2020-04-27T00:09:41.712Z caller=main.go:740
err="opening storage failed: block dir:
\"/prometheus/01DZCNZNP4SB2VVD8EY1V7HYSJ\": open
/prometheus/01DZCNZNP4SB2VVD8EY1V7HYSJ/meta.json: no such file or
directory" ```
Observation : 1. Checked the Datablocks & findout we lost around 300GB of
datablocks.
2. We have one disk space alert , Assuming that disk
got full before invoking this issue and something trigger the deleting on
old datablock the disk got free.
3. Moved the corrupted datablocks to /tmp directory
so promethus start working but old metrics are lost & After start its using
high resource 2X CPU load & high Memory utilisation.
Q1. Any Idea Why this error got triggered ?
Q2. What is the Normal behaviour of Promethus if no disk space is
available for storing the metrics ?
Q3 . If its deleted automatically then any parameters are available to
tweak this configurations ?
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/45855900-e87e-43a0-ae18-9ae50a14c88b%40googlegroups.com.
