Hi, On 10/23/20 12:56 AM, Shox wrote: > We are experiencing a catch22, when there is "no space left on device" > on attempt to write to /wal and disk space is not freed. After some > investigation, it looks like removing old data is happening only after > compaction, but compaction can't happen as no pace left on device. > > Is that a expected behavior? Is there any solutions around this? Or any > ongoing work to prevent this? > > I actually do not understand a reason why that was done by this way. > Could somebody please help?
I would say this is expected behavior and you would be expected to leave enough space for Prometheus to breathe. As far as I understand, compaction works by reading the existing blocks, merging them and writing the new files to disk. The original files can only be safely deleted when the new file has been written. There is ongoing work regarding compaction [2], but I think it's related to those cases where it is explcitly triggered via clean_tombstones. [1] https://prometheus.io/docs/prometheus/latest/storage/ [2] https://github.com/prometheus/prometheus/issues/7957 Kind regards, Christian -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/32d0cc5c-97ff-bf6c-b389-e7334480900c%40hoffmann-christian.info.
