We upgraded our Thanos from *v0.6.0* to* v0.13.0* and our Prometheus instances from *v2.9.2* to *v2.19.2* a while ago and ran into an issue where some instances weren't uploading to S3.
Upon further investigation we realized that some of our Prometheus instances haven't been compacting and creating blocks, so the sidecar-component doesn't see anything to upload. The other instances are doing fine. Other than that, we can see that there are still old blocks, and they have already been uploaded according to the *thanos.shipper.json* file, but they are still there (Long past the retention of 24h). We can see the WAL keeps growing as well. It doesn't look like deleting it fixes it, it does gather data but Prometheus just never compacts the data into a block.The only way so far that we've been able to fix this was to delete the instance's PersistantVolume and scale the pod back up. It ran without issues from there on and uploaded fine, but we did lose the old data. But then just earlier I noticed a brand new instance also didn't compact, which had no previous blocks. The logs are not mentioning anything about the TSDB after replaying the WAL when restarted. Are there any solutions to this? We would rather try and recover the data if possible, but if there is no other choice we'll have to delete the volumes for all the problematic instances. The following flags are what we are using for all of our Prometheus instances: - - --storage.tsdb.path=/data - --web.console.libraries=/etc/prometheus/console_libraries - --web.console.templates=/etc/prometheus/consoles - --web.enable-lifecycle - --web.enable-admin-api - --storage.tsdb.min-block-duration=2h - --storage.tsdb.max-block-duration=2h - --storage.tsdb.retention=24h -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/126e35b7-6d97-4654-b7c6-9d131b152c6dn%40googlegroups.com.
