On 29 Jul 07:09, Theunis Botha wrote: > We upgraded our Thanos from *v0.6.0* to* v0.13.0* and our Prometheus > instances from *v2.9.2* to *v2.19.2* a while ago and ran into an issue > where some instances weren't uploading to S3. > > Upon further investigation we realized that some of our Prometheus > instances haven't been compacting and creating blocks, so the > sidecar-component doesn't see anything to upload. The other instances are > doing fine. Other than that, we can see that there are still old blocks, > and they have already been uploaded according to the *thanos.shipper.json* > file, > but they are still there (Long past the retention of 24h). > > We can see the WAL keeps growing as well. It doesn't look like deleting it > fixes it, it does gather data but Prometheus just never compacts the data > into a block.The only way so far that we've been able to fix this was to > delete the instance's PersistantVolume and scale the pod back up. It ran > without issues from there on and uploaded fine, but we did lose the old > data. But then just earlier I noticed a brand new instance also didn't > compact, which had no previous blocks. The logs are not mentioning anything > about the TSDB after replaying the WAL when restarted. > > Are there any solutions to this? We would rather try and recover the data > if possible, but if there is no other choice we'll have to delete the > volumes for all the problematic instances.
Which file system are you using? Can you please provide the logs? Thank you > > The following flags are what we are using for all of our Prometheus > instances: > > - > - --storage.tsdb.path=/data > - --web.console.libraries=/etc/prometheus/console_libraries > - --web.console.templates=/etc/prometheus/consoles > - --web.enable-lifecycle > - --web.enable-admin-api > - --storage.tsdb.min-block-duration=2h > - --storage.tsdb.max-block-duration=2h > - --storage.tsdb.retention=24h > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/126e35b7-6d97-4654-b7c6-9d131b152c6dn%40googlegroups.com. -- Julien Pivotto @roidelapluie -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/20200729141144.GA596883%40oxygen.
