That could have been triggered by something else (oom, disk full..) Do you have the logs from before that point?
On 10 Jun 23:50, neel patel wrote: > Prometheus was working fine before 2 days and today when i stop and start > it again, it shows below error and not able to start the prometheus. > > I build prometheus from master branch and using that binary since 2 days. > Below are the logs when i start prometheus. > > > [neel@localhost prometheus]$ ./prometheus --config.file=prometheus.yml > --storage.tsdb.path=data/ > level=info ts=2020-06-11T06:25:10.967Z caller=main.go:302 msg="No time or > size retention was set so using the default time retention" duration=15d > level=info ts=2020-06-11T06:25:10.967Z caller=main.go:337 msg="Starting > Prometheus" version="(version=2.18.1, branch=master, > revision=18d9ebf0ffc26b8bd0e136f552c8e9886d29ade4)" > level=info ts=2020-06-11T06:25:10.967Z caller=main.go:338 > build_context="(go=go1.14.3, [email protected], > date=20200604-05:51:34)" > level=info ts=2020-06-11T06:25:10.967Z caller=main.go:339 > host_details="(Linux 3.10.0-1062.9.1.el7.x86_64 #1 SMP Fri Dec 6 15:49:49 > UTC 2019 x86_64 localhost.localdomain (none))" > level=info ts=2020-06-11T06:25:10.967Z caller=main.go:340 > fd_limits="(soft=1024, hard=4096)" > level=info ts=2020-06-11T06:25:10.967Z caller=main.go:341 > vm_limits="(soft=unlimited, hard=unlimited)" > level=info ts=2020-06-11T06:25:10.973Z caller=main.go:678 msg="Starting > TSDB ..." > level=info ts=2020-06-11T06:25:10.973Z caller=web.go:524 component=web > msg="Start listening for connections" address=0.0.0.0:9090 > level=info ts=2020-06-11T06:25:10.974Z caller=repair.go:59 component=tsdb > msg="Found healthy block" mint=1591250134254 maxt=1591257600000 > ulid=01EACBSE6H4EP5CS2K2G6KWJ0W > level=info ts=2020-06-11T06:25:10.974Z caller=repair.go:59 component=tsdb > msg="Found healthy block" mint=1591682400000 maxt=1591747200000 > ulid=01EAE4FPDZ932CDRBAGH019TK7 > level=info ts=2020-06-11T06:25:10.974Z caller=repair.go:59 component=tsdb > msg="Found healthy block" mint=1591747200000 maxt=1591768800000 > ulid=01EAEQAB8X70893C6ZC2T2ZK38 > level=info ts=2020-06-11T06:25:10.975Z caller=repair.go:59 component=tsdb > msg="Found healthy block" mint=1591790400000 maxt=1591797600000 > ulid=01EAGR8C1PMZZACQMN185NC254 > level=info ts=2020-06-11T06:25:10.975Z caller=repair.go:59 component=tsdb > msg="Found healthy block" mint=1591797600000 maxt=1591804800000 > ulid=01EAGR8D3YVDV517XCMBMMKCS7 > level=info ts=2020-06-11T06:25:10.975Z caller=repair.go:59 component=tsdb > msg="Found healthy block" mint=1591768800000 maxt=1591790400000 > ulid=01EAGR8ETQQC8MZ09SSC62N1D3 > level=info ts=2020-06-11T06:25:10.982Z caller=main.go:547 msg="Stopping > scrape discovery manager..." > level=info ts=2020-06-11T06:25:10.983Z caller=main.go:561 msg="Stopping > notify discovery manager..." > level=info ts=2020-06-11T06:25:10.983Z caller=main.go:583 msg="Stopping > scrape manager..." > level=info ts=2020-06-11T06:25:10.983Z caller=main.go:557 msg="Notify > discovery manager stopped" > level=info ts=2020-06-11T06:25:10.983Z caller=main.go:543 msg="Scrape > discovery manager stopped" > level=info ts=2020-06-11T06:25:10.983Z caller=manager.go:882 > component="rule manager" msg="Stopping rule manager..." > level=info ts=2020-06-11T06:25:10.983Z caller=manager.go:892 > component="rule manager" msg="Rule manager stopped" > level=info ts=2020-06-11T06:25:10.983Z caller=notifier.go:601 > component=notifier msg="Stopping notification manager..." > level=info ts=2020-06-11T06:25:10.983Z caller=main.go:749 msg="Notifier > manager stopped" > level=info ts=2020-06-11T06:25:10.983Z caller=main.go:577 msg="Scrape > manager stopped" > level=error ts=2020-06-11T06:25:10.983Z caller=main.go:758 err="opening > storage failed: found unsequential head chunk files 144 and 148" > > ######################### > > Data directory content as below > > 01EACBSE6H4EP5CS2K2G6KWJ0W 01EAEQAB8X70893C6ZC2T2ZK38 > 01EAGR8D3YVDV517XCMBMMKCS7 chunks_head queries.active > 01EAE4FPDZ932CDRBAGH019TK7 01EAGR8C1PMZZACQMN185NC254 > 01EAGR8ETQQC8MZ09SSC62N1D3 lock wal > > ########################## > > > > Prometheus config file as below. > > # my global config > global: > scrape_interval: 15s # Set the scrape interval to every 15 seconds. > Default is every 1 minute. > evaluation_interval: 15s # Evaluate rules every 15 seconds. The default > is every 1 minute. > # scrape_timeout is set to the global default (10s). > > # Alertmanager configuration > alerting: > alertmanagers: > - static_configs: > - targets: > # - alertmanager:9093 > > # Load rules once and periodically evaluate them according to the global > 'evaluation_interval'. > rule_files: > # - "first_rules.yml" > # - "second_rules.yml" > > # A scrape configuration containing exactly one endpoint to scrape: > # Here it's Prometheus itself. > scrape_configs: > # The job name is added as a label `job=<job_name>` to any timeseries > scraped from this config. > - job_name: 'prometheus' > > # Override the global default and scrape targets from this job every 5 > seconds. > scrape_interval: 5s > > static_configs: > - targets: ['localhost:9090'] > > # The job name is added as a label `job=<job_name>` to any timeseries > scraped from this config. > - job_name: 'postgres-exporter' > > # Override the global default and scrape targets from this job every 5 > seconds. > scrape_interval: 15s > > static_configs: > - targets: ['localhost:9187'] > ############################################# > > Let me know, is the TSDB is corrupted ? If yes, is there anyway to recover ? > > Thanks in Advance > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/a7b51fdd-1b14-4c15-8342-ea113ab24315o%40googlegroups.com. -- Julien Pivotto @roidelapluie -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/20200611071434.GA11274%40oxygen.
