Hi Ben, I have not configured the CPU/Memory limit values. Only the CPU/Memory requests values are configured. For storage, we are using portworx. After deleting the wal directory, the pod suddenly came up and started running. No restart for the last 21 hours. However, I can expect the restart after 1 or 2 days. That is what's been happening for more than 2 weeks.
Thanks & Regards, Anoop On Thu, Oct 5, 2023 at 3:31 AM Ben Kochie <sup...@gmail.com> wrote: > *ts=2023-10-04T20:33:32.135Z caller=main.go:776 level=warn msg="Received > SIGTERM, exiting gracefully..."* > > The time between your first log line and this line is about 15 minutes. > Kubernetes is timing out the startup and terminating your process. > > Your startup is _very_ slow. Without any other information it's too hard > to say why it's slow. Do you have CPU limits? What kind of storage? Too > many variables. > > On Wed, Oct 4, 2023 at 10:52 PM Anoop <anoopmohan....@gmail.com> wrote: > >> Hi, >> >> I have configured Prometheus in Kubernetes with 2 replicas. One of the >> two pods keeps restarting, but the other pod is working fine without any >> issues. I could not find any errors when describing the pod. Resource >> utilization is normal. The node has enough resources and storage available >> to run the Prometheus pod and store the data. The Prometheus log for the >> affected pod is provided below.. >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> *ts=2023-10-04T20:18:42.029Z caller=repair.go:57 level=info >> component=tsdb msg="Found healthy block" mint=1696298400000 >> maxt=1696305600000 >> ulid=01HBSYGH69GH5KVHETGXCVA0K6ts=2023-10-04T20:18:42.030Z >> caller=dir_locker.go:77 level=warn component=tsdb msg="A lockfile from a >> previous execution already existed. It was replaced" >> file=/prometheus/lockts=2023-10-04T20:18:43.722Z caller=head.go:488 >> level=info component=tsdb msg="Replaying on-disk memory mappable chunks if >> any"ts=2023-10-04T20:18:43.722Z caller=head.go:522 level=info >> component=tsdb msg="On-disk memory mappable chunks replay completed" >> duration=25.416µsts=2023-10-04T20:18:43.722Z caller=head.go:528 level=info >> component=tsdb msg="Replaying WAL, this may take a >> while"ts=2023-10-04T20:18:48.538Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=0 >> maxSegment=97ts=2023-10-04T20:21:37.477Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=1 >> maxSegment=97ts=2023-10-04T20:24:44.730Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=2 >> maxSegment=97ts=2023-10-04T20:28:52.391Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=3 >> maxSegment=97ts=2023-10-04T20:32:49.630Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=4 >> maxSegment=97ts=2023-10-04T20:33:32.135Z caller=main.go:776 level=warn >> msg="Received SIGTERM, exiting gracefully..."ts=2023-10-04T20:33:32.135Z >> caller=main.go:799 level=info msg="Stopping scrape discovery >> manager..."ts=2023-10-04T20:33:32.135Z caller=main.go:813 level=info >> msg="Stopping notify discovery manager..."ts=2023-10-04T20:33:32.135Z >> caller=main.go:835 level=info msg="Stopping scrape >> manager..."ts=2023-10-04T20:33:32.135Z caller=main.go:829 level=info >> msg="Scrape manager stopped"ts=2023-10-04T20:33:32.135Z caller=main.go:795 >> level=info msg="Scrape discovery manager >> stopped"ts=2023-10-04T20:33:32.135Z caller=main.go:809 level=info >> msg="Notify discovery manager stopped"ts=2023-10-04T20:33:32.135Z >> caller=manager.go:945 level=info component="rule manager" msg="Stopping >> rule manager..."ts=2023-10-04T20:33:32.135Z caller=manager.go:955 >> level=info component="rule manager" msg="Rule manager >> stopped"ts=2023-10-04T20:33:32.135Z caller=notifier.go:600 level=info >> component=notifier msg="Stopping notification >> manager..."ts=2023-10-04T20:33:32.135Z caller=main.go:1055 level=info >> msg="Notifier manager stopped"ts=2023-10-04T20:36:25.242Z >> caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" >> segment=5 maxSegment=97ts=2023-10-04T20:37:16.354Z caller=head.go:599 >> level=info component=tsdb msg="WAL segment loaded" segment=6 >> maxSegment=97ts=2023-10-04T20:37:18.434Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=7 >> maxSegment=97ts=2023-10-04T20:37:20.454Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=8 >> maxSegment=97ts=2023-10-04T20:37:22.493Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=9 >> maxSegment=97ts=2023-10-04T20:37:24.548Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=10 >> maxSegment=97ts=2023-10-04T20:37:26.625Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=11 >> maxSegment=97ts=2023-10-04T20:37:28.597Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=12 >> maxSegment=97ts=2023-10-04T20:37:30.632Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=13 >> maxSegment=97ts=2023-10-04T20:37:32.638Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=14 >> maxSegment=97ts=2023-10-04T20:37:34.585Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=15 >> maxSegment=97ts=2023-10-04T20:37:36.726Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=16 >> maxSegment=97ts=2023-10-04T20:37:38.708Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=17 >> maxSegment=97ts=2023-10-04T20:37:40.706Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=18 >> maxSegment=97ts=2023-10-04T20:37:42.688Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=19 >> maxSegment=97ts=2023-10-04T20:37:44.635Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=20 >> maxSegment=97ts=2023-10-04T20:37:46.616Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=21 >> maxSegment=97ts=2023-10-04T20:37:48.611Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=22 >> maxSegment=97ts=2023-10-04T20:37:53.567Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=23 >> maxSegment=97ts=2023-10-04T20:37:57.835Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=24 >> maxSegment=97ts=2023-10-04T20:37:59.312Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=25 >> maxSegment=97ts=2023-10-04T20:38:00.824Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=26 >> maxSegment=97ts=2023-10-04T20:38:02.312Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=27 >> maxSegment=97ts=2023-10-04T20:38:03.740Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=28 >> maxSegment=97ts=2023-10-04T20:38:05.160Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=29 >> maxSegment=97ts=2023-10-04T20:38:06.648Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=30 >> maxSegment=97ts=2023-10-04T20:38:08.184Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=31 >> maxSegment=97ts=2023-10-04T20:38:09.148Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=32 >> maxSegment=97ts=2023-10-04T20:38:09.148Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=33 >> maxSegment=97ts=2023-10-04T20:38:09.148Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=34 >> maxSegment=97ts=2023-10-04T20:38:09.149Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=35 >> maxSegment=97ts=2023-10-04T20:38:09.149Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=36 >> maxSegment=97ts=2023-10-04T20:38:09.149Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=37 >> maxSegment=97ts=2023-10-04T20:38:09.150Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=38 >> maxSegment=97ts=2023-10-04T20:38:09.150Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=39 >> maxSegment=97ts=2023-10-04T20:38:09.150Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=40 >> maxSegment=97ts=2023-10-04T20:38:09.151Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=41 >> maxSegment=97ts=2023-10-04T20:38:09.151Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=42 >> maxSegment=97ts=2023-10-04T20:38:09.151Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=43 >> maxSegment=97ts=2023-10-04T20:38:09.152Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=44 >> maxSegment=97ts=2023-10-04T20:38:09.152Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=45 >> maxSegment=97ts=2023-10-04T20:38:09.152Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=46 >> maxSegment=97ts=2023-10-04T20:38:09.152Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=47 >> maxSegment=97ts=2023-10-04T20:38:09.153Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=48 >> maxSegment=97ts=2023-10-04T20:38:09.153Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=49 >> maxSegment=97ts=2023-10-04T20:38:09.153Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=50 >> maxSegment=97ts=2023-10-04T20:38:09.154Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=51 >> maxSegment=97ts=2023-10-04T20:38:09.154Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=52 >> maxSegment=97ts=2023-10-04T20:38:09.154Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=53 >> maxSegment=97ts=2023-10-04T20:38:09.154Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=54 >> maxSegment=97ts=2023-10-04T20:38:09.155Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=55 >> maxSegment=97ts=2023-10-04T20:38:09.155Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=56 >> maxSegment=97ts=2023-10-04T20:38:09.155Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=57 >> maxSegment=97ts=2023-10-04T20:38:09.156Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=58 >> maxSegment=97ts=2023-10-04T20:38:09.156Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=59 >> maxSegment=97ts=2023-10-04T20:38:09.156Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=60 >> maxSegment=97ts=2023-10-04T20:38:09.156Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=61 >> maxSegment=97ts=2023-10-04T20:38:09.157Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=62 >> maxSegment=97ts=2023-10-04T20:38:09.157Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=63 >> maxSegment=97ts=2023-10-04T20:38:09.157Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=64 >> maxSegment=97ts=2023-10-04T20:38:09.158Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=65 >> maxSegment=97ts=2023-10-04T20:38:09.158Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=66 >> maxSegment=97ts=2023-10-04T20:38:09.158Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=67 >> maxSegment=97ts=2023-10-04T20:38:09.158Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=68 >> maxSegment=97ts=2023-10-04T20:38:09.159Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=69 >> maxSegment=97ts=2023-10-04T20:38:09.159Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=70 >> maxSegment=97ts=2023-10-04T20:38:09.159Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=71 >> maxSegment=97ts=2023-10-04T20:38:09.160Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=72 >> maxSegment=97ts=2023-10-04T20:38:09.160Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=73 >> maxSegment=97ts=2023-10-04T20:38:09.160Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=74 >> maxSegment=97ts=2023-10-04T20:38:09.161Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=75 >> maxSegment=97ts=2023-10-04T20:38:09.161Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=76 >> maxSegment=97ts=2023-10-04T20:38:09.161Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=77 >> maxSegment=97ts=2023-10-04T20:38:09.161Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=78 >> maxSegment=97ts=2023-10-04T20:38:09.162Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=79 >> maxSegment=97ts=2023-10-04T20:38:09.162Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=80 >> maxSegment=97ts=2023-10-04T20:38:09.162Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=81 >> maxSegment=97ts=2023-10-04T20:38:09.162Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=82 >> maxSegment=97ts=2023-10-04T20:38:09.163Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=83 >> maxSegment=97ts=2023-10-04T20:38:09.163Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=84 >> maxSegment=97ts=2023-10-04T20:38:09.163Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=85 >> maxSegment=97ts=2023-10-04T20:38:09.164Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=86 >> maxSegment=97ts=2023-10-04T20:38:09.164Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=87 >> maxSegment=97ts=2023-10-04T20:38:09.164Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=88 >> maxSegment=97ts=2023-10-04T20:38:09.165Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=89 >> maxSegment=97ts=2023-10-04T20:38:09.165Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=90 >> maxSegment=97ts=2023-10-04T20:38:09.165Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=91 >> maxSegment=97ts=2023-10-04T20:38:09.166Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=92 >> maxSegment=97ts=2023-10-04T20:38:09.166Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=93 >> maxSegment=97ts=2023-10-04T20:38:09.166Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=94 >> maxSegment=97ts=2023-10-04T20:38:09.167Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=95 >> maxSegment=97ts=2023-10-04T20:38:09.167Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=96 >> maxSegment=97ts=2023-10-04T20:38:09.167Z caller=head.go:599 level=info >> component=tsdb msg="WAL segment loaded" segment=97 >> maxSegment=97ts=2023-10-04T20:38:09.167Z caller=head.go:605 level=info >> component=tsdb msg="WAL replay completed" >> checkpoint_replay_duration=911.162µs wal_replay_duration=19m25.444199083s >> total_replay_duration=19m25.445156166sts=2023-10-04T20:38:22.160Z >> caller=main.go:945 level=info >> fs_type=EXT4_SUPER_MAGICts=2023-10-04T20:38:22.160Z caller=main.go:948 >> level=info msg="TSDB started"ts=2023-10-04T20:38:22.160Z >> caller=main.go:1067 level=info msg="See you next time!"* >> >> Any thoughts? >> >> Thanks, >> Anoop >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to prometheus-users+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/fee699aa-d41d-4cd1-b70b-9adc8dc0db20n%40googlegroups.com >> <https://groups.google.com/d/msgid/prometheus-users/fee699aa-d41d-4cd1-b70b-9adc8dc0db20n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAEQqSpdj7bZGLY7Ac6eL8eD7LcF90D3mScuhWGvZ%2BueM6k9Kcw%40mail.gmail.com.
