Hello Guys,
1. I'm getting for the first time a Readiness Probe issue with ly Prom
running on AWS EKS, the pod was running nice since weeks ant it suddenly
turns into failing.
*$ kdpon monitoring prometheus *(Kube pod describe command )
Unhealthy 4m (x182535 over 6d) kubelet, ip-100-xxxxxxxx.ec2.internal
Readiness
probe failed: HTTP probe failed with statuscode: 503
2. However, seems my logs are doing well, and restarting pods doesn't fix
the issue but this is a normal. behaviour for readiness Prob
*$ klon monitoring prometheus *(Kube log command )
level=info ts=2020-05-25T09:38:56.653582783Z caller=main.go:243
msg="Starting Prometheus" version="(version=2.6.1, branch=HEAD,
revision=b639fe140c1f71b2cbad3fc322b17efe60839e7e)"
level=info ts=2020-05-25T09:38:56.653650678Z caller=main.go:244
build_context="(go=go1.11.4, user=root@4c0e286fe2b3,
date=20190115-19:12:04)"
level=info ts=2020-05-25T09:38:56.653691321Z caller=main.go:245
host_details="(Linux 4.14.154-128.181.amzn2.x86_64 #1 SMP Sat Nov 16
21:49:00 UTC 2019 x86_64 prometheus-0 (none))"
level=info ts=2020-05-25T09:38:56.653727427Z caller=main.go:246
fd_limits="(soft=65536, hard=65536)"
level=info ts=2020-05-25T09:38:56.653762149Z caller=main.go:247
vm_limits="(soft=unlimited, hard=unlimited)"
level=info ts=2020-05-25T09:38:56.65457275Z caller=main.go:561
msg="Starting TSDB ..."
level=info ts=2020-05-25T09:38:56.655755388Z caller=web.go:429
component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2020-05-25T09:38:56.817276722Z caller=repair.go:48
component=tsdb msg="found healthy block" mint=1583940235972
maxt=1583971200000 ulid=01E36HT83DK3TG22QWE02AP50X
level=info ts=2020-05-25T09:38:56.822556254Z caller=repair.go:48
component=tsdb msg="found healthy block" mint=1583971200000
maxt=1584165600000 ulid=01E3C4BCPDEWX998E1EMR8RWTB
level=info ts=2020-05-25T09:38:56.826661565Z caller=repair.go:48
component=tsdb msg="found healthy block" mint=1584165600000
maxt=1584360000000 ulid=01E3HXRADQ09H3PD3KHPS5GAXK
level=info ts=2020-05-25T09:38:56.831874475Z caller=repair.go:48
component=tsdb msg="found healthy block" mint=1584360000000
maxt=1584554400000 ulid=01E3QQ4PWYR2D0TF0ERWR7KTZA
level=info ts=2020-05-25T09:38:56.837772421Z caller=repair.go:48
component=tsdb msg="found healthy block" mint=1584554400000
maxt=1584748800000 ulid=01E3XGHHPWR4QHYZBZGGSR41FS
level=info ts=2020-05-25T09:38:56.842330027Z caller=repair.go:48
component=tsdb msg="found healthy block" mint=1584748800000
maxt=1584943200000 ulid=01E439YBW2HYS58CK67540JRDP
level=info ts=2020-05-25T09:38:56.847023184Z caller=repair.go:48
component=tsdb msg="found healthy block" mint=1584943200000
maxt=1585137600000 ulid=01E493B0FCHAT74EZFVX63AA8J
level=info ts=2020-05-25T09:38:56.852126117Z caller=repair.go:48
component=tsdb msg="found healthy block" mint=1585137600000
maxt=1585332000000 ulid=01E4EWV0G5ZV7G0M5TH3XEV3X5
level=info ts=2020-05-25T09:38:56.856215058Z caller=repair.go:48
component=tsdb msg="found healthy block" mint=1585332000000
maxt=1585526400000 ulid=01E4MP49E017G3SF46C4MXND0A
level=info ts=2020-05-25T09:38:56.860292298Z caller=repair.go:48
component=tsdb msg="found healthy block" mint=1585526400000
maxt=1585720800000 ulid=01E4TFMEZ1YMEM8PJAQJA9WZA7
level=info ts=2020-05-25T09:38:56.865162002Z caller=repair.go:48
component=tsdb msg="found healthy block" mint=1585720800000
maxt=1585915200000 ulid=01E508Y5F1V3RBPRMPVYMQCFQM
level=info ts=2020-05-25T09:38:56.869511088Z caller=repair.go:48
component=tsdb msg="found healthy block" mint=1585915200000
maxt=1586109600000 ulid=01E562AXM9EHH3BV11MQYHGX80
I'm trying to figure if the issue is rrmated to Code or infrastructure layer
Thanks you in advance for you Help
Best,
ABH
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/8ae22e72-c753-4af0-b5b1-d28a60a1654e%40googlegroups.com.
