ygalashov commented on issue #272: Controller crash loop URL: https://github.com/apache/incubator-openwhisk-deploy-kube/issues/272#issuecomment-412124179 Yes it's failing liveness probe, here is describe output: ``` Name: controller-0 Namespace: openwhisk Node: gke-cluster-eu-default-pool-3ca68bf7-ph58/10.156.0.2 Start Time: Fri, 10 Aug 2018 18:33:24 +0300 Labels: controller-revision-hash=controller-757c6bf95b name=controller Annotations: kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"StatefulSet","namespace":"openwhisk","name":"controller","uid":"b8112e6f-9cb2-11e8-89e0-42010a9c0121","api... Status: Running IP: 10.12.0.236 Controlled By: StatefulSet/controller Init Containers: wait-for-kafka: Container ID: docker://4dbe9a5a8af0d767c8c3679b6bb0194fb1383c800a253622a868d96b62c33203 Image: busybox Image ID: docker-pullable://busybox@sha256:5e8e0509e829bb8f990249135a36e81a3ecbe94294e7a185cc14616e5fad96bd Port: <none> Command: sh -c result=1; until [ $result -eq 0 ]; do OK=$(echo ruok | nc -w 1 zookeeper-0.zookeeper.openwhisk.svc.cluster.local 2181); if [ "$OK" == "imok" ]; then result=0; echo "zookeeper returned imok!"; fi; echo waiting for zookeeper to be ready; sleep 1; done; echo zookeeper is up, sleeping for 10 seconds; sleep 10; State: Terminated Reason: Completed Exit Code: 0 Started: Fri, 10 Aug 2018 18:33:37 +0300 Finished: Fri, 10 Aug 2018 18:34:08 +0300 Ready: True Restart Count: 0 Environment: <none> Mounts: /var/run/secrets/kubernetes.io/serviceaccount from ow-core-token-zhcqk (ro) wait-for-couchdb: Container ID: docker://f53bf75775348020d4269c4b5c0216148938adb78be9340c43a5edcd92f76a5e Image: busybox Image ID: docker-pullable://busybox@sha256:5e8e0509e829bb8f990249135a36e81a3ecbe94294e7a185cc14616e5fad96bd Port: <none> Command: sh -c result=1; until [ $result -eq 0 ]; do echo verifying CouchDB readiness; wget -T 5 --spider $READINESS_URL --header="Authorization: Basic d2hpc2tfYWRtaW46c29tZV9wYXNzdzByZA=="; result=$?; sleep 1; done; State: Terminated Reason: Completed Exit Code: 0 Started: Fri, 10 Aug 2018 18:34:10 +0300 Finished: Fri, 10 Aug 2018 18:35:52 +0300 Ready: True Restart Count: 0 Environment: READINESS_URL: http://couchdb.openwhisk.svc.cluster.local:5984/test_activations Mounts: /var/run/secrets/kubernetes.io/serviceaccount from ow-core-token-zhcqk (ro) Containers: controller: Container ID: docker://52fa02af0e05fa671606a6cc29fa003609dfe6298d2a0d04ebb49fe8e245e50d Image: openwhisk/controller:latest Image ID: docker-pullable://openwhisk/controller@sha256:797fd17bea1894063293640e88de5deb439a7fd0e714de15596b0896aebf5345 Ports: 8080/TCP, 2552/TCP, 19999/TCP Command: /bin/bash -c /init.sh `hostname | cut -d'-' -f2` State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Error Exit Code: 143 Started: Fri, 10 Aug 2018 18:40:36 +0300 Finished: Fri, 10 Aug 2018 18:40:54 +0300 Ready: False Restart Count: 6 Liveness: http-get http://:8080/ping delay=5s timeout=1s period=10s #success=1 #failure=3 Environment: PORT: 8080 CONFIG_whisk_info_date: <set to the key 'whisk_info_date' of config map 'whisk.config'> Optional: false CONFIG_whisk_info_buildNo: <set to the key 'whisk_info_buildNo' of config map 'whisk.config'> Optional: false JAVA_OPTS: -Xmx1024M CONTROLLER_OPTS: RUNTIMES_MANIFEST: { "runtimes": { "nodejs": [ { "kind": "nodejs", "image": { "prefix": "openwhisk", "name": "nodejsaction", "tag": "latest" }, "deprecated": true }, { "kind": "nodejs:6", "default": true, "image": { "prefix": "openwhisk", "name": "nodejs6action", "tag": "latest" }, "deprecated": false, "stemCells": [{ "count": 2, "memory": "256 MB" }] }, { "kind": "nodejs:8", "default": false, "image": { "prefix": "openwhisk", "name": "action-nodejs-v8", "tag": "latest" }, "deprecated": false } ], "python": [ { "kind": "python", "image": { "prefix": "openwhisk", "name": "python2action", "tag": "latest" }, "deprecated": false }, { "kind": "python:2", "default": true, "image": { "prefix": "openwhisk", "name": "python2action", "tag": "latest" }, "deprecated": false }, { "kind": "python:3", "image": { "prefix": "openwhisk", "name": "python3action", "tag": "latest" }, "deprecated": false } ], "swift": [ { "kind": "swift:3.1.1", "image": { "prefix": "openwhisk", "name": "action-swift-v3.1.1", "tag": "latest" }, "deprecated": false }, { "kind": "swift:4.1", "default": true, "image": { "prefix": "openwhisk", "name": "action-swift-v4.1", "tag": "latest" }, "deprecated": false } ], "java": [ { "kind": "java", "default": true, "image": { "prefix": "openwhisk", "name": "java8action", "tag": "latest" }, "deprecated": false, "attached": { "attachmentName": "jarfile", "attachmentType": "application/java-archive" }, "sentinelledLogs": false, "requireMain": true } ], "php": [ { "kind": "php:7.1", "default": true, "deprecated": false, "image": { "prefix": "openwhisk", "name": "action-php-v7.1", "tag": "latest" } } ] }, "blackboxes": [ { "prefix": "openwhisk", "name": "dockerskeleton", "tag": "latest" } ] } CONFIG_whisk_loadbalancer_invokerBusyThreshold: 8 KAFKA_HOSTS: kafka.openwhisk.svc.cluster.local:9092 KAFKA_HOST_PORT: 9092 CONFIG_whisk_couchdb_username: <set to the key 'db_username' in secret 'db.auth'> Optional: false CONFIG_whisk_couchdb_password: <set to the key 'db_password' in secret 'db.auth'> Optional: false CONFIG_whisk_couchdb_port: <set to the key 'db_port' of config map 'db.config'> Optional: false CONFIG_whisk_couchdb_protocol: <set to the key 'db_protocol' of config map 'db.config'> Optional: false CONFIG_whisk_couchdb_host: couchdb.openwhisk.svc.cluster.local CONFIG_whisk_couchdb_provider: <set to the key 'db_provider' of config map 'db.config'> Optional: false CONFIG_whisk_couchdb_databases_WhiskActivation: <set to the key 'db_whisk_activations' of config map 'db.config'> Optional: false CONFIG_whisk_couchdb_databases_WhiskEntity: <set to the key 'db_whisk_actions' of config map 'db.config'> Optional: false CONFIG_whisk_couchdb_databases_WhiskAuth: <set to the key 'db_whisk_auths' of config map 'db.config'> Optional: false LIMITS_ACTIONS_SEQUENCE_MAXLENGTH: 50 LIMITS_TRIGGERS_FIRES_PERMINUTE: 60 LIMITS_ACTIONS_INVOKES_PERMINUTE: 60 LIMITS_ACTIONS_INVOKES_CONCURRENT: 30 CONTROLLER_INSTANCES: 1 Mounts: /var/run/secrets/kubernetes.io/serviceaccount from ow-core-token-zhcqk (ro) Conditions: Type Status Initialized True Ready False PodScheduled True Volumes: ow-core-token-zhcqk: Type: Secret (a volume populated by a Secret) SecretName: ow-core-token-zhcqk Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.alpha.kubernetes.io/notReady:NoExecute for 300s node.alpha.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 13m default-scheduler Successfully assigned controller-0 to gke-cluster-eu-default-pool-3ca68bf7-ph58 Normal SuccessfulMountVolume 13m kubelet, gke-cluster-eu-default-pool-3ca68bf7-ph58 MountVolume.SetUp succeeded for volume "ow-core-token-zhcqk" Normal Pulled 13m kubelet, gke-cluster-eu-default-pool-3ca68bf7-ph58 Container image "busybox" already present on machine Normal Created 13m kubelet, gke-cluster-eu-default-pool-3ca68bf7-ph58 Created container Normal Started 13m kubelet, gke-cluster-eu-default-pool-3ca68bf7-ph58 Started container Normal Pulled 12m kubelet, gke-cluster-eu-default-pool-3ca68bf7-ph58 Container image "busybox" already present on machine Normal Created 12m kubelet, gke-cluster-eu-default-pool-3ca68bf7-ph58 Created container Normal Started 12m kubelet, gke-cluster-eu-default-pool-3ca68bf7-ph58 Started container Normal Started 10m (x2 over 10m) kubelet, gke-cluster-eu-default-pool-3ca68bf7-ph58 Started container Normal Pulling 9m (x3 over 10m) kubelet, gke-cluster-eu-default-pool-3ca68bf7-ph58 pulling image "openwhisk/controller:latest" Warning Unhealthy 9m (x5 over 10m) kubelet, gke-cluster-eu-default-pool-3ca68bf7-ph58 Liveness probe failed: Get http://10.12.0.236:8080/ping: dial tcp 10.12.0.236:8080: getsockopt: connection refused Normal Killing 9m (x2 over 10m) kubelet, gke-cluster-eu-default-pool-3ca68bf7-ph58 Killing container with id docker://controller:Container failed liveness probe.. Container will be killed and recreated. Normal Pulled 9m (x3 over 10m) kubelet, gke-cluster-eu-default-pool-3ca68bf7-ph58 Successfully pulled image "openwhisk/controller:latest" Normal Created 9m (x3 over 10m) kubelet, gke-cluster-eu-default-pool-3ca68bf7-ph58 Created container Warning FailedSync 8m (x4 over 8m) kubelet, gke-cluster-eu-default-pool-3ca68bf7-ph58 Error syncing pod ``` As far as I understand liveness probe calls pod directly, not through external ip?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services