Hi, I have deployed Casablanca (few components only) using OOM on K8S on Azure VMs. Setup has 1 Rancher VM, 1 master VM and 5 slave VMs. Deployed components are: aaf, aai, appc, clamp, cli, esr, log, oof, msb, nbi, policy, portal, robot, sdc, sdnc, so, uui, vfc, vid, vnfsdk
Issue is - many pods are failing with liveness error and I am unable to access portal UI although all portal pods are 'Running'. Due to retry of failing pods, the box space gets all used up and after a day I cannot even access the rancher UI to debug. Kindly advise. Details of some pod issues are below. I was able to bring up portal-casablanca (from Init:Error) by increasing liveness:DelaySeconds from 10 to 180 as per some a suggestion on wiki. But this solution hasn't worked for other pods failing with same liveness error (aaf, oof pods are still failing with liveness error) aai-data-router is failing with "Readiness probe failed: dial tcp XXXXXX:9502: connect: connection refused". Many pods are 'Running' with their older version pods in 'Terminating' state. [I assumed Terminating ones should get removed but they haven't] e.g. onap dev-portal-portal-app-6f57b79969-28mtj 1/2 Terminating 6 3h 10.42.110.56 onap-vm5 <none> onap dev-portal-portal-app-6f57b79969-5glxj 2/2 Running 0 1h 10.42.149.141 onap-vm4 <none> ERROR PODS onap-rancher@onap-rancher:~/oom/kubernetes$ kubectl get pods --all-namespaces -o=wide | grep Error onap dev-policy-pap-6cfdf64867-v94q8 0/2 Init:Error 2 1h 10.42.XXX onap-vm6 <none> onap dev-sdc-sdc-dcae-be-tools-nchp4 0/1 Init:Error 0 5h 10.42.XXX onap-vm6 <none> onap dev-sdc-sdc-wfd-be-workflow-init-xg2cf 0/1 Init:Error 0 5h 10.XXX onap-vm2 <none> onap dev-vnfsdk-vnfsdk-init-postgres-2psxl 0/1 Init:Error 0 5h 10.XXXX onap-vm1 <none> INIT PODS onap-rancher@onap-rancher:~/oom/kubernetes$ kubectl get pods --all-namespaces -o=wide | grep Init onap dev-aai-aai-champ-54c559d86c-gf8c2 0/2 Init:0/1 0 1h 10.XXXX onap-vm5 <none> onap dev-aai-aai-spike-565c54f554-lvjrd 0/2 Init:0/1 5 1h 10.XXXX onap-vm1 <none> onap dev-appc-appc-dgbuilder-768d9f4fb8-bxsgr 0/1 Init:0/1 0 1h 10.XXXXX onap-vm2 <none> onap dev-clamp-clamp-dash-kibana-5d5d977fb4-dtj9s 0/1 Init:0/1 0 1h 10.XXXXX onap-vm5 <none> onap dev-log-log-logstash-746f464bd7-pmgfs 0/1 Init:0/1 0 1h 10.XXXXX onap-vm5 <none> onap dev-log-log-logstash-746f464bd7-sjpmh 0/1 Init:0/1 0 1h 10.42.123.231 onap-vm5 <none> onap dev-oof-music-tomcat-64d4c64db7-8bsxq 0/1 Init:0/3 0 1h 10.XXXXX onap-vm5 <none> onap dev-oof-oof-has-controller-9469b9ff8-jzfb9 0/1 Init:0/3 0 1h 10.XXXXX onap-vm1 <none> onap dev-oof-oof-has-data-d559897dc-pbzdp 0/1 Init:2/4 17 3h 10.XXXXX onap-vm6 <none> onap dev-oof-oof-has-reservation-868c7c88ff-tbnlr 0/1 Init:2/4 26 5h 10.XXXXXX onap-vm2 <none> onap dev-oof-oof-has-solver-6f8bc6fdf4-sklc4 0/1 Init:2/4 31 5h 10.XXXX onap-vm1 <none> onap dev-policy-brmsgw-5b8f5d6bd5-kt7rp 0/1 Init:0/1 0 1h 10.XXXXX onap-vm5 <none> onap dev-policy-pap-6cfdf64867-v94q8 0/2 Init:Error 2 1h 10.XXXXX onap-vm6 <none> onap dev-portal-portal-app-6f57b79969-d4n8x 0/2 Init:0/1 0 1h 10.XXXXX onap-vm5 <none> onap dev-sdc-sdc-dcae-be-tools-nchp4 0/1 Init:Error 0 5h 10.XXXXX onap-vm6 <none> onap dev-sdc-sdc-wfd-be-workflow-init-xg2cf 0/1 Init:Error 0 5h 10.XXXXX onap-vm2 <none> onap dev-sdnc-sdnc-ansible-server-64bbf769d7-776f9 0/1 Init:0/1 1 1h 10.XXXXX onap-vm6 <none> onap dev-sdnc-sdnc-dmaap-listener-84bffc54-627z2 0/1 Init:0/1 0 5h 10.XXXXX onap-vm2 <none> onap dev-sdnc-sdnc-portal-5dcd4bfbf6-k48q5 0/1 Init:0/1 9 1h 10.XXXXX onap-vm1 <none> onap dev-sdnc-sdnc-ueb-listener-6d74459c6-5rtcj 0/1 Init:0/1 0 1h 10.XXXX onap-vm5 <none> onap dev-so-so-bpmn-infra-58bf4f656d-pt4l5 0/1 Init:0/1 0 1h 10.XXXXX onap-vm5 <none> onap dev-vid-vid-7c5df6554b-2kkv9 0/2 Init:0/1 0 1h 10.XXXX onap-vm5 <none> onap dev-vnfsdk-vnfsdk-init-postgres-2psxl 0/1 Init:Error 0 5h 10.XXXX onap-vm1 <none> Regards, Atif -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#15198): https://lists.onap.org/g/onap-discuss/message/15198 Mute This Topic: https://lists.onap.org/mt/29572781/21656 Group Owner: onap-discuss+ow...@lists.onap.org Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
