Hi All, We are not able to access oom portal after deploying Beijing release. We did see the portal coming up 2 days back, now it seems to be broken.
Here is the list of logs for your reference, it seems like 4 PODs failing to start with Image pull error and it continues to stay there for more than 2 hours. Could someone point us to any workaround to resolve this one? *Image pull error* onap dev-aaf-sms-vault-0 0/2 ErrImagePull 0 2h Failed to pull image "nexus3.onap.org:10001/library/consul:1.0.6": rpc error: code = Canceled desc = context canceled onap dev-nexus-6558979c95-dxcql 0/1 ImagePullBackOff 0 2h Failed to pull image "nexus3.onap.org:10001/sonatype/nexus:2.14.8-01": rpc error: code = Canceled desc = context canceled onap dev-portal-zookeeper-db466fc-vmlrv 0/1 ImagePullBackOff 0 2h Failed to pull image "nexus3.onap.org:10001/zookeeper:3.4": rpc error: code = Unknown desc = Error: image zookeeper:3.4 not found onap dev-vid-mariadb-galera-0 0/1 ErrImagePull 0 2h Failed to pull image " nexus3.onap.org:10001/adfinissygroup/k8s-mariadb-galera-centos:v002": rpc error: code = Unknown desc = Error: image adfinissygroup/k8s-mariadb-galera-centos:v002 not found *Containers struck with INIT* onap dev-drools-0 0/1 Init:0/1 1 2h onap dev-vid-7974559bc7-6bcbr 0/2 Init:0/1 7 2h *ONAP Portal cassandra logs (*kubectl logs -n onap dev-portal-cassandra-5ddbc59ffd-ndpd9) Connection error: ('Unable to connect to any servers', {'127.0.0.1': AuthenticationFailed('Failed to authenticate to 127.0.0.1: Error from server: code=0100 [Bad credentials] message="Username and/or password are incorrect"',)}) Cassandra is unavailable - sleeping Connection error: ('Unable to connect to any servers', {'127.0.0.1': AuthenticationFailed('Failed to authenticate to 127.0.0.1: Error from server: code=0100 [Bad credentials] message="Username and/or password are incorrect"',)}) Cassandra is unavailable - sleeping *All PODs* NAMESPACE NAME READY STATUS RESTARTS AGE kube-system heapster-76b8cd7b5-wcwbv 1/1 Running 0 2d kube-system kube-dns-5d7b4487c9-6rpzj 3/3 Running 0 2d kube-system kubernetes-dashboard-f9577fffd-dwb7m 1/1 Running 0 2d kube-system monitoring-grafana-997796fcf-mtqrc 1/1 Running 0 2d kube-system monitoring-influxdb-56fdcd96b-xf4b7 1/1 Running 1 2d kube-system tiller-deploy-54bcc55dd5-nj8px 1/1 Running 0 2d onap dev-aaf-cm-6685b46b6-k7dxj 1/1 Running 0 2h onap dev-aaf-cs-7b7648974c-6x2fn 1/1 Running 0 2h onap dev-aaf-fs-69cff7b886-svnxb 1/1 Running 0 2h onap dev-aaf-gui-5bf458c89f-fqf79 1/1 Running 0 2h onap dev-aaf-hello-6598bbd4c4-lcvmr 1/1 Running 0 2h onap dev-aaf-locate-6d66fbddd8-7wjhd 1/1 Running 0 2h onap dev-aaf-oauth-68cdd75bc7-s4rjb 1/1 Running 0 2h onap dev-aaf-service-75d964d8fc-q66wc 1/1 Running 0 2h onap dev-aaf-sms-78cc97648b-glkg5 0/1 CrashLoopBackOff 16 2h onap dev-aaf-sms-quorumclient-0 1/1 Running 0 2h onap dev-aaf-sms-quorumclient-1 1/1 Running 0 2h onap dev-aaf-sms-quorumclient-2 1/1 Running 0 2h onap dev-aaf-sms-vault-0 0/2 Error 0 2h onap dev-aai-5994d7f774-nknxf 1/1 Running 0 2h onap dev-aai-babel-d749d5db6-xmhgv 2/2 Running 0 2h onap dev-aai-cassandra-0 1/1 Running 0 2h onap dev-aai-cassandra-1 1/1 Running 0 2h onap dev-aai-cassandra-2 1/1 Running 0 2h onap dev-aai-champ-689595cfbb-2xhxn 0/1 Running 0 2h onap dev-aai-data-router-86cfd6c95b-f7qzr 1/1 Running 0 2h onap dev-aai-elasticsearch-548b68c46f-k7zjr 1/1 Running 0 2h onap dev-aai-gizmo-5799b67cdf-pgk9p 2/2 Running 0 2h onap dev-aai-hbase-868f949597-n8566 1/1 Running 3 2h onap dev-aai-modelloader-54ff794977-nsgps 2/2 Running 0 2h onap dev-aai-resources-d489cd699-c86zm 2/2 Running 0 2h onap dev-aai-search-data-788dd9458b-wzq2n 2/2 Running 0 2h onap dev-aai-sparky-be-6f997f65f9-gnxvq 2/2 Running 0 2h onap dev-aai-traversal-7bbc5c77d-d4s8j 2/2 Running 0 2h onap dev-appc-0 2/2 Running 0 2h onap dev-appc-cdt-cc7bd674f-qqlgv 1/1 Running 0 2h onap dev-appc-db-0 2/2 Running 0 2h onap dev-appc-dgbuilder-589bb888f8-qmpms 1/1 Running 0 2h onap dev-brmsgw-599dd84f76-xh5fm 1/1 Running 0 2h onap dev-cli-7ddb5b8849-rs7sc 1/1 Running 0 2h onap dev-consul-78dd6bd466-sc8qj 1/1 Running 0 2h onap dev-consul-server-0 1/1 Running 0 2h onap dev-consul-server-1 1/1 Running 0 2h onap dev-consul-server-2 1/1 Running 0 2h onap dev-dbc-pg-0 1/1 Running 0 2h onap dev-dbc-pg-1 1/1 Running 0 2h onap dev-dmaap-bus-controller-fcbcd76f6-th6ck 1/1 Running 0 2h onap dev-drools-0 0/1 Init:0/1 1 2h onap dev-esr-797474d445-c8xg7 2/2 Running 0 2h onap dev-esr-gui-75bd78fff6-8lfgr 1/1 Running 0 2h onap dev-kube2msb-55894945cf-8xwk4 1/1 Running 0 2h onap dev-log-elasticsearch-8595978ffd-9rh2z 1/1 Running 0 2h onap dev-log-kibana-59c97697f4-gqtks 1/1 Running 0 2h onap dev-log-logstash-4nt6j 1/1 Running 0 2h onap dev-message-router-559f5964c7-d2thk 1/1 Running 0 2h onap dev-message-router-kafka-748bbcb7cd-8w7f9 1/1 Running 0 2h onap dev-message-router-zookeeper-7bdff654b4-9wbwj 1/1 Running 0 2h onap dev-msb-consul-868f84f4d-v9lm5 1/1 Running 0 2h onap dev-msb-discovery-7d78d5948d-wcf5l 2/2 Running 0 2h onap dev-msb-eag-5bb6f644df-mf6q4 2/2 Running 0 2h onap dev-msb-iag-699f549dfd-gqsc7 2/2 Running 0 2h onap dev-nbi-6d57c747bc-rc2tp 1/1 Running 1 2h onap dev-nbi-mariadb-5b9597d49f-64zmp 1/1 Running 0 2h onap dev-nbi-mongo-0 1/1 Running 0 2h onap dev-nexus-6558979c95-dxcql 0/1 ImagePullBackOff 0 2h onap dev-oof-545746994f-t6lh8 1/1 Running 0 2h onap dev-oof-has-api-74ddd7dcf4-74h4d 1/1 Running 0 2h onap dev-oof-has-cassandra-94bd57d8f-ht7qb 1/1 Running 0 2h onap dev-oof-has-controller-6d9694ccd9-hhshw 1/1 Running 0 2h onap dev-oof-has-data-67776c78c-x4n8v 1/1 Running 0 2h onap dev-oof-has-music-8584d555fd-cmlxt 1/1 Running 0 2h onap dev-oof-has-reservation-d657ddcdf-xcvvm 1/1 Running 0 2h onap dev-oof-has-solver-7468767965-qbzm7 1/1 Running 0 2h onap dev-oof-has-zookeeper-7dd694b949-pxjbv 1/1 Running 0 2h onap dev-pap-5c68c6d847-6z4v6 2/2 Running 0 2h onap dev-pdp-0 2/2 Running 0 2h onap dev-policydb-7d4b75869-fhqv9 1/1 Running 0 2h onap dev-portal-app-54f8bc85d8-5skqr 2/2 Running 0 2h onap dev-portal-cassandra-5ddbc59ffd-ndpd9 1/1 Running 0 2h onap dev-portal-db-84f48ddccb-pg2nz 1/1 Running 0 2h onap dev-portal-sdk-756d9597b7-gvxmn 2/2 Running 0 2h onap dev-portal-widget-7dc7c978c7-mpcnk 1/1 Running 0 2h onap dev-portal-zookeeper-db466fc-vmlrv 0/1 ImagePullBackOff 0 2h onap dev-robot-7776fd8cd4-25j68 1/1 Running 0 2h onap dev-sdc-be-5cf5bc989f-dzm5m 2/2 Running 0 2h onap dev-sdc-cs-7ffdb996cf-jvcl2 1/1 Running 0 2h onap dev-sdc-es-588646d89f-wtv2h 1/1 Running 0 2h onap dev-sdc-fe-644b4bbc5b-gdnwn 2/2 Running 0 2h onap dev-sdc-kb-7946d5ccbd-m5sd6 1/1 Running 0 2h onap dev-sdc-onboarding-be-788c5ddd4d-m4dxb 2/2 Running 0 2h onap dev-sdc-wfd-9b9c67b79-ltmd7 1/1 Running 0 2h onap dev-sdnc-0 2/2 Running 0 2h onap dev-sdnc-ansible-server-8447bdbd8c-fkrrd 1/1 Running 0 2h onap dev-sdnc-db-0 2/2 Running 0 2h onap dev-sdnc-dgbuilder-d4888dbb6-qmddr 1/1 Running 0 2h onap dev-sdnc-dmaap-listener-5467c788cb-v754p 1/1 Running 0 2h onap dev-sdnc-portal-7df4c655df-qt8pq 1/1 Running 0 2h onap dev-sdnc-ueb-listener-6b6976795f-vrpr4 1/1 Running 0 2h onap dev-so-cc9fff84f-27w6c 2/2 Running 0 2h onap dev-so-db-6b654bb7dd-nnnxk 1/1 Running 0 2h onap dev-uui-585f89c868-w55ck 1/1 Running 0 2h onap dev-uui-server-fcbb88d7f-qbrdb 1/1 Running 0 2h onap dev-vid-7974559bc7-6bcbr 0/2 Init:0/1 7 2h onap dev-vid-mariadb-galera-0 0/1 ImagePullBackOff 0 2h *All services* NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE aaf-cass ClusterIP None <none> 7000/TCP,7001/TCP,9042/TCP,9160/TCP 2h aaf-cm ClusterIP 10.43.225.151 <none> 8150/TCP 2h aaf-fs ClusterIP 10.43.80.175 <none> 8096/TCP 2h aaf-gui NodePort 10.43.84.41 <none> 8200:30251/TCP 2h aaf-hello ClusterIP 10.43.205.209 <none> 8130/TCP 2h aaf-locate ClusterIP 10.43.24.186 <none> 8095/TCP 2h aaf-oauth ClusterIP 10.43.50.232 <none> 8140/TCP 2h aaf-service NodePort 10.43.60.155 <none> 8100:30247/TCP 2h aaf-sms NodePort 10.43.254.175 <none> 10443:30243/TCP 2h aaf-sms-db NodePort 10.43.219.6 <none> 8200:30244/TCP 2h aai NodePort 10.43.225.195 <none> 8080:30232/TCP,8443:30233/TCP 2h aai-babel NodePort 10.43.217.175 <none> 9516:30279/TCP 2h aai-cassandra ClusterIP None <none> 9042/TCP,9160/TCP,61621/TCP 2h aai-champ NodePort 10.43.171.87 <none> 9522:30278/TCP 2h aai-crud-service NodePort 10.43.197.217 <none> 9520:30268/TCP 2h aai-elasticsearch ClusterIP None <none> 9200/TCP 2h aai-hbase ClusterIP None <none> 2181/TCP,8080/TCP,8085/TCP,9090/TCP,16000/TCP,16010/TCP,16201/TCP 2h aai-modelloader NodePort 10.43.228.147 <none> 8080:30210/TCP,8443:30229/TCP 2h aai-resources ClusterIP None <none> 8447/TCP,5005/TCP 2h aai-search-data ClusterIP None <none> 9509/TCP 2h aai-sparky-be ClusterIP None <none> 9517/TCP 2h aai-traversal ClusterIP None <none> 8446/TCP,5005/TCP 2h appc NodePort 10.43.228.180 <none> 8282:30230/TCP,1830:30231/TCP,9090:30211/TCP 2h appc-cdt NodePort 10.43.211.244 <none> 80:30289/TCP 2h appc-cluster ClusterIP None <none> 2550/TCP 2h appc-dbhost ClusterIP None <none> 3306/TCP 2h appc-dbhost-read ClusterIP 10.43.236.248 <none> 3306/TCP 2h appc-dgbuilder NodePort 10.43.122.70 <none> 3000:30228/TCP 2h appc-sdnctldb01 ClusterIP None <none> 3306/TCP 2h appc-sdnctldb02 ClusterIP None <none> 3306/TCP 2h brmsgw NodePort 10.43.165.37 <none> 9989:30216/TCP 2h cli NodePort 10.43.154.111 <none> 8080:30260/TCP,9090:30271/TCP 2h consul-server ClusterIP None <none> 8301/TCP 2h consul-server-ui NodePort 10.43.10.97 <none> 8500:30270/TCP 2h dbc-pg-primary ClusterIP 10.43.77.250 <none> 5432/TCP 2h dbc-pg-replica ClusterIP 10.43.120.99 <none> 5432/TCP 2h dbc-postgres ClusterIP 10.43.169.228 <none> 5432/TCP 2h dmaap-bc NodePort 10.43.57.81 <none> 8080:30241/TCP,8443:30242/TCP 2h drools NodePort 10.43.107.39 <none> 6969:30217/TCP,9696:30221/TCP 2h esr ClusterIP 10.43.227.131 <none> 9518/TCP 2h esr-gui ClusterIP 10.43.211.89 <none> 8080/TCP 2h log-es NodePort 10.43.222.235 <none> 9200:30254/TCP 2h log-es-tcp ClusterIP 10.43.215.107 <none> 9300/TCP 2h log-kibana NodePort 10.43.32.179 <none> 5601:30253/TCP 2h log-ls NodePort 10.43.111.132 <none> 5044:30255/TCP 2h log-ls-http ClusterIP 10.43.164.194 <none> 9600/TCP 2h message-router NodePort 10.43.65.221 <none> 3904:30227/TCP,3905:30226/TCP 2h message-router-kafka ClusterIP 10.43.13.17 <none> 9092/TCP 2h message-router-zookeeper ClusterIP None <none> 2181/TCP 2h msb-consul NodePort 10.43.242.228 <none> 8500:30285/TCP 2h msb-discovery NodePort 10.43.249.194 <none> 10081:30281/TCP 2h msb-eag NodePort 10.43.162.65 <none> 80:30282/TCP,443:30284/TCP 2h msb-iag NodePort 10.43.155.52 <none> 80:30280/TCP,443:30283/TCP 2h nbi NodePort 10.43.204.18 <none> 8080:30274/TCP 2h nbi-mariadbhost ClusterIP None <none> 3306/TCP 2h nbi-mongohost ClusterIP None <none> 27017/TCP 2h nbi-mongohost-read ClusterIP 10.43.226.216 <none> 27017/TCP 2h nexus NodePort 10.43.92.225 <none> 8081:30236/TCP 2h oof-has-api NodePort 10.43.26.71 <none> 8091:30275/TCP 2h oof-has-cassandra ClusterIP 10.43.211.135 <none> 9160/TCP,7000/TCP,7001/TCP,7199/TCP,9042/TCP 2h oof-has-music NodePort 10.43.146.101 <none> 8080:30276/TCP 2h oof-has-zk ClusterIP 10.43.220.122 <none> 2181/TCP 2h oof-osdf NodePort 10.43.40.4 <none> 8698:30248/TCP 2h pap NodePort 10.43.102.146 <none> 8443:30219/TCP,9091:30218/TCP 2h pdp ClusterIP None <none> 8081/TCP 2h policydb ClusterIP None <none> 3306/TCP 2h portal-app LoadBalancer 10.43.220.0 192.168.122.231 8989:30215/TCP,8006:30213/TCP,8010:30214/TCP 2h portal-cassandra ClusterIP 10.43.60.228 <none> 9160/TCP,7000/TCP,7001/TCP,7199/TCP,9042/TCP 2h portal-db ClusterIP 10.43.240.58 <none> 3306/TCP 2h portal-sdk ClusterIP 10.43.212.134 <none> 8990/TCP 2h portal-widget ClusterIP 10.43.217.146 <none> 8082/TCP 2h portal-zookeeper ClusterIP 10.43.152.159 <none> 2181/TCP 2h robot NodePort 10.43.13.169 <none> 88:30209/TCP 2h sdc-be NodePort 10.43.71.178 <none> 8443:30204/TCP,8080:30205/TCP 2h sdc-cs ClusterIP 10.43.75.106 <none> 9160/TCP,9042/TCP 2h sdc-es ClusterIP 10.43.129.250 <none> 9200/TCP,9300/TCP 2h sdc-fe NodePort 10.43.191.213 <none> 8181:30206/TCP,9443:30207/TCP 2h sdc-kb ClusterIP 10.43.82.109 <none> 5601/TCP 2h sdc-onboarding-be ClusterIP 10.43.71.172 <none> 8445/TCP,8081/TCP 2h sdc-wfd NodePort 10.43.166.95 <none> 8080:30256/TCP 2h sdnc NodePort 10.43.246.191 <none> 8282:30202/TCP,8202:30208/TCP,8280:30246/TCP,8443:30267/TCP 2h sdnc-ansible-server ClusterIP 10.43.135.93 <none> 8000/TCP 2h sdnc-cluster ClusterIP None <none> 2550/TCP 2h sdnc-dbhost ClusterIP None <none> 3306/TCP 2h sdnc-dbhost-read ClusterIP 10.43.133.28 <none> 3306/TCP 2h sdnc-dgbuilder NodePort 10.43.200.78 <none> 3000:30203/TCP 2h sdnc-dmaap-listener ClusterIP None <none> <none> 2h sdnc-portal NodePort 10.43.157.104 <none> 8843:30201/TCP 2h sdnc-sdnctldb01 ClusterIP None <none> 3306/TCP 2h sdnc-sdnctldb02 ClusterIP None <none> 3306/TCP 2h sdnc-ueb-listener ClusterIP None <none> <none> 2h so NodePort 10.43.3.198 <none> 8080:30223/TCP,3904:30225/TCP,3905:30224/TCP,9990:30222/TCP,8787:30250/TCP 2h so-db NodePort 10.43.234.60 <none> 3306:30252/TCP 2h uui NodePort 10.43.55.220 <none> 8080:30398/TCP 2h uui-server NodePort 10.43.73.195 <none> 8082:30399/TCP 2h vid NodePort 10.43.1.73 <none> 8080:30200/TCP 2h vid-galera ClusterIP None <none> 3306/TCP Regards Vivek -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#10551): https://lists.onap.org/g/onap-discuss/message/10551 Mute This Topic: https://lists.onap.org/mt/22677732/21656 Group Owner: onap-discuss+ow...@lists.onap.org Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-