Hi All,
We are not able to access oom portal after deploying Beijing release. We
did see the portal coming up 2 days back, now it seems to be broken.
Here is the list of logs for your reference, it seems like 4 PODs failing
to start with Image pull error and it continues to stay there for more than
2 hours.
Could someone point us to any workaround to resolve this one?
*Image pull error*
onap dev-aaf-sms-vault-0 0/2
ErrImagePull 0 2h
Failed to pull image "nexus3.onap.org:10001/library/consul:1.0.6":
rpc error: code = Canceled desc = context canceled
onap dev-nexus-6558979c95-dxcql 0/1
ImagePullBackOff 0 2h
Failed to pull image "nexus3.onap.org:10001/sonatype/nexus:2.14.8-01":
rpc error: code = Canceled desc = context canceled
onap dev-portal-zookeeper-db466fc-vmlrv 0/1
ImagePullBackOff 0 2h
Failed to pull image "nexus3.onap.org:10001/zookeeper:3.4": rpc
error: code = Unknown desc = Error: image zookeeper:3.4 not found
onap dev-vid-mariadb-galera-0 0/1
ErrImagePull 0 2h
Failed to pull image "
nexus3.onap.org:10001/adfinissygroup/k8s-mariadb-galera-centos:v002": rpc
error: code = Unknown desc = Error: image
adfinissygroup/k8s-mariadb-galera-centos:v002 not found
*Containers struck with INIT*
onap dev-drools-0 0/1
Init:0/1 1 2h
onap dev-vid-7974559bc7-6bcbr 0/2
Init:0/1 7 2h
*ONAP Portal cassandra logs (*kubectl logs -n onap
dev-portal-cassandra-5ddbc59ffd-ndpd9)
Connection error: ('Unable to connect to any servers', {'127.0.0.1':
AuthenticationFailed('Failed to authenticate to 127.0.0.1: Error from
server: code=0100 [Bad credentials] message="Username and/or password are
incorrect"',)})
Cassandra is unavailable - sleeping
Connection error: ('Unable to connect to any servers', {'127.0.0.1':
AuthenticationFailed('Failed to authenticate to 127.0.0.1: Error from
server: code=0100 [Bad credentials] message="Username and/or password are
incorrect"',)})
Cassandra is unavailable - sleeping
*All PODs*
NAMESPACE NAME READY
STATUS RESTARTS AGE
kube-system heapster-76b8cd7b5-wcwbv 1/1
Running 0 2d
kube-system kube-dns-5d7b4487c9-6rpzj 3/3
Running 0 2d
kube-system kubernetes-dashboard-f9577fffd-dwb7m 1/1
Running 0 2d
kube-system monitoring-grafana-997796fcf-mtqrc 1/1
Running 0 2d
kube-system monitoring-influxdb-56fdcd96b-xf4b7 1/1
Running 1 2d
kube-system tiller-deploy-54bcc55dd5-nj8px 1/1
Running 0 2d
onap dev-aaf-cm-6685b46b6-k7dxj 1/1
Running 0 2h
onap dev-aaf-cs-7b7648974c-6x2fn 1/1
Running 0 2h
onap dev-aaf-fs-69cff7b886-svnxb 1/1
Running 0 2h
onap dev-aaf-gui-5bf458c89f-fqf79 1/1
Running 0 2h
onap dev-aaf-hello-6598bbd4c4-lcvmr 1/1
Running 0 2h
onap dev-aaf-locate-6d66fbddd8-7wjhd 1/1
Running 0 2h
onap dev-aaf-oauth-68cdd75bc7-s4rjb 1/1
Running 0 2h
onap dev-aaf-service-75d964d8fc-q66wc 1/1
Running 0 2h
onap dev-aaf-sms-78cc97648b-glkg5 0/1
CrashLoopBackOff 16 2h
onap dev-aaf-sms-quorumclient-0 1/1
Running 0 2h
onap dev-aaf-sms-quorumclient-1 1/1
Running 0 2h
onap dev-aaf-sms-quorumclient-2 1/1
Running 0 2h
onap dev-aaf-sms-vault-0 0/2
Error 0 2h
onap dev-aai-5994d7f774-nknxf 1/1
Running 0 2h
onap dev-aai-babel-d749d5db6-xmhgv 2/2
Running 0 2h
onap dev-aai-cassandra-0 1/1
Running 0 2h
onap dev-aai-cassandra-1 1/1
Running 0 2h
onap dev-aai-cassandra-2 1/1
Running 0 2h
onap dev-aai-champ-689595cfbb-2xhxn 0/1
Running 0 2h
onap dev-aai-data-router-86cfd6c95b-f7qzr 1/1
Running 0 2h
onap dev-aai-elasticsearch-548b68c46f-k7zjr 1/1
Running 0 2h
onap dev-aai-gizmo-5799b67cdf-pgk9p 2/2
Running 0 2h
onap dev-aai-hbase-868f949597-n8566 1/1
Running 3 2h
onap dev-aai-modelloader-54ff794977-nsgps 2/2
Running 0 2h
onap dev-aai-resources-d489cd699-c86zm 2/2
Running 0 2h
onap dev-aai-search-data-788dd9458b-wzq2n 2/2
Running 0 2h
onap dev-aai-sparky-be-6f997f65f9-gnxvq 2/2
Running 0 2h
onap dev-aai-traversal-7bbc5c77d-d4s8j 2/2
Running 0 2h
onap dev-appc-0 2/2
Running 0 2h
onap dev-appc-cdt-cc7bd674f-qqlgv 1/1
Running 0 2h
onap dev-appc-db-0 2/2
Running 0 2h
onap dev-appc-dgbuilder-589bb888f8-qmpms 1/1
Running 0 2h
onap dev-brmsgw-599dd84f76-xh5fm 1/1
Running 0 2h
onap dev-cli-7ddb5b8849-rs7sc 1/1
Running 0 2h
onap dev-consul-78dd6bd466-sc8qj 1/1
Running 0 2h
onap dev-consul-server-0 1/1
Running 0 2h
onap dev-consul-server-1 1/1
Running 0 2h
onap dev-consul-server-2 1/1
Running 0 2h
onap dev-dbc-pg-0 1/1
Running 0 2h
onap dev-dbc-pg-1 1/1
Running 0 2h
onap dev-dmaap-bus-controller-fcbcd76f6-th6ck 1/1
Running 0 2h
onap dev-drools-0 0/1
Init:0/1 1 2h
onap dev-esr-797474d445-c8xg7 2/2
Running 0 2h
onap dev-esr-gui-75bd78fff6-8lfgr 1/1
Running 0 2h
onap dev-kube2msb-55894945cf-8xwk4 1/1
Running 0 2h
onap dev-log-elasticsearch-8595978ffd-9rh2z 1/1
Running 0 2h
onap dev-log-kibana-59c97697f4-gqtks 1/1
Running 0 2h
onap dev-log-logstash-4nt6j 1/1
Running 0 2h
onap dev-message-router-559f5964c7-d2thk 1/1
Running 0 2h
onap dev-message-router-kafka-748bbcb7cd-8w7f9 1/1
Running 0 2h
onap dev-message-router-zookeeper-7bdff654b4-9wbwj 1/1
Running 0 2h
onap dev-msb-consul-868f84f4d-v9lm5 1/1
Running 0 2h
onap dev-msb-discovery-7d78d5948d-wcf5l 2/2
Running 0 2h
onap dev-msb-eag-5bb6f644df-mf6q4 2/2
Running 0 2h
onap dev-msb-iag-699f549dfd-gqsc7 2/2
Running 0 2h
onap dev-nbi-6d57c747bc-rc2tp 1/1
Running 1 2h
onap dev-nbi-mariadb-5b9597d49f-64zmp 1/1
Running 0 2h
onap dev-nbi-mongo-0 1/1
Running 0 2h
onap dev-nexus-6558979c95-dxcql 0/1
ImagePullBackOff 0 2h
onap dev-oof-545746994f-t6lh8 1/1
Running 0 2h
onap dev-oof-has-api-74ddd7dcf4-74h4d 1/1
Running 0 2h
onap dev-oof-has-cassandra-94bd57d8f-ht7qb 1/1
Running 0 2h
onap dev-oof-has-controller-6d9694ccd9-hhshw 1/1
Running 0 2h
onap dev-oof-has-data-67776c78c-x4n8v 1/1
Running 0 2h
onap dev-oof-has-music-8584d555fd-cmlxt 1/1
Running 0 2h
onap dev-oof-has-reservation-d657ddcdf-xcvvm 1/1
Running 0 2h
onap dev-oof-has-solver-7468767965-qbzm7 1/1
Running 0 2h
onap dev-oof-has-zookeeper-7dd694b949-pxjbv 1/1
Running 0 2h
onap dev-pap-5c68c6d847-6z4v6 2/2
Running 0 2h
onap dev-pdp-0 2/2
Running 0 2h
onap dev-policydb-7d4b75869-fhqv9 1/1
Running 0 2h
onap dev-portal-app-54f8bc85d8-5skqr 2/2
Running 0 2h
onap dev-portal-cassandra-5ddbc59ffd-ndpd9 1/1
Running 0 2h
onap dev-portal-db-84f48ddccb-pg2nz 1/1
Running 0 2h
onap dev-portal-sdk-756d9597b7-gvxmn 2/2
Running 0 2h
onap dev-portal-widget-7dc7c978c7-mpcnk 1/1
Running 0 2h
onap dev-portal-zookeeper-db466fc-vmlrv 0/1
ImagePullBackOff 0 2h
onap dev-robot-7776fd8cd4-25j68 1/1
Running 0 2h
onap dev-sdc-be-5cf5bc989f-dzm5m 2/2
Running 0 2h
onap dev-sdc-cs-7ffdb996cf-jvcl2 1/1
Running 0 2h
onap dev-sdc-es-588646d89f-wtv2h 1/1
Running 0 2h
onap dev-sdc-fe-644b4bbc5b-gdnwn 2/2
Running 0 2h
onap dev-sdc-kb-7946d5ccbd-m5sd6 1/1
Running 0 2h
onap dev-sdc-onboarding-be-788c5ddd4d-m4dxb 2/2
Running 0 2h
onap dev-sdc-wfd-9b9c67b79-ltmd7 1/1
Running 0 2h
onap dev-sdnc-0 2/2
Running 0 2h
onap dev-sdnc-ansible-server-8447bdbd8c-fkrrd 1/1
Running 0 2h
onap dev-sdnc-db-0 2/2
Running 0 2h
onap dev-sdnc-dgbuilder-d4888dbb6-qmddr 1/1
Running 0 2h
onap dev-sdnc-dmaap-listener-5467c788cb-v754p 1/1
Running 0 2h
onap dev-sdnc-portal-7df4c655df-qt8pq 1/1
Running 0 2h
onap dev-sdnc-ueb-listener-6b6976795f-vrpr4 1/1
Running 0 2h
onap dev-so-cc9fff84f-27w6c 2/2
Running 0 2h
onap dev-so-db-6b654bb7dd-nnnxk 1/1
Running 0 2h
onap dev-uui-585f89c868-w55ck 1/1
Running 0 2h
onap dev-uui-server-fcbb88d7f-qbrdb 1/1
Running 0 2h
onap dev-vid-7974559bc7-6bcbr 0/2
Init:0/1 7 2h
onap dev-vid-mariadb-galera-0 0/1
ImagePullBackOff 0 2h
*All services*
NAME TYPE CLUSTER-IP EXTERNAL-IP
PORT(S)
AGE
aaf-cass ClusterIP None <none>
7000/TCP,7001/TCP,9042/TCP,9160/TCP
2h
aaf-cm ClusterIP 10.43.225.151 <none>
8150/TCP
2h
aaf-fs ClusterIP 10.43.80.175 <none>
8096/TCP
2h
aaf-gui NodePort 10.43.84.41 <none>
8200:30251/TCP
2h
aaf-hello ClusterIP 10.43.205.209 <none>
8130/TCP
2h
aaf-locate ClusterIP 10.43.24.186 <none>
8095/TCP
2h
aaf-oauth ClusterIP 10.43.50.232 <none>
8140/TCP
2h
aaf-service NodePort 10.43.60.155 <none>
8100:30247/TCP
2h
aaf-sms NodePort 10.43.254.175 <none>
10443:30243/TCP
2h
aaf-sms-db NodePort 10.43.219.6 <none>
8200:30244/TCP
2h
aai NodePort 10.43.225.195 <none>
8080:30232/TCP,8443:30233/TCP
2h
aai-babel NodePort 10.43.217.175 <none>
9516:30279/TCP
2h
aai-cassandra ClusterIP None <none>
9042/TCP,9160/TCP,61621/TCP
2h
aai-champ NodePort 10.43.171.87 <none>
9522:30278/TCP
2h
aai-crud-service NodePort 10.43.197.217 <none>
9520:30268/TCP
2h
aai-elasticsearch ClusterIP None <none>
9200/TCP
2h
aai-hbase ClusterIP None <none>
2181/TCP,8080/TCP,8085/TCP,9090/TCP,16000/TCP,16010/TCP,16201/TCP
2h
aai-modelloader NodePort 10.43.228.147 <none>
8080:30210/TCP,8443:30229/TCP
2h
aai-resources ClusterIP None <none>
8447/TCP,5005/TCP
2h
aai-search-data ClusterIP None <none>
9509/TCP
2h
aai-sparky-be ClusterIP None <none>
9517/TCP
2h
aai-traversal ClusterIP None <none>
8446/TCP,5005/TCP
2h
appc NodePort 10.43.228.180 <none>
8282:30230/TCP,1830:30231/TCP,9090:30211/TCP
2h
appc-cdt NodePort 10.43.211.244 <none>
80:30289/TCP
2h
appc-cluster ClusterIP None <none>
2550/TCP
2h
appc-dbhost ClusterIP None <none>
3306/TCP
2h
appc-dbhost-read ClusterIP 10.43.236.248 <none>
3306/TCP
2h
appc-dgbuilder NodePort 10.43.122.70 <none>
3000:30228/TCP
2h
appc-sdnctldb01 ClusterIP None <none>
3306/TCP
2h
appc-sdnctldb02 ClusterIP None <none>
3306/TCP
2h
brmsgw NodePort 10.43.165.37 <none>
9989:30216/TCP
2h
cli NodePort 10.43.154.111 <none>
8080:30260/TCP,9090:30271/TCP
2h
consul-server ClusterIP None <none>
8301/TCP
2h
consul-server-ui NodePort 10.43.10.97 <none>
8500:30270/TCP
2h
dbc-pg-primary ClusterIP 10.43.77.250 <none>
5432/TCP
2h
dbc-pg-replica ClusterIP 10.43.120.99 <none>
5432/TCP
2h
dbc-postgres ClusterIP 10.43.169.228 <none>
5432/TCP
2h
dmaap-bc NodePort 10.43.57.81 <none>
8080:30241/TCP,8443:30242/TCP
2h
drools NodePort 10.43.107.39 <none>
6969:30217/TCP,9696:30221/TCP
2h
esr ClusterIP 10.43.227.131 <none>
9518/TCP
2h
esr-gui ClusterIP 10.43.211.89 <none>
8080/TCP
2h
log-es NodePort 10.43.222.235 <none>
9200:30254/TCP
2h
log-es-tcp ClusterIP 10.43.215.107 <none>
9300/TCP
2h
log-kibana NodePort 10.43.32.179 <none>
5601:30253/TCP
2h
log-ls NodePort 10.43.111.132 <none>
5044:30255/TCP
2h
log-ls-http ClusterIP 10.43.164.194 <none>
9600/TCP
2h
message-router NodePort 10.43.65.221 <none>
3904:30227/TCP,3905:30226/TCP
2h
message-router-kafka ClusterIP 10.43.13.17 <none>
9092/TCP
2h
message-router-zookeeper ClusterIP None <none>
2181/TCP
2h
msb-consul NodePort 10.43.242.228 <none>
8500:30285/TCP
2h
msb-discovery NodePort 10.43.249.194 <none>
10081:30281/TCP
2h
msb-eag NodePort 10.43.162.65 <none>
80:30282/TCP,443:30284/TCP
2h
msb-iag NodePort 10.43.155.52 <none>
80:30280/TCP,443:30283/TCP
2h
nbi NodePort 10.43.204.18 <none>
8080:30274/TCP
2h
nbi-mariadbhost ClusterIP None <none>
3306/TCP
2h
nbi-mongohost ClusterIP None <none>
27017/TCP
2h
nbi-mongohost-read ClusterIP 10.43.226.216 <none>
27017/TCP
2h
nexus NodePort 10.43.92.225 <none>
8081:30236/TCP
2h
oof-has-api NodePort 10.43.26.71 <none>
8091:30275/TCP
2h
oof-has-cassandra ClusterIP 10.43.211.135 <none>
9160/TCP,7000/TCP,7001/TCP,7199/TCP,9042/TCP
2h
oof-has-music NodePort 10.43.146.101 <none>
8080:30276/TCP
2h
oof-has-zk ClusterIP 10.43.220.122 <none>
2181/TCP
2h
oof-osdf NodePort 10.43.40.4 <none>
8698:30248/TCP
2h
pap NodePort 10.43.102.146 <none>
8443:30219/TCP,9091:30218/TCP
2h
pdp ClusterIP None <none>
8081/TCP
2h
policydb ClusterIP None <none>
3306/TCP
2h
portal-app LoadBalancer 10.43.220.0 192.168.122.231
8989:30215/TCP,8006:30213/TCP,8010:30214/TCP
2h
portal-cassandra ClusterIP 10.43.60.228 <none>
9160/TCP,7000/TCP,7001/TCP,7199/TCP,9042/TCP
2h
portal-db ClusterIP 10.43.240.58 <none>
3306/TCP
2h
portal-sdk ClusterIP 10.43.212.134 <none>
8990/TCP
2h
portal-widget ClusterIP 10.43.217.146 <none>
8082/TCP
2h
portal-zookeeper ClusterIP 10.43.152.159 <none>
2181/TCP
2h
robot NodePort 10.43.13.169 <none>
88:30209/TCP
2h
sdc-be NodePort 10.43.71.178 <none>
8443:30204/TCP,8080:30205/TCP
2h
sdc-cs ClusterIP 10.43.75.106 <none>
9160/TCP,9042/TCP
2h
sdc-es ClusterIP 10.43.129.250 <none>
9200/TCP,9300/TCP
2h
sdc-fe NodePort 10.43.191.213 <none>
8181:30206/TCP,9443:30207/TCP
2h
sdc-kb ClusterIP 10.43.82.109 <none>
5601/TCP
2h
sdc-onboarding-be ClusterIP 10.43.71.172 <none>
8445/TCP,8081/TCP
2h
sdc-wfd NodePort 10.43.166.95 <none>
8080:30256/TCP
2h
sdnc NodePort 10.43.246.191 <none>
8282:30202/TCP,8202:30208/TCP,8280:30246/TCP,8443:30267/TCP
2h
sdnc-ansible-server ClusterIP 10.43.135.93 <none>
8000/TCP
2h
sdnc-cluster ClusterIP None <none>
2550/TCP
2h
sdnc-dbhost ClusterIP None <none>
3306/TCP
2h
sdnc-dbhost-read ClusterIP 10.43.133.28 <none>
3306/TCP
2h
sdnc-dgbuilder NodePort 10.43.200.78 <none>
3000:30203/TCP
2h
sdnc-dmaap-listener ClusterIP None <none>
<none>
2h
sdnc-portal NodePort 10.43.157.104 <none>
8843:30201/TCP
2h
sdnc-sdnctldb01 ClusterIP None <none>
3306/TCP
2h
sdnc-sdnctldb02 ClusterIP None <none>
3306/TCP
2h
sdnc-ueb-listener ClusterIP None <none>
<none>
2h
so NodePort 10.43.3.198 <none>
8080:30223/TCP,3904:30225/TCP,3905:30224/TCP,9990:30222/TCP,8787:30250/TCP
2h
so-db NodePort 10.43.234.60 <none>
3306:30252/TCP
2h
uui NodePort 10.43.55.220 <none>
8080:30398/TCP
2h
uui-server NodePort 10.43.73.195 <none>
8082:30399/TCP
2h
vid NodePort 10.43.1.73 <none>
8080:30200/TCP
2h
vid-galera ClusterIP None <none>
3306/TCP
Regards
Vivek
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#10551): https://lists.onap.org/g/onap-discuss/message/10551
Mute This Topic: https://lists.onap.org/mt/22677732/21656
Group Owner: [email protected]
Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
