Hi, Kapil/Team, I just tried to bring up cds pods in my ONAP cluster (Frankfurt). But I always get dev-cds-listener failure. Any insight? Please help.
dev-cds-blueprints-processor-788f99666f-xq99x 1/1 Running 0 23m 10.42.1.150 onap-k8s-vm-2 <none> <none> dev-cds-command-executor-75bf75c564-bfcb4 1/1 Running 0 23m 10.42.2.248 onap-k8s-vm-1 <none> <none> dev-cds-db-0 1/1 Running 0 23m 10.42.2.250 onap-k8s-vm-1 <none> <none> dev-cds-py-executor-6d5dd756f-fr68c 1/1 Running 0 23m 10.42.2.247 onap-k8s-vm-1 <none> <none> dev-cds-sdc-listener-678cbdcd4d-x6rkx 0/1 CrashLoopBackOff 8 23m 10.42.2.246 onap-k8s-vm-1 <none> <none> dev-cds-ui-6d55754b65-zkrfb 1/1 Running 0 23m 10.42.2.249 onap-k8s-vm-1 <none> <none> Here is the log: ------------------------------------------------------------------------------------------------------------------------------- ubuntu@onap-rancher-vm:~$ kubectl logs -n onap dev-cds-sdc-listener-678cbdcd4d-x6rkx Starting SDC Listener APP Config HOME : /opt/app/onap/config 20:58:33,624 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml] 20:58:33,624 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.groovy] 20:58:33,624 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [jar:file:/opt/app/onap/lib/org.onap.ccsdk.cds.sdclistener-application-0.7.2.jar!/logback.xml] 20:58:33,625 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs multiple times on the classpath. 20:58:33,625 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs at [jar:file:/opt/app/onap/lib/org.onap.ccsdk.cds.sdclistener-application-0.7.2.jar!/logback.xml] 20:58:33,625 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs at [file:/opt/app/onap/config/logback.xml] 20:58:33,643 |-INFO in ch.qos.logback.core.joran.spi.ConfigurationWatchList@77cd7a0 - URL [jar:file:/opt/app/onap/lib/org.onap.ccsdk.cds.sdclistener-application-0.7.2.jar!/logback.xml] is not of type file 20:58:33,705 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - debug attribute not set 20:58:33,706 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender] 20:58:33,715 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [STDOUT] 20:58:33,723 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property 20:58:33,790 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to INFO 20:58:33,790 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to Logger[ROOT] 20:58:33,792 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.springframework] to INFO 20:58:33,792 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.springframework.web] to INFO 20:58:33,792 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.onap.ccsdk.cds] to INFO 20:58:33,792 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration. 20:58:33,794 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@204f30ec - Registering current configuration as safe fallback point . ____ _ __ _ _ /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \ ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \ \\/ ___)| |_)| | | | | || (_| | ) ) ) ) ' |____| .__|_| |_|_| |_\__, | / / / / =========|_|==============|___/=/_/_/_/ :: Spring Boot :: (v2.2.4.RELEASE) 2020-08-05 20:58:34,624|||main||||INFO||||||| Starting SdcListenerApplication v0.7.2 on dev-cds-sdc-listener-678cbdcd4d-x6rkx with PID 6 (/opt/app/onap/lib/org.onap.ccsdk.cds.sdclistener-application-0.7.2.jar started by onap in /) 2020-08-05 20:58:34,628|||main||||INFO||||||| No active profile set, falling back to default profiles: default 2020-08-05 20:58:38,466|||main||||INFO||||||| Exposing 13 endpoint(s) beneath base path '/actuator' 2020-08-05 20:58:38,813|||main||||INFO||||||| Using generated security password: 035a7a05-4dc7-4de8-a9bb-29f774c4446c 2020-08-05 20:58:39,162|||main||||INFO||||||| Netty started on port(s): 8080 2020-08-05 20:58:39,165|||main||||INFO||||||| Started SdcListenerApplication in 5.163 seconds (JVM running for 5.901) 2020-08-05 20:58:39,167|||main||||INFO||||||| Initialize the SDC distribution client 2020-08-05 20:58:40,062|||main||||ERROR||||||| status from ASDC is org.onap.sdc.http.HttpAsdcResponse@502c2278 2020-08-05 20:58:40,063|||main||||ERROR||||||| DistributionClientResultImpl [responseStatus=ASDC_SERVER_PROBLEM, responseMessage=ASDC server problem] 2020-08-05 20:58:40,074|||main||||WARN||||||| DELETE http://message-router.onap:3904/apiKeys/EocCVzI1y7IlLBzc will send credentials over a clear channel. 2020-08-05 20:58:40,129|||main||||ERROR||||||| Application run failed java.lang.reflect.UndeclaredThrowableException: Failed to invoke event listener method HandlerMethod details: Bean [org.onap.ccsdk.cds.sdclistener.client.SdcListenerClient] Method [public void org.onap.ccsdk.cds.sdclistener.client.SdcListenerClient.initSdcClient() throws org.onap.ccsdk.cds.sdclistener.exceptions.SdcListenerException] Resolved arguments: at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:322) at org.springframework.context.event.ApplicationListenerMethodAdapter.processEvent(ApplicationListenerMethodAdapter.java:190) at org.springframework.context.event.ApplicationListenerMethodAdapter.onApplicationEvent(ApplicationListenerMethodAdapter.java:153) at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172) at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165) at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139) at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:403) at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:360) at org.springframework.boot.context.event.EventPublishingRunListener.running(EventPublishingRunListener.java:103) at org.springframework.boot.SpringApplicationRunListeners.running(SpringApplicationRunListeners.java:77) at org.springframework.boot.SpringApplication.run(SpringApplication.java:330) at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226) at org.springframework.boot.SpringApplication.run(SpringApplication.java:1215) at org.onap.ccsdk.cds.sdclistener.SdcListenerApplication.main(SdcListenerApplication.java:30) Caused by: org.onap.ccsdk.cds.sdclistener.exceptions.SdcListenerException: SDC distribution client init failed with reason:ASDC server problem at org.onap.ccsdk.cds.sdclistener.client.SdcListenerClient.startSdcClientBasedOnTheResult(SdcListenerClient.java:75) at org.onap.ccsdk.cds.sdclistener.client.SdcListenerClient.initSdcClient(SdcListenerClient.java:69) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:305) ... 13 common frames omitted 2020-08-05 20:58:40,136|||main||||INFO||||||| Stopping GRPC connection to CDS backend And description: ubuntu@onap-rancher-vm:~$ kubectl describe po -n onap dev-cds-sdc-listener-678cbdcd4d-x6rkx Name: dev-cds-sdc-listener-678cbdcd4d-x6rkx Namespace: onap Priority: 0 Node: onap-k8s-vm-1/167.254.204.51 Start Time: Wed, 05 Aug 2020 20:49:43 +0000 Labels: app=cds-sdc-listener pod-template-hash=678cbdcd4d release=dev Annotations: cni.projectcalico.org/podIP: 10.42.2.246/32 Status: Running IP: 10.42.2.246 Controlled By: ReplicaSet/dev-cds-sdc-listener-678cbdcd4d Init Containers: cds-sdc-listener-readiness: Container ID: docker://773097670ad4689e0765084b05fc965bc2d2d2a18f2b07c83c9c7d6504ec18fe Image: oomk8s/readiness-check:2.0.2 Image ID: docker-pullable://oomk8s/readiness-check@sha256:875814cc853d7d9b3227165740584c347884bc8c0d831b4148cb5f7952e4cbff Port: <none> Host Port: <none> Command: /root/ready.py Args: --container-name sdc-be --container-name message-router --container-name cds-blueprints-processor State: Terminated Reason: Completed Exit Code: 0 Started: Wed, 05 Aug 2020 20:50:03 +0000 Finished: Wed, 05 Aug 2020 20:53:55 +0000 Ready: True Restart Count: 0 Environment: NAMESPACE: onap (v1:metadata.namespace) Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-2qr2x (ro) Containers: cds-sdc-listener: Container ID: docker://3b5432c3e7b6eaeeb57b0d9c0b372963f0ac205f6932c82c25363bbb499c9a45 Image: nexus3.onap.org:10001/onap/ccsdk-sdclistener:0.7.2 Image ID: docker-pullable://nexus3.onap.org:10001/onap/ccsdk-sdclistener@sha256:50b33cb972a20a2a56a5bf997c852a1f4824be77ed71669181af359f69bf1f5e Port: 8080/TCP Host Port: 0/TCP State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Error Exit Code: 1 Started: Wed, 05 Aug 2020 20:58:33 +0000 Finished: Wed, 05 Aug 2020 20:58:43 +0000 Ready: False Restart Count: 5 Liveness: http-get http://:8080/api/v1/sdclistener/healthcheck delay=10s timeout=1s period=10s #success=1 #failure=3 Readiness: http-get http://:8080/api/v1/sdclistener/healthcheck delay=10s timeout=1s period=10s #success=1 #failure=3 Environment: APP_CONFIG_HOME: /opt/app/onap/config Mounts: /opt/app/onap/config/application.yaml from dev-cds-sdc-listener-config (rw,path="application.yaml") /opt/app/onap/config/logback.xml from dev-cds-sdc-listener-config (rw,path="logback.xml") /var/run/secrets/kubernetes.io/serviceaccount from default-token-2qr2x (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: dev-cds-sdc-listener-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: dev-cds-sdc-listener-configmap Optional: false default-token-2qr2x: Type: Secret (a volume populated by a Secret) SecretName: default-token-2qr2x Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Pulled 14m kubelet, onap-k8s-vm-1 Container image "oomk8s/readiness-check:2.0.2" already present on machine Normal Created 14m kubelet, onap-k8s-vm-1 Created container cds-sdc-listener-readiness Normal Started 14m kubelet, onap-k8s-vm-1 Started container cds-sdc-listener-readiness Normal Scheduled 10m default-scheduler Successfully assigned onap/dev-cds-sdc-listener-678cbdcd4d-x6rkx to onap-k8s-vm-1 Normal Pulling 10m kubelet, onap-k8s-vm-1 Pulling image "nexus3.onap.org:10001/onap/ccsdk-sdclistener:0.7.2" Normal Pulled 9m53s kubelet, onap-k8s-vm-1 Successfully pulled image "nexus3.onap.org:10001/onap/ccsdk-sdclistener:0.7.2" Warning Unhealthy 9m32s kubelet, onap-k8s-vm-1 Readiness probe failed: Get http://10.42.2.246:8080/api/v1/sdclistener/healthcheck: dial tcp 10.42.2.246:8080: connect: connection refused Warning Unhealthy 9m30s kubelet, onap-k8s-vm-1 Liveness probe failed: Get http://10.42.2.246:8080/api/v1/sdclistener/healthcheck: dial tcp 10.42.2.246:8080: connect: connection refused Normal Pulled 8m13s (x3 over 9m28s) kubelet, onap-k8s-vm-1 Container image "nexus3.onap.org:10001/onap/ccsdk-sdclistener:0.7.2" already present on machine Normal Created 8m12s (x4 over 9m45s) kubelet, onap-k8s-vm-1 Created container cds-sdc-listener Normal Started 8m11s (x4 over 9m43s) kubelet, onap-k8s-vm-1 Started container cds-sdc-listener Warning BackOff 4m13s (x23 over 9m15s) kubelet, onap-k8s-vm-1 Back-off restarting failed container Thanks, Xin Miao Solution Engineering Fujitsu Network Communication (W)972-479-2263 (M)469-268-5226 2811 Telecom Drive Richardson, TX 75081, USA -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#21840): https://lists.onap.org/g/onap-discuss/message/21840 Mute This Topic: https://lists.onap.org/mt/76016611/21656 Group Owner: onap-discuss+ow...@lists.onap.org Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-