Hello,
we are doing some onap evaluation and trying to deploy ONAP via OOM (master
branch), all pods are up except
onap-aaf aaf-c68d8877c-r62qk 0/1
Running 0 1d
which is I think expected behavior (AAF is blocked by OOM-324)
but few robot health checks are failing after deployment,
I am now analyzing ASDC health check failure and would appreciate any hint or
help … as I am currently out of ideas 😊
here is what I collected so far:
Basic ASDC Health check failing on U-EB cluster is not available,
which is most likely caused by connectivity issue between DMAAP & ZOOKEEPER
images used:
attos/dmaap latest b0ae220fcf1f 7
months ago 747 MB
wurstmeister/zookeeper latest 351aa00d2fe9 14 months ago
478 MB
dmaap service description:
Name: dmaap
Namespace: onap-message-router
Labels: app=dmaap
version=1.0.0
Annotations: msb.onap.org/service-info=[ { "serviceName":
"message-router", "version": "v1", "url": "/", "protocol": "REST", "port":
"3904", "visualRange":"1" } ]
Selector: app=dmaap
Type: NodePort
IP: 10.43.197.31
Port: mr1 3904/TCP
TargetPort: 3904/TCP
NodePort: mr1 30227/TCP
Endpoints: 10.42.174.244:3904
Port: mr2 3905/TCP
TargetPort: 3905/TCP
NodePort: mr2 30226/TCP
Endpoints: 10.42.174.244:3905
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
Zookeeper service description:
Name: zookeeper
Namespace: onap-message-router
Labels: app=zookeeper
Annotations: <none>
Selector: app=zookeeper
Type: ClusterIP
IP: None
Port: zookeeper1 2181/TCP
TargetPort: 2181/TCP
Endpoints: 10.42.15.26:2181
Session Affinity: None
Events: <none>
both should be reachable and I see some ESTABLISHED connections between them !!
root@zookeeper-7df6479654-xjcp4:/opt/zookeeper-3.4.9# netstat -an
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
tcp6 0 0 :::2181 :::* LISTEN
tcp6 0 0 :::33128 :::* LISTEN
tcp6 0 0 :::22 :::* LISTEN
tcp6 0 0 10.42.15.26:2181 10.42.174.244:58198 ESTABLISHED
tcp6 0 0 10.42.15.26:2181 10.42.174.244:33018 ESTABLISHED
tcp6 0 0 10.42.15.26:2181 10.42.174.244:50764 ESTABLISHED
tcp6 0 0 10.42.15.26:2181 10.42.148.187:34914 ESTABLISHED
tcp6 0 0 10.42.15.26:2181 10.42.174.244:56294 ESTABLISHED
tcp6 0 0 10.42.15.26:2181 10.42.174.244:57152 ESTABLISHED
tcp6 0 0 10.42.15.26:2181 10.42.174.244:49676 ESTABLISHED
tcp6 0 0 10.42.15.26:2181 10.42.174.244:50214 ESTABLISHED
tcp6 0 0 10.42.15.26:2181 10.42.174.244:57150 ESTABLISHED
but still zookeeper is reporting (in loop):
2018-02-15 10:33:48,010 [myid:] - INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted
socket connection from /10.42.0.1:57184
2018-02-15 10:33:48,010 [myid:] - WARN
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of
stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x0,
likely client has closed socket
at
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
at
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
at java.lang.Thread.run(Thread.java:745)
2018-02-15 10:33:48,010 [myid:] - INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1008] - Closed socket
connection for client /10.42.0.1:57184 (no session established for client)
and DMAAP is reporting (in loop):
10:35:11.729 [qtp379110473-299] ERROR
org.apache.cxf.interceptor.AbstractFaultChainInitiatorObserver - Error occurred
during error handling, give up!
org.apache.cxf.interceptor.Fault: Unable to connect to zookeeper server within
timeout: 10000
at
org.apache.cxf.service.invoker.AbstractInvoker.createFault(AbstractInvoker.java:163)
at
org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:129)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:200)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:99)
at
org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59)
at
org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96)
at
org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:307)
at
org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
at
org.apache.camel.component.cxf.cxfbean.CxfBeanDestination.process(CxfBeanDestination.java:83)
at
org.apache.camel.impl.ProcessorEndpoint.onExchange(ProcessorEndpoint.java:103)
at
org.apache.camel.impl.ProcessorEndpoint$1.process(ProcessorEndpoint.java:71)
at
org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)
at
org.apache.camel.processor.SendProcessor.process(SendProcessor.java:129)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:448)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:118)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:109)
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:87)
at
org.apache.camel.component.http.CamelServlet.service(CamelServlet.java:144)
at ajsc.servlet.AjscCamelServlet.service(AjscCamelServlet.java:66)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:808)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
at ajsc.filter.PassthruFilter.doFilter(PassthruFilter.java:26)
at
org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:343)
at
org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:260)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at
com.att.ajsc.csi.writeablerequestfilter.WriteableRequestFilter.doFilter(WriteableRequestFilter.java:41)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at
com.att.nsa.dmaap.util.DMaaPAuthFilter.doFilter(DMaaPAuthFilter.java:108)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:497)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect
to zookeeper server within timeout: 10000
at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:880)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:98)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:84)
at
com.att.nsa.cambria.beans.DMaaPKafkaMetaBroker.createTopic(DMaaPKafkaMetaBroker.java:181)
at
com.att.nsa.cambria.service.impl.TopicServiceImpl.createTopic(TopicServiceImpl.java:283)
at
com.att.nsa.dmaap.service.TopicRestService.createTopic(TopicRestService.java:327)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:181)
at
org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:97)
... 50 common frames omitted
10:35:11.735 [qtp379110473-299] ERROR
org.apache.camel.processor.DefaultErrorHandler - Failed delivery for
(MessageId: ID-dmaap-84bd655dd6-wb87m-35362-1518690347087-0-1347 on ExchangeId:
ID-dmaap-84bd655dd6-wb87m-35362-1518690347087-0-1348). Exhausted after delivery
attempt: 1 caught: java.lang.RuntimeException:
org.apache.cxf.interceptor.Fault: Unable to connect to zookeeper server within
timeout: 10000
Message History
---------------------------------------------------------------------------------------------------------------------------------------
RouteId ProcessorId Processor
Elapsed (ms)
[dmaap:topicService] [dmaap:topicService]
[att-dme2-servlet:///topics?matchOnUriPrefix=true
] [ 10034]
[dmaap:topicService] [to6 ]
[cxfbean:topicService?providers=#jacksonProvider,#cambriaExMapper,#webExMapper
] [ 10034]
Exchange
---------------------------------------------------------------------------------------------------------------------------------------
Exchange[
Id ID-dmaap-84bd655dd6-wb87m-35362-1518690347087-0-1348
ExchangePattern InOut
Headers {Accept=application/json,
Accept-Encoding=gzip,deflate,
breadcrumbId=ID-dmaap-84bd655dd6-wb87m-35362-1518690347087-0-1347,
CamelHttpCharacterEncoding=UTF-8, CamelHttpMethod=POST, CamelHttpPath=/create,
CamelHttpQuery=null,
CamelHttpServletRequest=com.att.ajsc.csi.writeablerequestfilter.WriteableRequestWrapper@169b75d3,
CamelHttpServletResponse=HTTP/1.1 200
Date: Thu, 15 Feb 2018 10:35:01 GMT
, CamelHttpUri=/topics/create,
CamelHttpUrl=http://dmaap.onap-message-router:3904/topics/create,
CamelRedelivered=false, CamelRedeliveryCounter=0,
CamelServletContextPath=/topics, Connection=keep-alive, Content-Length=141,
Content-Type=application/json, Host=dmaap.onap-message-router:3904,
User-Agent=Apache-HttpClient/4.4.1 (Java/1.8.0_141),
X-CambriaAuth=iPIxkpAMI8qTcQj8:NAgw6T0CXKEk9AKJdstVlWDtITA=, X-CambriaDate=Thu,
15 Feb 2018 10:35:01 UTC, X-CSI-Internal-WriteableRequest=true}
BodyType org.apache.camel.converter.stream.InputStreamCache
Body [Body is instance of org.apache.camel.StreamCache]
]
Stacktrace
---------------------------------------------------------------------------------------------------------------------------------------
java.lang.RuntimeException: org.apache.cxf.interceptor.Fault: Unable to connect
to zookeeper server within timeout: 10000
at
org.apache.cxf.interceptor.AbstractFaultChainInitiatorObserver.onMessage(AbstractFaultChainInitiatorObserver.java:116)
at
org.apache.cxf.phase.PhaseInterceptorChain.wrapExceptionAsFault(PhaseInterceptorChain.java:365)
at
org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:323)
at
org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
at
org.apache.camel.component.cxf.cxfbean.CxfBeanDestination.process(CxfBeanDestination.java:83)
at
org.apache.camel.impl.ProcessorEndpoint.onExchange(ProcessorEndpoint.java:103)
at
org.apache.camel.impl.ProcessorEndpoint$1.process(ProcessorEndpoint.java:71)
at
org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61)
at
org.apache.camel.processor.SendProcessor.process(SendProcessor.java:129)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:448)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:118)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:109)
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:87)
at
org.apache.camel.component.http.CamelServlet.service(CamelServlet.java:144)
at ajsc.servlet.AjscCamelServlet.service(AjscCamelServlet.java:66)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:808)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
at ajsc.filter.PassthruFilter.doFilter(PassthruFilter.java:26)
at
org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:343)
at
org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:260)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at
com.att.ajsc.csi.writeablerequestfilter.WriteableRequestFilter.doFilter(WriteableRequestFilter.java:41)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at
com.att.nsa.dmaap.util.DMaaPAuthFilter.doFilter(DMaaPAuthFilter.java:108)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:497)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.cxf.interceptor.Fault: Unable to connect to zookeeper
server within timeout: 10000
at
org.apache.cxf.service.invoker.AbstractInvoker.createFault(AbstractInvoker.java:163)
at
org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:129)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:200)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:99)
at
org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59)
at
org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96)
at
org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:307)
... 45 common frames omitted
Caused by: org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect
to zookeeper server within timeout: 10000
at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:880)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:98)
at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:84)
at
com.att.nsa.cambria.beans.DMaaPKafkaMetaBroker.createTopic(DMaaPKafkaMetaBroker.java:181)
at
com.att.nsa.cambria.service.impl.TopicServiceImpl.createTopic(TopicServiceImpl.java:283)
at
com.att.nsa.dmaap.service.TopicRestService.createTopic(TopicRestService.java:327)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:181)
at
org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:97)
... 50 common frames omitted
Thanks for reading that !
Michal
_______________________________________________
onap-discuss mailing list
onap-discuss@lists.onap.org
https://lists.onap.org/mailman/listinfo/onap-discuss
