Update:
As on unsubscription, cartridge VM not deleted from vCD, I stopped & deleted it
manually. After this I start receiving Faulty Member Detected [member-id]
message & Publishing member fault event for [member-id] on periodic basis.
These messages are still coming even after I undeployed the cartridge. Now
Deployment of cartridge is also not taking place.
TID: [0] [STRATOS] [2014-11-11 09:12:27,925] INFO
{org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
Successfully undeployed the Cartridge definition: uag
{org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl}
TID: [0] [STRATOS] [2014-11-11 09:12:41,979] INFO
{org.apache.stratos.rest.endpoint.services.ServiceUtils} - Starting to deploy
a Cartridge [type] uag {org.apache.stratos.rest.endpoint.services.ServiceUtils}
TID: [0] [STRATOS] [2014-11-11 09:12:51,167] INFO
{org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} - Faulty
member detected [member-id] uag.uag.domain38306e0a-9295-4031-ba2b-825128db9dab
with [last time-stamp] 1415693680835 [time-out] 60000 milliseconds
{org.apache.stratos.cep.extension.FaultHandlingWindowProcessor}
TID: [0] [STRATOS] [2014-11-11 09:12:51,167] INFO
{org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} - Publishing
member fault event for [member-id]
uag.uag.domain38306e0a-9295-4031-ba2b-825128db9dab
{org.apache.stratos.cep.extension.FaultHandlingWindowProcessor}
TID: [0] [STRATOS] [2014-11-11 09:13:02,669] INFO
{org.wso2.carbon.databridge.core.DataBridge} - admin connected
{org.wso2.carbon.databridge.core.DataBridge}
TID: [0] [STRATOS] [2014-11-11 09:13:26,796] INFO
{org.wso2.carbon.databridge.core.DataBridge} - admin connected
{org.wso2.carbon.databridge.core.DataBridge}
TID: [0] [STRATOS] [2014-11-11 09:13:51,168] INFO
{org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} - Faulty
member detected [member-id] uag.uag.domain38306e0a-9295-4031-ba2b-825128db9dab
with [last time-stamp] 1415693680835 [time-out] 60000 milliseconds
{org.apache.stratos.cep.extension.FaultHandlingWindowProcessor}
TID: [0] [STRATOS] [2014-11-11 09:13:51,169] INFO
{org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} - Publishing
member fault event for [member-id]
uag.uag.domain38306e0a-9295-4031-ba2b-825128db9dab
{org.apache.stratos.cep.extension.FaultHandlingWindowProcessor}
TID: [0] [STRATOS] [2014-11-11 09:14:51,170] INFO
{org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} - Faulty
member detected [member-id] uag.uag.domain38306e0a-9295-4031-ba2b-825128db9dab
with [last time-stamp] 1415693680835 [time-out] 60000 milliseconds
{org.apache.stratos.cep.extension.FaultHandlingWindowProcessor}
TID: [0] [STRATOS] [2014-11-11 09:14:51,171] INFO
{org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} - Publishing
member fault event for [member-id]
uag.uag.domain38306e0a-9295-4031-ba2b-825128db9dab
{org.apache.stratos.cep.extension.FaultHandlingWindowProcessor}
TID: [0] [STRATOS] [2014-11-11 09:15:51,172] INFO
{org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} - Faulty
member detected [member-id] uag.uag.domain38306e0a-9295-4031-ba2b-825128db9dab
with [last time-stamp] 1415693680835 [time-out] 60000 milliseconds
{org.apache.stratos.cep.extension.FaultHandlingWindowProcessor}
TID: [0] [STRATOS] [2014-11-11 09:15:51,172] INFO
{org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} - Publishing
member fault event for [member-id]
uag.uag.domain38306e0a-9295-4031-ba2b-825128db9dab
{org.apache.stratos.cep.extension.FaultHandlingWindowProcessor}
TID: [0] [STRATOS] [2014-11-11 09:16:51,174] INFO
{org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} - Faulty
member detected [member-id] uag.uag.domain38306e0a-9295-4031-ba2b-825128db9dab
with [last time-stamp] 1415693680835 [time-out] 60000 milliseconds
{org.apache.stratos.cep.extension.FaultHandlingWindowProcessor}
TID: [0] [STRATOS] [2014-11-11 09:16:51,174] INFO
{org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} - Publishing
member fault event for [member-id]
uag.uag.domain38306e0a-9295-4031-ba2b-825128db9dab
{org.apache.stratos.cep.extension.FaultHandlingWindowProcessor}
TID: [0] [STRATOS] [2014-11-11 09:17:26,354] INFO
{org.wso2.carbon.databridge.core.DataBridge} - admin connected
{org.wso2.carbon.databridge.core.DataBridge}
TID: [0] [STRATOS] [2014-11-11 09:17:42,085] INFO
{org.apache.axis2.transport.http.HTTPSender} - Unable to sendViaPost to
url[https://octl.qmog.cisco.com:9443/services/CloudControllerService/]
{org.apache.axis2.transport.http.HTTPSender}
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
at sun.security.ssl.InputRecord.read(InputRecord.java:480)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at
org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)
at
org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
at
org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)
at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413)
at
org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)
at
org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
at
org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
at
org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
at
org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at
org.apache.axis2.transport.http.AbstractHTTPSender.executeMethod(AbstractHTTPSender.java:622)
at
org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:193)
at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:75)
at
org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:451)
at
org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:278)
at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442)
at
org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:430)
at
org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:225)
at
org.apache.axis2.client.OperationClient.execute(OperationClient.java:149)
at
org.apache.stratos.cloud.controller.stub.CloudControllerServiceStub.deployCartridgeDefinition(CloudControllerServiceStub.java:2636)
at
org.apache.stratos.manager.client.CloudControllerServiceClient.deployCartridgeDefinition(CloudControllerServiceClient.java:86)
at
org.apache.stratos.rest.endpoint.services.ServiceUtils.deployCartridge(ServiceUtils.java:117)
at
org.apache.stratos.rest.endpoint.services.StratosAdmin.deployCartridgeDefinition(StratosAdmin.java:193)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:180)
at
org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:194)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:100)
at
org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:57)
at
org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:93)
at
org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:271)
at
org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
at
org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:239)
at
org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:223)
at
org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:203)
at
org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:137)
at
org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:159)
at
org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:286)
at
org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:206)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:755)
at
org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:262)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
at
org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:178)
at
org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
at
org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:56)
at
org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
at
org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:141)
at
org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:156)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
at
org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:52)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1653)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
TID: [0] [STRATOS] [2014-11-11 09:17:42,091] ERROR
{org.apache.stratos.rest.endpoint.services.ServiceUtils} - Read timed out
{org.apache.stratos.rest.endpoint.services.ServiceUtils}
org.apache.axis2.AxisFault: Read timed out
at org.apache.axis2.AxisFault.makeFault(AxisFault.java:430)
at
org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:197)
at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:75)
at
org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:451)
at
org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:278)
at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442)
at
org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:430)
at
org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:225)
at
org.apache.axis2.client.OperationClient.execute(OperationClient.java:149)
at
org.apache.stratos.cloud.controller.stub.CloudControllerServiceStub.deployCartridgeDefinition(CloudControllerServiceStub.java:2636)
at
org.apache.stratos.manager.client.CloudControllerServiceClient.deployCartridgeDefinition(CloudControllerServiceClient.java:86)
at
org.apache.stratos.rest.endpoint.services.ServiceUtils.deployCartridge(ServiceUtils.java:117)
at
org.apache.stratos.rest.endpoint.services.StratosAdmin.deployCartridgeDefinition(StratosAdmin.java:193)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:180)
at
org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:194)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:100)
at
org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:57)
at
org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:93)
at
org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:271)
at
org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
at
org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:239)
at
org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:223)
at
org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:203)
at
org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:137)
at
org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:159)
at
org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:286)
at
org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:206)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:755)
at
org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:262)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
at
org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:178)
at
org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
at
org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:56)
at
org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
at
org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:141)
at
org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:156)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
at
org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:52)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1653)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
at sun.security.ssl.InputRecord.read(InputRecord.java:480)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:102)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
at
org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)
at
org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
at
org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1116)
at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.readLine(MultiThreadedHttpConnectionManager.java:1413)
at
org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1973)
at
org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1735)
at
org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1098)
at
org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
at
org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at
org.apache.axis2.transport.http.AbstractHTTPSender.executeMethod(AbstractHTTPSender.java:622)
at
org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:193)
... 55 more
TID: [0] [STRATOS] [2014-11-11 09:17:51,177] INFO
{org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} - Faulty
member detected [member-id] uag.uag.domain38306e0a-9295-4031-ba2b-825128db9dab
with [last time-stamp] 1415693680835 [time-out] 60000 milliseconds
{org.apache.stratos.cep.extension.FaultHandlingWindowProcessor}
TID: [0] [STRATOS] [2014-11-11 09:17:51,177] INFO
{org.apache.stratos.cep.extension.FaultHandlingWindowProcessor} - Publishing
member fault event for [member-id]
uag.uag.domain38306e0a-9295-4031-ba2b-825128db9dab
{org.apache.stratos.cep.extension.FaultHandlingWindowProcessor}
Regards
Manas
From: Manas Padegaonkar -T (mpadegao - ARICENT TECHNOLOGIES HOLDINGS LIMITED at
Cisco)
Sent: Tuesday, November 11, 2014 1:51 PM
To: [email protected]
Subject: Cartridge-UnSubscription Not Deleting the Cartridge-VM Instance
Hi,
I have multiple VMs running on vCloud-Director through Stratos & all of them
are in ACTIVE state for more than 12 hours. I unsubscribe one of the VM which
is not showing in output of list-subscribed-cartridges
but the cartridge-vm is still up & running on vCloud-Director.
Attached is the section of wso2carbon.log after unsubscription of cartridge.
Regards
Manas
