I can see two errors from this.
1. A Thread was stuck at startos cc level, and it was stopped by the
CarbonStuckThreadDetectionValve.
2. A closed channel exception at axis2 transport level, this should
have been caused by the termination of the stuck thread.
You may want to check (debug) why the thread (call to startInstace) go
in to that situation when unsubscribing?
But I have a question. Why auto-scaling (from the trace, I can see new
instance start is getting called) is happening when you are trying to
unsubscribe? May be because the unsubscribe was not successful
according to the below log?
[2013-11-06 10:12:36,569] INFO
{org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl}
- Termination of an instance which is belong to domain
'fedup.php.stratos.com.php.domain' and sub domain '__$default' ,
failed! Reason: No matching running instance found in lastly used
IaaS.
On Wed, Nov 6, 2013 at 10:23 AM, Udara Liyanage <[email protected]> wrote:
> Hi,
>
> I got the fallowing error when unsubscribing from a cartridge. I got a error
> message( Unable to sendViaPost to
> url[https://cc.stratos.apache.org:9444/services/CloudControllerService) from
> ELB also. It is also pasted at the end of this mail
>
> [2013-11-06 10:12:36,568] INFO
> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
> Starting to terminate all instances of domain :
> fedup.php.stratos.com.php.domain and sub domain : __$default
> [2013-11-06 10:12:36,569] INFO
> {org.apache.stratos.cloud.controller.impl.CloudControllerServiceImpl} -
> Termination of an instance which is belong to domain
> 'fedup.php.stratos.com.php.domain' and sub domain '__$default' , failed!
> Reason: No matching running instance found in lastly used IaaS.
> [2013-11-06 10:12:42,931] INFO
> {org.apache.stratos.cloud.controller.deployers.ServiceDeployer} -
> Successfully undeployed the Service definition specified at
> /home/nalin/Desktop/udara-vcloud/install/apache-stratos-cc-3.0.0-incubating/repository/deployment/server/services/fedup.php.stratos.com.php.domain-__$default-729977ae-1318-4150-961a-993f14d0911b.xml
> [2013-11-06 10:18:37,491] ERROR
> {org.apache.axis2.rpc.receivers.RPCMessageReceiver} - Exception occurred
> while trying to invoke service method startInstance
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:212)
> at
> org.apache.axis2.rpc.receivers.RPCMessageReceiver.invokeBusinessLogic(RPCMessageReceiver.java:117)
> at
> org.apache.axis2.receivers.AbstractInOutMessageReceiver.invokeBusinessLogic(AbstractInOutMessageReceiver.java:40)
> at
> org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:110)
> at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
> at
> org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:172)
> at org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:146)
> at
> org.wso2.carbon.core.transports.CarbonServlet.doPost(CarbonServlet.java:231)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:755)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
> at
> org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
> at
> org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128)
> at
> org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
> at
> org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at
> org.wso2.carbon.tomcat.ext.filter.CharacterSetFilter.doFilter(CharacterSetFilter.java:61)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
> at
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
> at
> org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:177)
> at
> org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:161)
> at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
> at
> org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:57)
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
> at
> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
> at
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
> at
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1653)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.ThreadDeath
> at java.lang.Thread.stop(Thread.java:758)
> at
> org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.handleStuckThread(CarbonStuckThreadDetectionValve.java:126)
> at
> org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.backgroundProcess(CarbonStuckThreadDetectionValve.java:180)
> at
> org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1368)
> at
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1530)
> at
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1540)
> at
> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1519)
> ... 1 more
> [2013-11-06 10:18:37,508] ERROR
> {org.apache.axis2.transport.http.CommonsHTTPTransportSender} -
> org.apache.axis2.AxisFault
> at org.apache.axis2.AxisFault.makeFault(AxisFault.java:430)
> at
> org.apache.axis2.transport.http.SOAPMessageFormatter.writeTo(SOAPMessageFormatter.java:78)
> at
> org.apache.axis2.transport.http.CommonsHTTPTransportSender.sendUsingOutputStream(CommonsHTTPTransportSender.java:411)
> at
> org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:288)
> at org.apache.axis2.engine.AxisEngine.sendFault(AxisEngine.java:525)
> at
> org.apache.axis2.transport.http.AxisServlet.handleFault(AxisServlet.java:433)
> at
> org.apache.axis2.transport.http.AxisServlet.processAxisFault(AxisServlet.java:398)
> at org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:188)
> at
> org.wso2.carbon.core.transports.CarbonServlet.doPost(CarbonServlet.java:231)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:755)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
> at
> org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
> at
> org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128)
> at
> org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
> at
> org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at
> org.wso2.carbon.tomcat.ext.filter.CharacterSetFilter.doFilter(CharacterSetFilter.java:61)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
> at
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
> at
> org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:177)
> at
> org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:161)
> at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
> at
> org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:57)
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
> at
> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
> at
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
> at
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1653)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: com.ctc.wstx.exc.WstxIOException: null
> at
> com.ctc.wstx.sw.BaseStreamWriter._finishDocument(BaseStreamWriter.java:1406)
> at com.ctc.wstx.sw.BaseStreamWriter.close(BaseStreamWriter.java:247)
> at
> org.apache.axiom.util.stax.wrapper.XMLStreamWriterWrapper.close(XMLStreamWriterWrapper.java:46)
> at
> org.apache.axiom.om.impl.MTOMXMLStreamWriter.close(MTOMXMLStreamWriter.java:188)
> at
> org.apache.axiom.om.impl.llom.OMSerializableImpl.serializeAndConsume(OMSerializableImpl.java:197)
> at
> org.apache.axis2.transport.http.SOAPMessageFormatter.writeTo(SOAPMessageFormatter.java:74)
> ... 36 more
> Caused by: ClientAbortException: java.nio.channels.ClosedChannelException
> at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:346)
> at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:306)
> at
> org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:101)
> at com.ctc.wstx.io.UTF8Writer.flush(UTF8Writer.java:103)
> at com.ctc.wstx.sw.BufferingXmlWriter.flush(BufferingXmlWriter.java:225)
> at com.ctc.wstx.sw.BufferingXmlWriter.close(BufferingXmlWriter.java:198)
> at
> com.ctc.wstx.sw.BaseStreamWriter._finishDocument(BaseStreamWriter.java:1404)
> ... 41 more
> Caused by: java.nio.channels.ClosedChannelException
> at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:133)
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
> at
> org.apache.tomcat.util.net.SecureNioChannel.flush(SecureNioChannel.java:135)
> at
> org.apache.tomcat.util.net.SecureNioChannel.write(SecureNioChannel.java:488)
> at
> org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:94)
> at
> org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:174)
> at
> org.apache.coyote.http11.InternalNioOutputBuffer.writeToSocket(InternalNioOutputBuffer.java:163)
> at
> org.apache.coyote.http11.InternalNioOutputBuffer.flushBuffer(InternalNioOutputBuffer.java:242)
> at
> org.apache.coyote.http11.InternalNioOutputBuffer.flush(InternalNioOutputBuffer.java:94)
> at
> org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Processor.java:790)
> at org.apache.coyote.Response.action(Response.java:173)
> at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:341)
> ... 47 more
>
>
>
> Error log of ELB
>
> 2013-11-06 10:06:37,419] INFO - HTTPSender Unable to sendViaPost to
> url[https://cc.stratos.apache.org:9444/services/CloudControllerService]
> java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
> at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
> at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
> at java.net.Socket.connect(Socket.java:529)
> at
> com.sun.net.ssl.internal.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:559)
> at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.commons.httpclient.protocol.ReflectionSocketFactory.createSocket(ReflectionSocketFactory.java:140)
> at
> org.apache.commons.httpclient.protocol.SSLProtocolSocketFactory.createSocket(SSLProtocolSocketFactory.java:130)
> at
> org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
> at
> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361)
> at
> org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
> at
> org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
> at
> org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
> at
> org.apache.axis2.transport.http.AbstractHTTPSender.executeMethod(AbstractHTTPSender.java:622)
> at
> org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:193)
> at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:75)
> at
> org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:451)
> at
> org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:278)
> at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:442)
> at
> org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:398)
> at
> org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:224)
> at org.apache.axis2.client.OperationClient.execute(OperationClient.java:149)
> at
> org.apache.stratos.cloud.controller.stub.CloudControllerServiceStub.startInstance(CloudControllerServiceStub.java:1386)
> at
> org.apache.stratos.mediator.autoscale.lbautoscale.clients.CloudControllerStubClient.startInstance(CloudControllerStubClient.java:59)
> at
> org.apache.stratos.mediator.autoscale.lbautoscale.util.AutoscaleUtil.runInstances(AutoscaleUtil.java:322)
> at
> org.apache.stratos.mediator.autoscale.lbautoscale.callables.AppNodeSanityCheckCallable.call(AppNodeSanityCheckCallable.java:79)
> at
> org.apache.stratos.mediator.autoscale.lbautoscale.callables.AppNodeSanityCheckCallable.call(AppNodeSanityCheckCallable.java:33)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> [2013-11-06 10:06:37,420] ERROR - AutoscaleUtil Failed to start an instance
> of Domain: fedup.php.stratos.com.php.domain - Sub Domain: __$default.
>
>
> --
> Udara Liyanage
> Software Engineer
> WSO2, Inc.: http://wso2.com
> lean. enterprise. middleware
>
> web: http://udaraliyanage.wordpress.com
> phone: +94 71 443 6897
