Hi Vanson, I added a validation in Autoscaler to avoid applications being deleted while they are being undeployed in the background.
Now the following error would be raised if this condition is met: "Application cannot be deleted, undeployment process is still in progress: [application-id] <application-id>". In addition I have implemented your suggestion to set the application status to CREATED once the application is completely undeployed. Now we can check from the API whether the application is undeployed properly. Please take a pull and verify. Thanks On Sat, Mar 28, 2015 at 2:40 AM, Vanson Lim <v...@cisco.com> wrote: > On 3/27/15, 2:26 PM, Imesh Gunaratne wrote: > > Hi Vanson, > > I analyzed this issue. Logically we cannot delete an application soon > after un-deploying it. The reason is that the un-deployment process > executes gracefully on each member in all the clusters in an application. > Depending on the complexity of the application, it may take some time to > completely terminate all the members, remove the clusters and then > un-deploy the application. > > IMO what we are missing here is a validation in the delete application > API method, to tell the API client that an application cannot be deleted > until it is completely un-deployed. > > Imesh, > > Thanks for looking at this. > > It sounds reasonable to hold off the removing to application and also have > the remove rest api call return that it's not ready. For applications > consisting of a lot of groups and cartridges the time to clean up could be > variable. Could we use the Rest api to get the application status, ie > check that status=Created before deleting. > > ie this api: > > > https://cwiki.apache.org/confluence/display/STRATOS/4.1.0+Getting+Details+of+an+Application+via+REST+API > > It looks like an application has two states: "Created" and "Deployed". > If we had a Undeploying state, then we could check and hold off > requesting the remove-application until it returns to the Created state. > > -Vanson > > > Thanks > Imesh > > On Wed, Mar 25, 2015 at 12:10 PM, Udara Liyanage <ud...@wso2.com> wrote: > >> Hi, >> >> This arises when the application is deleted immediately after the >> application underemployment. I guess CC does not have the member context >> when the application is removed. I will have a look. >> >> A simple workaround it to wait sometime to remove application after >> undeployment. >> >> On Tue, Mar 24, 2015 at 9:06 AM, Imesh Gunaratne <im...@apache.org> >> wrote: >> >>> Thanks Martin! We will have a look at this. >>> >>> On Tue, Mar 24, 2015 at 1:21 AM, Martin Eppel (meppel) <mep...@cisco.com >>> > wrote: >>> >>>> I opened a jira https://issues.apache.org/jira/browse/STRATOS-1281 to >>>> track the issue, >>>> >>>> >>>> >>>> Thanks >>>> >>>> >>>> >>>> Martin >>>> >>>> >>>> >>>> *From:* Imesh Gunaratne [mailto:im...@apache.org] >>>> *Sent:* Friday, March 20, 2015 3:27 AM >>>> *To:* dev >>>> *Subject:* Re: Stratos 4.1.0 - tracebacks seen when issuing >>>> application undeploy/remove >>>> >>>> >>>> >>>> Hi Vanson, >>>> >>>> >>>> >>>> Thanks for reporting this problem. According to the logs stratos is >>>> trying to remove the same member twice in this flow: >>>> >>>> >>>> >>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,952] INFO >>>> {org.apache.stratos.messaging.message.processor.application.ApplicationDeletedMessageProcessor} >>>> - [Application] cisco-sample-vm has been successfully removed >>>> >>>> TID: [0] [STRATOS] [2015-03-19 19:45:44,786] INFO >>>> {org.apache.stratos.cloud.controller.iaases.JcloudsIaas} - Member >>>> terminated: [member-id] >>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d >>>> >>>> >>>> >>>> TID: [0] [STRATOS] [2015-03-19 19:45:50,012] INFO >>>> {org.apache.stratos.common.client.CloudControllerServiceClient} - >>>> Terminating instance via cloud controller: [member] >>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d >>>> >>>> TID: [0] [STRATOS] [2015-03-19 19:45:50,017] ERROR >>>> {org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl} >>>> - Could not terminate instance, member context not found: [member-id] >>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d >>>> >>>> >>>> >>>> We might need to investigate and see what's causing this. >>>> >>>> >>>> >>>> Thanks >>>> >>>> >>>> >>>> >>>> >>>> On Fri, Mar 20, 2015 at 6:55 AM, Vanson Lim <v...@cisco.com> wrote: >>>> >>>> Hi, >>>> >>>> We are testing the behavior of stratos 4.1.0 rest api's and found that >>>> if we issue an undeploy-application followed immediately with >>>> application-remove, that we see the following traceback. >>>> >>>> The VM successfully get's deleted but not sure what kind of side >>>> effects this has on the system. >>>> >>>> Here's the snippet from the wso2carbon.log file. I've also attached >>>> the entire log. >>>> >>>> Steps to reproduce this: >>>> >>>> 1) deploy an application which leads to startup a single instance of a >>>> cartridge. >>>> 2) wait for it to become active >>>> 3) issue the undeploy application and application remove rest api calls. >>>> >>>> >>>> -Vanson >>>> >>>> >>>> >>>> TID: [0] [STRATOS] [2015-03-19 19:45:08,681] INFO >>>> {org.wso2.carbon.databridge.core.DataBridge} - admin connected >>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,023] INFO >>>> {org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl} - >>>> Starting to undeploy application: [application-id] cisco-sample-vm >>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,024] INFO >>>> {org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl} - >>>> Removing application signup: [application-id] cisco-sample-vm >>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,051] INFO >>>> {org.apache.stratos.manager.components.ApplicationSignUpHandler} - Removing >>>> application signup: [application-id] cisco-sample-vm [tenant-id] -1234 >>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,084] INFO >>>> {org.apache.stratos.manager.components.ApplicationSignUpHandler} - >>>> Application signup removed successfully: [application-id] cisco-sample-vm >>>> [tenant-id] -1234 >>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,091] INFO >>>> {org.apache.stratos.autoscaler.context.AutoscalerContext} - Network >>>> partition algorithm context is removed successfully: [id] cisco-sample-vm >>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,092] INFO >>>> {org.apache.stratos.autoscaler.monitor.cluster.ClusterMonitor} - Publishing >>>> Cluster terminating event for [application] cisco-sample-vm [cluster] >>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain [instance] >>>> cisco-sample-vm-1 >>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,109] INFO >>>> {org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder} - >>>> Cluster Terminating adding status started >>>> forcisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain >>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,117] INFO >>>> {org.apache.stratos.autoscaler.applications.topic.ApplicationsEventPublisher} >>>> - Publishing application inactivated event: [application] cisco-sample-vm >>>> [instance] cisco-sample-vm-1 >>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,120] INFO >>>> {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher} >>>> - Publishing Cluster terminating event: [application-id] cisco-sample-vm >>>> [cluster id] cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain >>>> [instance-id] cisco-sample-vm-1 >>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,144] INFO >>>> {org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl} - >>>> Application undeployed successfully: [application-id] cisco-sample-vm >>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,154] INFO >>>> {org.apache.stratos.autoscaler.event.receiver.topology.AutoscalerTopologyEventReceiver} >>>> - [ClusterTerminatingEvent] Received: class >>>> org.apache.stratos.messaging.event.topology.ClusterInstanceTerminatingEvent >>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,155] INFO >>>> {org.apache.stratos.autoscaler.event.publisher.InstanceNotificationPublisher} >>>> - Publishing Instance Cleanup Event: [cluster] >>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain >>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,171] WARN >>>> {org.apache.stratos.autoscaler.status.processor.cluster.ClusterStatusActiveProcessor} >>>> - No possible state change found for [type] [cluster] >>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain [instance] >>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,192] INFO >>>> {org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder} - >>>> member maintenance mode event adding status started >>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,204] INFO >>>> {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher} >>>> - Publishing member in maintenance mode event: [service-name] >>>> cisco-sample-vm [cluster-id] >>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain >>>> [cluster-instance-id] cisco-sample-vm-1 [member-id] >>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d >>>> [network-partition-id] RegionOne [partition-id] whole-region >>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,230] INFO >>>> {org.apache.stratos.messaging.message.processor.topology.MemberMaintenanceModeProcessor} >>>> - Member updated as In_Maintenance: [service] cisco-sample-vm [cluster] >>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain [member] >>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d >>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,231] INFO >>>> {org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder} - >>>> Member Ready to shut down event adding status started >>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,242] INFO >>>> {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher} >>>> - Publishing member ready to shut down event: [service-name] >>>> cisco-sample-vm [cluster-id] >>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain >>>> [cluster-instance-id] cisco-sample-vm-1 [member-id] >>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d >>>> [network-partition-id] RegionOne [partition-id] whole-region >>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,262] INFO >>>> {org.apache.stratos.messaging.message.processor.topology.MemberReadyToShutdownMessageProcessor} >>>> - Member updated as Ready to shutdown: [service] cisco-sample-vm [cluster] >>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain [member] >>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d >>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,264] INFO >>>> {org.apache.stratos.autoscaler.event.receiver.topology.AutoscalerTopologyEventReceiver} >>>> - [MemberReadyToShutdownEvent] Received: class >>>> org.apache.stratos.messaging.event.topology.MemberReadyToShutdownEvent >>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,264] INFO >>>> {org.apache.stratos.autoscaler.context.partition.ClusterLevelPartitionContext} >>>> - Starting the moving of termination pending to obsolete for [member] >>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d >>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,264] INFO >>>> {org.apache.stratos.autoscaler.context.partition.ClusterLevelPartitionContext} >>>> - Found termination pending member and trying to move [member] >>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d >>>> to obsolete list >>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,266] INFO >>>> {org.apache.stratos.common.client.CloudControllerServiceClient} - >>>> Terminating instance via cloud controller: [member] >>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d >>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,275] INFO >>>> {org.apache.stratos.cloud.controller.iaases.JcloudsIaas} - Starting to >>>> terminate member: [cartridge-type] cisco-sample-vm [member-id] >>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d >>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,883] INFO >>>> {org.apache.stratos.messaging.domain.application.locking.ApplicationLockHierarchy} >>>> - Removed lock for application: [application-id] cisco-sample-vm >>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,909] INFO >>>> {org.apache.stratos.cloud.controller.messaging.receiver.application.ApplicationEventReceiver} >>>> - Application deleted event received: [application-id] cisco-sample-vm >>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,909] INFO >>>> {org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder} - >>>> Removed application [ cisco-sample-vm ]'s Cluster [ >>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain ] from the topology >>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,910] INFO >>>> {org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl} - >>>> Application deleted successfully: [application-id] >>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,930] INFO >>>> {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher} >>>> - Publishing application clusters removed event: [application-id] >>>> cisco-sample-vm >>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,950] INFO >>>> {org.apache.stratos.messaging.message.processor.topology.ApplicationClustersRemovedMessageProcessor} >>>> - Cluster cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain removed >>>> from topology for application cisco-sample-vm >>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,952] INFO >>>> {org.apache.stratos.messaging.message.processor.application.ApplicationDeletedMessageProcessor} >>>> - [Application] cisco-sample-vm has been successfully removed >>>> TID: [0] [STRATOS] [2015-03-19 19:45:44,786] INFO >>>> {org.apache.stratos.cloud.controller.iaases.JcloudsIaas} - Member >>>> terminated: [member-id] >>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d >>>> TID: [0] [STRATOS] [2015-03-19 19:45:44,786] WARN >>>> {org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder} - >>>> Cluster cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain does not >>>> exist >>>> TID: [0] [STRATOS] [2015-03-19 19:45:50,012] INFO >>>> {org.apache.stratos.common.client.CloudControllerServiceClient} - >>>> Terminating instance via cloud controller: [member] >>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d >>>> TID: [0] [STRATOS] [2015-03-19 19:45:50,017] ERROR >>>> {org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl} >>>> - Could not terminate instance, member context not found: [member-id] >>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d >>>> TID: [0] [STRATOS] [2015-03-19 19:45:50,018] ERROR >>>> {org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl} >>>> - Could not terminate instance: [member-id] >>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d >>>> org.apache.stratos.cloud.controller.exception.InvalidMemberException: >>>> Could not terminate instance, member context not found: [member-id] >>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d >>>> at >>>> org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl.terminateInstance(CloudControllerServiceImpl.java:553) >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>> at >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>>> at >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>> at >>>> org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:212) >>>> at >>>> org.apache.axis2.rpc.receivers.RPCInOnlyMessageReceiver.invokeBusinessLogic(RPCInOnlyMessageReceiver.java:66) >>>> at >>>> org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:110) >>>> at >>>> org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180) >>>> at >>>> org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:172) >>>> at >>>> org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:146) >>>> at >>>> org.wso2.carbon.core.transports.CarbonServlet.doPost(CarbonServlet.java:231) >>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:755) >>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) >>>> at >>>> org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61) >>>> at >>>> org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128) >>>> at >>>> org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68) >>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) >>>> at >>>> org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68) >>>> at >>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) >>>> at >>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) >>>> at >>>> org.wso2.carbon.tomcat.ext.filter.CharacterSetFilter.doFilter(CharacterSetFilter.java:61) >>>> at >>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) >>>> at >>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) >>>> at >>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222) >>>> at >>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) >>>> at >>>> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472) >>>> at >>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) >>>> at >>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99) >>>> at >>>> org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:178) >>>> at >>>> org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47) >>>> at >>>> org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:56) >>>> at >>>> org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47) >>>> at >>>> org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:141) >>>> at >>>> org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:156) >>>> at >>>> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936) >>>> at >>>> org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:52) >>>> at >>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) >>>> at >>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407) >>>> at >>>> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004) >>>> at >>>> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589) >>>> at org.apache.tomcat.util.net >>>> .NioEndpoint$SocketProcessor.run(NioEndpoint.java:1653) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>> at java.lang.Thread.run(Thread.java:745) >>>> TID: [0] [STRATOS] [2015-03-19 19:45:50,019] WARN >>>> {org.apache.stratos.common.concurrent.locks.ReadWriteLock} - System >>>> warning! Trying to release a lock which has not been taken by the same >>>> thread: [lock-name] topology-manager [thread-id] 164 [thread-name] >>>> http-nio-9443-exec-29 >>>> TID: [0] [STRATOS] [2015-03-19 19:45:50,020] ERROR >>>> {org.apache.axis2.rpc.receivers.RPCInOnlyMessageReceiver} - Could not >>>> terminate instance, member context not found: [member-id] >>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d >>>> org.apache.stratos.cloud.controller.exception.InvalidMemberException: >>>> Could not terminate instance, member context not found: [member-id] >>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d >>>> at >>>> org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl.terminateInstance(CloudControllerServiceImpl.java:553) >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>> at >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>>> at >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>> at >>>> org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:212) >>>> at >>>> org.apache.axis2.rpc.receivers.RPCInOnlyMessageReceiver.invokeBusinessLogic(RPCInOnlyMessageReceiver.java:66) >>>> at >>>> org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:110) >>>> at >>>> org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180) >>>> at >>>> org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:172) >>>> at >>>> org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:146) >>>> at >>>> org.wso2.carbon.core.transports.CarbonServlet.doPost(CarbonServlet.java:231) >>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:755) >>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) >>>> at >>>> org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61) >>>> at >>>> org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128) >>>> at >>>> org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68) >>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) >>>> at >>>> org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68) >>>> at >>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) >>>> at >>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) >>>> at >>>> org.wso2.carbon.tomcat.ext.filter.CharacterSetFilter.doFilter(CharacterSetFilter.java:61) >>>> at >>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) >>>> at >>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) >>>> at >>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222) >>>> at >>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) >>>> at >>>> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472) >>>> at >>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) >>>> at >>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99) >>>> at >>>> org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:178) >>>> at >>>> org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47) >>>> at >>>> org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:56) >>>> at >>>> org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47) >>>> at >>>> org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:141) >>>> at >>>> org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:156) >>>> at >>>> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936) >>>> at >>>> org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:52) >>>> at >>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) >>>> at >>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407) >>>> at >>>> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004) >>>> at >>>> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589) >>>> at org.apache.tomcat.util.net >>>> .NioEndpoint$SocketProcessor.run(NioEndpoint.java:1653) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>> at java.lang.Thread.run(Thread.java:745) >>>> TID: [0] [STRATOS] [2015-03-19 19:45:50,027] ERROR >>>> {org.apache.stratos.autoscaler.rule.RuleTasksDelegator} - Cannot terminate >>>> instance >>>> org.apache.axis2.AxisFault: Could not terminate instance, member >>>> context not found: [member-id] >>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d >>>> at >>>> org.apache.axis2.util.Utils.getInboundFaultFromMessageContext(Utils.java:531) >>>> at >>>> org.apache.axis2.description.RobustOutOnlyAxisOperation$RobustOutOnlyOperationClient.handleResponse(RobustOutOnlyAxisOperation.java:91) >>>> at >>>> org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:445) >>>> at >>>> org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:225) >>>> at >>>> org.apache.axis2.client.OperationClient.execute(OperationClient.java:149) >>>> at >>>> org.apache.stratos.cloud.controller.stub.CloudControllerServiceStub.terminateInstance(CloudControllerServiceStub.java:7071) >>>> at >>>> org.apache.stratos.common.client.CloudControllerServiceClient.terminateInstance(CloudControllerServiceClient.java:120) >>>> at >>>> org.apache.stratos.autoscaler.rule.RuleTasksDelegator.terminateObsoleteInstance(RuleTasksDelegator.java:311) >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>> at >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>>> at >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>> at >>>> org.mvel2.optimizers.impl.refl.nodes.MethodAccessor.getValue(MethodAccessor.java:48) >>>> at >>>> org.mvel2.optimizers.impl.refl.nodes.VariableAccessor.getValue(VariableAccessor.java:37) >>>> at >>>> org.mvel2.ast.ASTNode.getReducedValueAccelerated(ASTNode.java:108) >>>> at org.mvel2.MVELRuntime.execute(MVELRuntime.java:85) >>>> at >>>> org.mvel2.compiler.CompiledExpression.getDirectValue(CompiledExpression.java:123) >>>> at >>>> org.mvel2.compiler.CompiledExpression.getValue(CompiledExpression.java:119) >>>> at org.mvel2.MVEL.executeExpression(MVEL.java:930) >>>> at >>>> org.drools.base.mvel.MVELConsequence.evaluate(MVELConsequence.java:104) >>>> at >>>> org.drools.common.DefaultAgenda.fireActivation(DefaultAgenda.java:1287) >>>> at >>>> org.drools.common.DefaultAgenda.fireNextItem(DefaultAgenda.java:1221) >>>> >>>> >>>> >>>> >>>> >>>> -- >>>> >>>> Imesh Gunaratne >>>> >>>> >>>> >>>> Technical Lead, WSO2 >>>> >>>> Committer & PMC Member, Apache Stratos >>>> >>> >>> >>> >>> -- >>> Imesh Gunaratne >>> >>> Technical Lead, WSO2 >>> Committer & PMC Member, Apache Stratos >>> >> >> >> >> -- >> >> Udara Liyanage >> Software Engineer >> WSO2, Inc.: http://wso2.com >> lean. enterprise. middleware >> >> web: http://udaraliyanage.wordpress.com >> phone: +94 71 443 6897 >> > > > > -- > Imesh Gunaratne > > Technical Lead, WSO2 > Committer & PMC Member, Apache Stratos > > > -- Imesh Gunaratne Technical Lead, WSO2 Committer & PMC Member, Apache Stratos
