Re: Stratos 4.1.0 - tracebacks seen when issuing application undeploy/remove

Fri, 27 Mar 2015 14:14:05 -0700 

On 3/27/15, 2:26 PM, Imesh Gunaratne wrote:

Hi Vanson,
I analyzed this issue. Logically we cannot delete an application soon after un-deploying it. The reason is that the un-deployment process executes gracefully on each member in all the clusters in an application. Depending on the complexity of the application, it may take some time to completely terminate all the members, remove the clusters and then un-deploy the application.
IMO what we are missing here is a validation in the delete application API method, to tell the API client that an application cannot be deleted until it is completely un-deployed. 


Imesh,

Thanks for looking at this.
It sounds reasonable to hold off the removing to application and also have the remove rest api call return that it's not ready. For applications consisting of a lot of groups and cartridges the time to clean up could be variable. Could we use the Rest api to get the application status, ie check that status=Created before deleting. 

ie this api:

https://cwiki.apache.org/confluence/display/STRATOS/4.1.0+Getting+Details+of+an+Application+via+REST+API

It looks like an application has two states:   "Created" and "Deployed".   If 
we had a Undeploying state, then we could check and hold off
requesting the remove-application until it returns to the Created state.

-Vanson


Thanks
Imesh

On Wed, Mar 25, 2015 at 12:10 PM, Udara Liyanage <ud...@wso2.com 
<mailto:ud...@wso2.com>> wrote:

    Hi,

    This arises when the application is deleted immediately after the 
application underemployment. I guess CC does not have the member
    context when the application is removed. I will have a look.

    A simple workaround it to wait sometime to remove application after 
undeployment.

    On Tue, Mar 24, 2015 at 9:06 AM, Imesh Gunaratne <im...@apache.org 
<mailto:im...@apache.org>> wrote:

        Thanks Martin! We will have a look at this.

        On Tue, Mar 24, 2015 at 1:21 AM, Martin Eppel (meppel) <mep...@cisco.com 
<mailto:mep...@cisco.com>> wrote:

            I opened a jira https://issues.apache.org/jira/browse/STRATOS-1281 
to track the issue,

            Thanks

            Martin

            *From:*Imesh Gunaratne [mailto:im...@apache.org 
<mailto:im...@apache.org>]
            *Sent:* Friday, March 20, 2015 3:27 AM
            *To:* dev
            *Subject:* Re: Stratos 4.1.0 - tracebacks seen when issuing 
application undeploy/remove

            Hi Vanson,

            Thanks for reporting this problem. According to the logs stratos is 
trying to remove the same member twice in this flow:

            TID: [0] [STRATOS] [2015-03-19 19:45:43,952]  INFO
            
{org.apache.stratos.messaging.message.processor.application.ApplicationDeletedMessageProcessor}
 -  [Application]
            cisco-sample-vm has been successfully removed

            TID: [0] [STRATOS] [2015-03-19 19:45:44,786]  INFO 
{org.apache.stratos.cloud.controller.iaases.JcloudsIaas} -  Member
            terminated: [member-id] 
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d

            TID: [0] [STRATOS] [2015-03-19 19:45:50,012]  INFO 
{org.apache.stratos.common.client.CloudControllerServiceClient} -
            Terminating instance via cloud controller: [member]
            
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d

            TID: [0] [STRATOS] [2015-03-19 19:45:50,017] ERROR
            
{org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl} 
-  Could not terminate instance, member
            context not found: [member-id] 
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d

            We might need to investigate and see what's causing this.

            Thanks

            On Fri, Mar 20, 2015 at 6:55 AM, Vanson Lim <v...@cisco.com 
<mailto:v...@cisco.com>> wrote:

            Hi,

            We are testing the behavior of stratos 4.1.0 rest api's  and found 
that if we issue an undeploy-application followed
            immediately with application-remove, that we see the following 
traceback.

            The VM successfully get's deleted but not sure what kind of side 
effects this has on the system.

            Here's the snippet from the wso2carbon.log file. I've also attached 
the entire log.

            Steps to reproduce this:

            1) deploy an application which leads to startup a single instance 
of a cartridge.
            2) wait for it to become active
            3) issue the undeploy application and application remove rest api 
calls.


            -Vanson



            TID: [0] [STRATOS] [2015-03-19 19:45:08,681] INFO 
{org.wso2.carbon.databridge.core.DataBridge} -  admin connected
TID: [0] [STRATOS] [2015-03-19 19:45:43,023] INFO {org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl} - Starting to undeploy application: [application-id] cisco-sample-vm TID: [0] [STRATOS] [2015-03-19 19:45:43,024] INFO {org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl} - Removing application signup: [application-id] cisco-sample-vm 
            TID: [0] [STRATOS] [2015-03-19 19:45:43,051] INFO 
{org.apache.stratos.manager.components.ApplicationSignUpHandler} - Removing
            application signup: [application-id] cisco-sample-vm [tenant-id] 
-1234
            TID: [0] [STRATOS] [2015-03-19 19:45:43,084] INFO 
{org.apache.stratos.manager.components.ApplicationSignUpHandler} -
            Application signup removed successfully: [application-id] 
cisco-sample-vm [tenant-id] -1234
            TID: [0] [STRATOS] [2015-03-19 19:45:43,091] INFO 
{org.apache.stratos.autoscaler.context.AutoscalerContext} -  Network
            partition algorithm context is removed successfully: [id] 
cisco-sample-vm
            TID: [0] [STRATOS] [2015-03-19 19:45:43,092] INFO 
{org.apache.stratos.autoscaler.monitor.cluster.ClusterMonitor} - Publishing
            Cluster terminating event for [application] cisco-sample-vm 
[cluster] cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain
            [instance] cisco-sample-vm-1
TID: [0] [STRATOS] [2015-03-19 19:45:43,109] INFO {org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder} - Cluster Terminating adding status started forcisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain 
            TID: [0] [STRATOS] [2015-03-19 19:45:43,117] INFO
            
{org.apache.stratos.autoscaler.applications.topic.ApplicationsEventPublisher} - 
 Publishing application inactivated event:
            [application] cisco-sample-vm [instance] cisco-sample-vm-1
            TID: [0] [STRATOS] [2015-03-19 19:45:43,120] INFO
            
{org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher}
 -  Publishing Cluster terminating event:
            [application-id] cisco-sample-vm [cluster id] 
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain [instance-id]
            cisco-sample-vm-1
TID: [0] [STRATOS] [2015-03-19 19:45:43,144] INFO {org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl} - Application undeployed successfully: [application-id] cisco-sample-vm 
            TID: [0] [STRATOS] [2015-03-19 19:45:43,154] INFO
            
{org.apache.stratos.autoscaler.event.receiver.topology.AutoscalerTopologyEventReceiver}
 - [ClusterTerminatingEvent] Received:
            class 
org.apache.stratos.messaging.event.topology.ClusterInstanceTerminatingEvent
            TID: [0] [STRATOS] [2015-03-19 19:45:43,155] INFO
            
{org.apache.stratos.autoscaler.event.publisher.InstanceNotificationPublisher} - 
 Publishing Instance Cleanup Event: [cluster]
            cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain
            TID: [0] [STRATOS] [2015-03-19 19:45:43,171] WARN
            
{org.apache.stratos.autoscaler.status.processor.cluster.ClusterStatusActiveProcessor}
 -  No possible state change found for
            [type] [cluster] 
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain [instance]
TID: [0] [STRATOS] [2015-03-19 19:45:43,192] INFO {org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder} - member maintenance mode event adding status started 
            TID: [0] [STRATOS] [2015-03-19 19:45:43,204] INFO
            
{org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher}
 -  Publishing member in maintenance mode
            event: [service-name] cisco-sample-vm [cluster-id] 
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain
            [cluster-instance-id] cisco-sample-vm-1 [member-id]
            
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
 [network-partition-id] RegionOne
            [partition-id] whole-region
            TID: [0] [STRATOS] [2015-03-19 19:45:43,230] INFO
            
{org.apache.stratos.messaging.message.processor.topology.MemberMaintenanceModeProcessor}
 -  Member updated as In_Maintenance:
            [service] cisco-sample-vm [cluster] 
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain [member]
            
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
TID: [0] [STRATOS] [2015-03-19 19:45:43,231] INFO {org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder} - Member Ready to shut down event adding status started 
            TID: [0] [STRATOS] [2015-03-19 19:45:43,242] INFO
            
{org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher}
 -  Publishing member ready to shut down
            event: [service-name] cisco-sample-vm [cluster-id] 
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain
            [cluster-instance-id] cisco-sample-vm-1 [member-id]
            
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
 [network-partition-id] RegionOne
            [partition-id] whole-region
            TID: [0] [STRATOS] [2015-03-19 19:45:43,262] INFO
            
{org.apache.stratos.messaging.message.processor.topology.MemberReadyToShutdownMessageProcessor}
 -  Member updated as Ready to
            shutdown: [service] cisco-sample-vm [cluster] 
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain [member]
            
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
            TID: [0] [STRATOS] [2015-03-19 19:45:43,264] INFO
            
{org.apache.stratos.autoscaler.event.receiver.topology.AutoscalerTopologyEventReceiver}
 - [MemberReadyToShutdownEvent]
            Received: class 
org.apache.stratos.messaging.event.topology.MemberReadyToShutdownEvent
            TID: [0] [STRATOS] [2015-03-19 19:45:43,264] INFO
            
{org.apache.stratos.autoscaler.context.partition.ClusterLevelPartitionContext} 
-  Starting the moving of termination pending
            to obsolete for [member] 
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
            TID: [0] [STRATOS] [2015-03-19 19:45:43,264] INFO
            
{org.apache.stratos.autoscaler.context.partition.ClusterLevelPartitionContext} 
-  Found termination pending member and trying
            to move [member] 
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
 to obsolete list
            TID: [0] [STRATOS] [2015-03-19 19:45:43,266] INFO 
{org.apache.stratos.common.client.CloudControllerServiceClient} -
            Terminating instance via cloud controller: [member]
            
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
            TID: [0] [STRATOS] [2015-03-19 19:45:43,275] INFO 
{org.apache.stratos.cloud.controller.iaases.JcloudsIaas} -  Starting to
            terminate member: [cartridge-type] cisco-sample-vm [member-id]
            
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
            TID: [0] [STRATOS] [2015-03-19 19:45:43,883] INFO
            
{org.apache.stratos.messaging.domain.application.locking.ApplicationLockHierarchy}
 -  Removed lock for application:
            [application-id] cisco-sample-vm
            TID: [0] [STRATOS] [2015-03-19 19:45:43,909] INFO
            
{org.apache.stratos.cloud.controller.messaging.receiver.application.ApplicationEventReceiver}
 -  Application deleted event
            received: [application-id] cisco-sample-vm
TID: [0] [STRATOS] [2015-03-19 19:45:43,909] INFO {org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder} - Removed application [ cisco-sample-vm ]'s Cluster [ cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain ] from the topology TID: [0] [STRATOS] [2015-03-19 19:45:43,910] INFO {org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl} - Application deleted successfully: [application-id] 
            TID: [0] [STRATOS] [2015-03-19 19:45:43,930] INFO
            
{org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher}
 -  Publishing application clusters removed
            event: [application-id] cisco-sample-vm
            TID: [0] [STRATOS] [2015-03-19 19:45:43,950] INFO
            
{org.apache.stratos.messaging.message.processor.topology.ApplicationClustersRemovedMessageProcessor}
 -  Cluster
            cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain removed from 
topology for application cisco-sample-vm
            TID: [0] [STRATOS] [2015-03-19 19:45:43,952] INFO
            
{org.apache.stratos.messaging.message.processor.application.ApplicationDeletedMessageProcessor}
 -  [Application]
            cisco-sample-vm has been successfully removed
            TID: [0] [STRATOS] [2015-03-19 19:45:44,786] INFO 
{org.apache.stratos.cloud.controller.iaases.JcloudsIaas} -  Member
            terminated: [member-id] 
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
TID: [0] [STRATOS] [2015-03-19 19:45:44,786] WARN {org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder} - Cluster cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain does not exist 
            TID: [0] [STRATOS] [2015-03-19 19:45:50,012] INFO 
{org.apache.stratos.common.client.CloudControllerServiceClient} -
            Terminating instance via cloud controller: [member]
            
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
            TID: [0] [STRATOS] [2015-03-19 19:45:50,017] ERROR
            
{org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl} 
-  Could not terminate instance, member
            context not found: [member-id] 
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
            TID: [0] [STRATOS] [2015-03-19 19:45:50,018] ERROR
            
{org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl} 
-  Could not terminate instance: [member-id]
            
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
            
org.apache.stratos.cloud.controller.exception.InvalidMemberException: Could not 
terminate instance, member context not found:
            [member-id] 
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
                    at
            
org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl.terminateInstance(CloudControllerServiceImpl.java:553)
                    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
                    at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
                    at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                    at java.lang.reflect.Method.invoke(Method.java:606)
                    at 
org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:212)
                    at 
org.apache.axis2.rpc.receivers.RPCInOnlyMessageReceiver.invokeBusinessLogic(RPCInOnlyMessageReceiver.java:66)
                    at 
org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:110)
                    at 
org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
                    at 
org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:172)
                    at 
org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:146)
                    at 
org.wso2.carbon.core.transports.CarbonServlet.doPost(CarbonServlet.java:231)
                    at 
javax.servlet.http.HttpServlet.service(HttpServlet.java:755)
                    at 
javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
                    at 
org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
                    at 
org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128)
                    at 
org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68)
                    at 
javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
                    at 
org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68)
                    at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
                    at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
                    at 
org.wso2.carbon.tomcat.ext.filter.CharacterSetFilter.doFilter(CharacterSetFilter.java:61)
                    at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
                    at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
                    at 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
                    at 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
                    at 
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
                    at 
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
                    at 
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
                    at 
org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:178)
                    at 
org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
                    at 
org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:56)
                    at 
org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
                    at 
org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:141)
                    at 
org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:156)
                    at 
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
                    at 
org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:52)
                    at 
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
                    at 
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
                    at 
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
                    at 
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
                    at org.apache.tomcat.util.net 
<http://org.apache.tomcat.util.net>.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1653)
                    at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                    at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
                    at java.lang.Thread.run(Thread.java:745)
            TID: [0] [STRATOS] [2015-03-19 19:45:50,019] WARN 
{org.apache.stratos.common.concurrent.locks.ReadWriteLock} -  System
            warning! Trying to release a lock which has not been taken by the 
same thread: [lock-name] topology-manager [thread-id] 164
            [thread-name] http-nio-9443-exec-29
            TID: [0] [STRATOS] [2015-03-19 19:45:50,020] ERROR 
{org.apache.axis2.rpc.receivers.RPCInOnlyMessageReceiver} -  Could not
            terminate instance, member context not found: [member-id]
            
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
            
org.apache.stratos.cloud.controller.exception.InvalidMemberException: Could not 
terminate instance, member context not found:
            [member-id] 
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
                    at
            
org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl.terminateInstance(CloudControllerServiceImpl.java:553)
                    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
                    at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
                    at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                    at java.lang.reflect.Method.invoke(Method.java:606)
                    at 
org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:212)
                    at 
org.apache.axis2.rpc.receivers.RPCInOnlyMessageReceiver.invokeBusinessLogic(RPCInOnlyMessageReceiver.java:66)
                    at 
org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:110)
                    at 
org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
                    at 
org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:172)
                    at 
org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:146)
                    at 
org.wso2.carbon.core.transports.CarbonServlet.doPost(CarbonServlet.java:231)
                    at 
javax.servlet.http.HttpServlet.service(HttpServlet.java:755)
                    at 
javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
                    at 
org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
                    at 
org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128)
                    at 
org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68)
                    at 
javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
                    at 
org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68)
                    at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
                    at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
                    at 
org.wso2.carbon.tomcat.ext.filter.CharacterSetFilter.doFilter(CharacterSetFilter.java:61)
                    at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
                    at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
                    at 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
                    at 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
                    at 
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
                    at 
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
                    at 
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
                    at 
org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:178)
                    at 
org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
                    at 
org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:56)
                    at 
org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
                    at 
org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:141)
                    at 
org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:156)
                    at 
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
                    at 
org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:52)
                    at 
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
                    at 
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
                    at 
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
                    at 
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
                    at org.apache.tomcat.util.net 
<http://org.apache.tomcat.util.net>.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1653)
                    at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                    at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
                    at java.lang.Thread.run(Thread.java:745)
            TID: [0] [STRATOS] [2015-03-19 19:45:50,027] ERROR 
{org.apache.stratos.autoscaler.rule.RuleTasksDelegator} -  Cannot
            terminate instance
            org.apache.axis2.AxisFault: Could not terminate instance, member 
context not found: [member-id]
            
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
                    at 
org.apache.axis2.util.Utils.getInboundFaultFromMessageContext(Utils.java:531)
                    at
            
org.apache.axis2.description.RobustOutOnlyAxisOperation$RobustOutOnlyOperationClient.handleResponse(RobustOutOnlyAxisOperation.java:91)
                    at 
org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:445)
                    at 
org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:225)
                    at 
org.apache.axis2.client.OperationClient.execute(OperationClient.java:149)
                    at
            
org.apache.stratos.cloud.controller.stub.CloudControllerServiceStub.terminateInstance(CloudControllerServiceStub.java:7071)
                    at 
org.apache.stratos.common.client.CloudControllerServiceClient.terminateInstance(CloudControllerServiceClient.java:120)
                    at 
org.apache.stratos.autoscaler.rule.RuleTasksDelegator.terminateObsoleteInstance(RuleTasksDelegator.java:311)
                    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
                    at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
                    at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                    at java.lang.reflect.Method.invoke(Method.java:606)
                    at 
org.mvel2.optimizers.impl.refl.nodes.MethodAccessor.getValue(MethodAccessor.java:48)
                    at 
org.mvel2.optimizers.impl.refl.nodes.VariableAccessor.getValue(VariableAccessor.java:37)
                    at 
org.mvel2.ast.ASTNode.getReducedValueAccelerated(ASTNode.java:108)
                    at org.mvel2.MVELRuntime.execute(MVELRuntime.java:85)
                    at 
org.mvel2.compiler.CompiledExpression.getDirectValue(CompiledExpression.java:123)
                    at 
org.mvel2.compiler.CompiledExpression.getValue(CompiledExpression.java:119)
                    at org.mvel2.MVEL.executeExpression(MVEL.java:930)
                    at 
org.drools.base.mvel.MVELConsequence.evaluate(MVELConsequence.java:104)
                    at 
org.drools.common.DefaultAgenda.fireActivation(DefaultAgenda.java:1287)
                    at 
org.drools.common.DefaultAgenda.fireNextItem(DefaultAgenda.java:1221)
-- 
            Imesh Gunaratne

            Technical Lead, WSO2

            Committer & PMC Member, Apache Stratos
-- Imesh Gunaratne 

        Technical Lead, WSO2
        Committer & PMC Member, Apache Stratos
-- 
    Udara Liyanage
    Software Engineer
    WSO2, Inc.: http://wso2.com <http://wso2.com/>
    lean. enterprise. middleware

    web: http://udaraliyanage.wordpress.com
    phone: +94 71 443 6897




--
Imesh Gunaratne

Technical Lead, WSO2
Committer & PMC Member, Apache Stratos

Reply via email to