Re: Stratos 4.1.0 - tracebacks seen when issuing application undeploy/remove

Fri, 27 Mar 2015 15:00:13 -0700 

On 3/27/15, 5:52 PM, Imesh Gunaratne wrote:

Hi Vanson,

I added a validation in Autoscaler to avoid applications being deleted while 
they are being undeployed in the background.
Now the following error would be raised if this condition is met: "Application cannot be deleted, undeployment process is still in progress: [application-id] <application-id>".
In addition I have implemented your suggestion to set the application status to CREATED once the application is completely undeployed. Now we can check from the API whether the application is undeployed properly. Please take a pull and verify. 



okay, will give it a try.

-Vanson


Thanks

On Sat, Mar 28, 2015 at 2:40 AM, Vanson Lim <v...@cisco.com 
<mailto:v...@cisco.com>> wrote:

    On 3/27/15, 2:26 PM, Imesh Gunaratne wrote:

    Hi Vanson,

    I analyzed this issue. Logically we cannot delete an application soon after 
un-deploying it. The reason is that the un-deployment
    process executes gracefully on each member in all the clusters in an 
application. Depending on the complexity of the application, it
    may take some time to completely terminate all the members, remove the 
clusters and then un-deploy the application.

    IMO what we are missing here is a validation in the delete application API 
method, to tell the API client that an application cannot
    be deleted until it is completely un-deployed.


    Imesh,

    Thanks for looking at this.

    It sounds reasonable to hold off the removing to application and also have 
the remove rest api call return that it's not ready.   For
    applications consisting of a lot of groups and cartridges the time to clean 
up could be variable.   Could we use the Rest api to get
    the application status, ie check that status=Created before deleting.

    ie this api:

    
https://cwiki.apache.org/confluence/display/STRATOS/4.1.0+Getting+Details+of+an+Application+via+REST+API

    It looks like an application has two states:   "Created" and "Deployed".   
If we had a Undeploying state, then we could check and
    hold off
    requesting the remove-application until it returns to the Created state.

    -Vanson



    Thanks
    Imesh

    On Wed, Mar 25, 2015 at 12:10 PM, Udara Liyanage <ud...@wso2.com 
<mailto:ud...@wso2.com>> wrote:

        Hi,

        This arises when the application is deleted immediately after the 
application underemployment. I guess CC does not have the
        member context when the application is removed. I will have a look.

        A simple workaround it to wait sometime to remove application after 
undeployment.

        On Tue, Mar 24, 2015 at 9:06 AM, Imesh Gunaratne <im...@apache.org 
<mailto:im...@apache.org>> wrote:

            Thanks Martin! We will have a look at this.

            On Tue, Mar 24, 2015 at 1:21 AM, Martin Eppel (meppel) <mep...@cisco.com 
<mailto:mep...@cisco.com>> wrote:

                I opened a jira 
https://issues.apache.org/jira/browse/STRATOS-1281 to track the issue,

                Thanks

                Martin

                *From:*Imesh Gunaratne [mailto:im...@apache.org 
<mailto:im...@apache.org>]
                *Sent:* Friday, March 20, 2015 3:27 AM
                *To:* dev
                *Subject:* Re: Stratos 4.1.0 - tracebacks seen when issuing 
application undeploy/remove

                Hi Vanson,

                Thanks for reporting this problem. According to the logs 
stratos is trying to remove the same member twice in this flow:

                TID: [0] [STRATOS] [2015-03-19 19:45:43,952]  INFO
                
{org.apache.stratos.messaging.message.processor.application.ApplicationDeletedMessageProcessor}
 -  [Application]
                cisco-sample-vm has been successfully removed

                TID: [0] [STRATOS] [2015-03-19 19:45:44,786]  INFO 
{org.apache.stratos.cloud.controller.iaases.JcloudsIaas} -  Member
                terminated: [member-id] 
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d

                TID: [0] [STRATOS] [2015-03-19 19:45:50,012]  INFO 
{org.apache.stratos.common.client.CloudControllerServiceClient} -
                Terminating instance via cloud controller: [member]
                
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d

                TID: [0] [STRATOS] [2015-03-19 19:45:50,017] ERROR
                
{org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl} 
-  Could not terminate instance, member
                context not found: [member-id] 
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d

                We might need to investigate and see what's causing this.

                Thanks

                On Fri, Mar 20, 2015 at 6:55 AM, Vanson Lim <v...@cisco.com 
<mailto:v...@cisco.com>> wrote:

                Hi,

                We are testing the behavior of stratos 4.1.0 rest api's  and 
found that if we issue an undeploy-application followed
                immediately with application-remove, that we see the following 
traceback.

                The VM successfully get's deleted but not sure what kind of 
side effects this has on the system.

                Here's the snippet from the wso2carbon.log file.  I've also 
attached the entire log.

                Steps to reproduce this:

                1) deploy an application which leads to startup a single 
instance of a cartridge.
                2) wait for it to become active
                3) issue the undeploy application and application remove rest 
api calls.


                -Vanson



                TID: [0] [STRATOS] [2015-03-19 19:45:08,681] INFO 
{org.wso2.carbon.databridge.core.DataBridge} -  admin connected
TID: [0] [STRATOS] [2015-03-19 19:45:43,023] INFO {org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl} - Starting to undeploy application: [application-id] cisco-sample-vm TID: [0] [STRATOS] [2015-03-19 19:45:43,024] INFO {org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl} - Removing application signup: [application-id] cisco-sample-vm 
                TID: [0] [STRATOS] [2015-03-19 19:45:43,051] INFO 
{org.apache.stratos.manager.components.ApplicationSignUpHandler} -
                Removing application signup: [application-id] cisco-sample-vm 
[tenant-id] -1234
                TID: [0] [STRATOS] [2015-03-19 19:45:43,084] INFO 
{org.apache.stratos.manager.components.ApplicationSignUpHandler} -
                Application signup removed successfully: [application-id] 
cisco-sample-vm [tenant-id] -1234
                TID: [0] [STRATOS] [2015-03-19 19:45:43,091] INFO 
{org.apache.stratos.autoscaler.context.AutoscalerContext} -  Network
                partition algorithm context is removed successfully: [id] 
cisco-sample-vm
                TID: [0] [STRATOS] [2015-03-19 19:45:43,092] INFO 
{org.apache.stratos.autoscaler.monitor.cluster.ClusterMonitor} -
                Publishing Cluster terminating event for [application] 
cisco-sample-vm [cluster]
                cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain 
[instance] cisco-sample-vm-1
                TID: [0] [STRATOS] [2015-03-19 19:45:43,109] INFO
                
{org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder} -  
Cluster Terminating adding status started
                forcisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain
                TID: [0] [STRATOS] [2015-03-19 19:45:43,117] INFO
                
{org.apache.stratos.autoscaler.applications.topic.ApplicationsEventPublisher} - 
 Publishing application inactivated
                event: [application] cisco-sample-vm [instance] 
cisco-sample-vm-1
                TID: [0] [STRATOS] [2015-03-19 19:45:43,120] INFO
                
{org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher}
 -  Publishing Cluster terminating
                event: [application-id] cisco-sample-vm [cluster id] 
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain
                [instance-id] cisco-sample-vm-1
TID: [0] [STRATOS] [2015-03-19 19:45:43,144] INFO {org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl} - Application undeployed successfully: [application-id] cisco-sample-vm 
                TID: [0] [STRATOS] [2015-03-19 19:45:43,154] INFO
                
{org.apache.stratos.autoscaler.event.receiver.topology.AutoscalerTopologyEventReceiver}
 - [ClusterTerminatingEvent]
                Received: class 
org.apache.stratos.messaging.event.topology.ClusterInstanceTerminatingEvent
                TID: [0] [STRATOS] [2015-03-19 19:45:43,155] INFO
                
{org.apache.stratos.autoscaler.event.publisher.InstanceNotificationPublisher} - 
 Publishing Instance Cleanup Event:
                [cluster] cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain
                TID: [0] [STRATOS] [2015-03-19 19:45:43,171] WARN
                
{org.apache.stratos.autoscaler.status.processor.cluster.ClusterStatusActiveProcessor}
 -  No possible state change found
                for [type] [cluster] 
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain [instance]
                TID: [0] [STRATOS] [2015-03-19 19:45:43,192] INFO
                
{org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder} -  
member maintenance mode event adding status
                started
                TID: [0] [STRATOS] [2015-03-19 19:45:43,204] INFO
                
{org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher}
 -  Publishing member in maintenance
                mode event: [service-name] cisco-sample-vm [cluster-id] 
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain
                [cluster-instance-id] cisco-sample-vm-1 [member-id]
                
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
 [network-partition-id]
                RegionOne [partition-id] whole-region
                TID: [0] [STRATOS] [2015-03-19 19:45:43,230] INFO
                
{org.apache.stratos.messaging.message.processor.topology.MemberMaintenanceModeProcessor}
 -  Member updated as
                In_Maintenance: [service] cisco-sample-vm [cluster] 
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain [member]
                
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
                TID: [0] [STRATOS] [2015-03-19 19:45:43,231] INFO
                
{org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder} -  
Member Ready to shut down event adding
                status started
                TID: [0] [STRATOS] [2015-03-19 19:45:43,242] INFO
                
{org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher}
 -  Publishing member ready to shut down
                event: [service-name] cisco-sample-vm [cluster-id] 
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain
                [cluster-instance-id] cisco-sample-vm-1 [member-id]
                
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
 [network-partition-id]
                RegionOne [partition-id] whole-region
                TID: [0] [STRATOS] [2015-03-19 19:45:43,262] INFO
                
{org.apache.stratos.messaging.message.processor.topology.MemberReadyToShutdownMessageProcessor}
 -  Member updated as
                Ready to shutdown: [service] cisco-sample-vm [cluster] 
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain [member]
                
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
                TID: [0] [STRATOS] [2015-03-19 19:45:43,264] INFO
                
{org.apache.stratos.autoscaler.event.receiver.topology.AutoscalerTopologyEventReceiver}
 - [MemberReadyToShutdownEvent]
                Received: class 
org.apache.stratos.messaging.event.topology.MemberReadyToShutdownEvent
                TID: [0] [STRATOS] [2015-03-19 19:45:43,264] INFO
                
{org.apache.stratos.autoscaler.context.partition.ClusterLevelPartitionContext} 
-  Starting the moving of termination
                pending to obsolete for [member] 
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
                TID: [0] [STRATOS] [2015-03-19 19:45:43,264] INFO
                
{org.apache.stratos.autoscaler.context.partition.ClusterLevelPartitionContext} 
-  Found termination pending member and
                trying to move [member] 
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
 to
                obsolete list
                TID: [0] [STRATOS] [2015-03-19 19:45:43,266] INFO 
{org.apache.stratos.common.client.CloudControllerServiceClient} -
                Terminating instance via cloud controller: [member]
                
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
                TID: [0] [STRATOS] [2015-03-19 19:45:43,275] INFO 
{org.apache.stratos.cloud.controller.iaases.JcloudsIaas} -  Starting
                to terminate member: [cartridge-type] cisco-sample-vm 
[member-id]
                
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
                TID: [0] [STRATOS] [2015-03-19 19:45:43,883] INFO
                
{org.apache.stratos.messaging.domain.application.locking.ApplicationLockHierarchy}
 -  Removed lock for application:
                [application-id] cisco-sample-vm
                TID: [0] [STRATOS] [2015-03-19 19:45:43,909] INFO
                
{org.apache.stratos.cloud.controller.messaging.receiver.application.ApplicationEventReceiver}
 -  Application deleted
                event received: [application-id] cisco-sample-vm
                TID: [0] [STRATOS] [2015-03-19 19:45:43,909] INFO
                
{org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder} -  
Removed application [ cisco-sample-vm ]'s
                Cluster [ 
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain ] from the topology
TID: [0] [STRATOS] [2015-03-19 19:45:43,910] INFO {org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl} - Application deleted successfully: [application-id] 
                TID: [0] [STRATOS] [2015-03-19 19:45:43,930] INFO
                
{org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher}
 -  Publishing application clusters
                removed event: [application-id] cisco-sample-vm
                TID: [0] [STRATOS] [2015-03-19 19:45:43,950] INFO
                
{org.apache.stratos.messaging.message.processor.topology.ApplicationClustersRemovedMessageProcessor}
 -  Cluster
                cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain removed 
from topology for application cisco-sample-vm
                TID: [0] [STRATOS] [2015-03-19 19:45:43,952] INFO
                
{org.apache.stratos.messaging.message.processor.application.ApplicationDeletedMessageProcessor}
 - [Application]
                cisco-sample-vm has been successfully removed
                TID: [0] [STRATOS] [2015-03-19 19:45:44,786] INFO 
{org.apache.stratos.cloud.controller.iaases.JcloudsIaas} -  Member
                terminated: [member-id] 
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
                TID: [0] [STRATOS] [2015-03-19 19:45:44,786] WARN
                
{org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder} -  
Cluster
                cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain does not 
exist
                TID: [0] [STRATOS] [2015-03-19 19:45:50,012] INFO 
{org.apache.stratos.common.client.CloudControllerServiceClient} -
                Terminating instance via cloud controller: [member]
                
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
                TID: [0] [STRATOS] [2015-03-19 19:45:50,017] ERROR
                
{org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl} 
-  Could not terminate instance, member
                context not found: [member-id] 
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
                TID: [0] [STRATOS] [2015-03-19 19:45:50,018] ERROR
                
{org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl} 
-  Could not terminate instance:
                [member-id] 
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
                
org.apache.stratos.cloud.controller.exception.InvalidMemberException: Could not 
terminate instance, member context not
                found: [member-id] 
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
                        at
                
org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl.terminateInstance(CloudControllerServiceImpl.java:553)
                        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
                        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
                        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                        at java.lang.reflect.Method.invoke(Method.java:606)
                        at 
org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:212)
                        at 
org.apache.axis2.rpc.receivers.RPCInOnlyMessageReceiver.invokeBusinessLogic(RPCInOnlyMessageReceiver.java:66)
                        at 
org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:110)
                        at 
org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
                        at 
org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:172)
                        at 
org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:146)
                        at 
org.wso2.carbon.core.transports.CarbonServlet.doPost(CarbonServlet.java:231)
                        at 
javax.servlet.http.HttpServlet.service(HttpServlet.java:755)
                        at 
javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
                        at 
org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
                        at 
org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128)
                        at 
org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68)
                        at 
javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
                        at 
org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68)
                        at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
                        at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
                        at 
org.wso2.carbon.tomcat.ext.filter.CharacterSetFilter.doFilter(CharacterSetFilter.java:61)
                        at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
                        at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
                        at 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
                        at 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
                        at 
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
                        at 
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
                        at 
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
                        at 
org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:178)
                        at 
org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
                        at 
org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:56)
                        at 
org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
                        at 
org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:141)
                        at
                
org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:156)
                        at 
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
                        at 
org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:52)
                        at 
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
                        at 
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
                        at 
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
                        at 
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
                        at org.apache.tomcat.util.net
                
<http://org.apache.tomcat.util.net>.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1653)
                        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
                        at java.lang.Thread.run(Thread.java:745)
                TID: [0] [STRATOS] [2015-03-19 19:45:50,019] WARN 
{org.apache.stratos.common.concurrent.locks.ReadWriteLock} -  System
                warning! Trying to release a lock which has not been taken by 
the same thread: [lock-name] topology-manager [thread-id]
                164 [thread-name] http-nio-9443-exec-29
                TID: [0] [STRATOS] [2015-03-19 19:45:50,020] ERROR 
{org.apache.axis2.rpc.receivers.RPCInOnlyMessageReceiver} -  Could
                not terminate instance, member context not found: [member-id]
                
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
                
org.apache.stratos.cloud.controller.exception.InvalidMemberException: Could not 
terminate instance, member context not
                found: [member-id] 
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
                        at
                
org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl.terminateInstance(CloudControllerServiceImpl.java:553)
                        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
                        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
                        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                        at java.lang.reflect.Method.invoke(Method.java:606)
                        at 
org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:212)
                        at 
org.apache.axis2.rpc.receivers.RPCInOnlyMessageReceiver.invokeBusinessLogic(RPCInOnlyMessageReceiver.java:66)
                        at 
org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:110)
                        at 
org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
                        at 
org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:172)
                        at 
org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:146)
                        at 
org.wso2.carbon.core.transports.CarbonServlet.doPost(CarbonServlet.java:231)
                        at 
javax.servlet.http.HttpServlet.service(HttpServlet.java:755)
                        at 
javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
                        at 
org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
                        at 
org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128)
                        at 
org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68)
                        at 
javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
                        at 
org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68)
                        at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
                        at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
                        at 
org.wso2.carbon.tomcat.ext.filter.CharacterSetFilter.doFilter(CharacterSetFilter.java:61)
                        at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
                        at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
                        at 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
                        at 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
                        at 
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
                        at 
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
                        at 
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
                        at 
org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:178)
                        at 
org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
                        at 
org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:56)
                        at 
org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
                        at 
org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:141)
                        at
                
org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:156)
                        at 
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
                        at 
org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:52)
                        at 
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
                        at 
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
                        at 
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
                        at 
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
                        at org.apache.tomcat.util.net
                
<http://org.apache.tomcat.util.net>.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1653)
                        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
                        at java.lang.Thread.run(Thread.java:745)
                TID: [0] [STRATOS] [2015-03-19 19:45:50,027] ERROR 
{org.apache.stratos.autoscaler.rule.RuleTasksDelegator} -  Cannot
                terminate instance
                org.apache.axis2.AxisFault: Could not terminate instance, 
member context not found: [member-id]
                
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
                        at 
org.apache.axis2.util.Utils.getInboundFaultFromMessageContext(Utils.java:531)
                        at
                
org.apache.axis2.description.RobustOutOnlyAxisOperation$RobustOutOnlyOperationClient.handleResponse(RobustOutOnlyAxisOperation.java:91)
                        at 
org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:445)
                        at 
org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:225)
                        at 
org.apache.axis2.client.OperationClient.execute(OperationClient.java:149)
                        at
                
org.apache.stratos.cloud.controller.stub.CloudControllerServiceStub.terminateInstance(CloudControllerServiceStub.java:7071)
                        at
                
org.apache.stratos.common.client.CloudControllerServiceClient.terminateInstance(CloudControllerServiceClient.java:120)
                        at 
org.apache.stratos.autoscaler.rule.RuleTasksDelegator.terminateObsoleteInstance(RuleTasksDelegator.java:311)
                        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
                        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
                        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                        at java.lang.reflect.Method.invoke(Method.java:606)
                        at 
org.mvel2.optimizers.impl.refl.nodes.MethodAccessor.getValue(MethodAccessor.java:48)
                        at 
org.mvel2.optimizers.impl.refl.nodes.VariableAccessor.getValue(VariableAccessor.java:37)
                        at 
org.mvel2.ast.ASTNode.getReducedValueAccelerated(ASTNode.java:108)
                        at org.mvel2.MVELRuntime.execute(MVELRuntime.java:85)
                        at 
org.mvel2.compiler.CompiledExpression.getDirectValue(CompiledExpression.java:123)
                        at 
org.mvel2.compiler.CompiledExpression.getValue(CompiledExpression.java:119)
                        at org.mvel2.MVEL.executeExpression(MVEL.java:930)
                        at 
org.drools.base.mvel.MVELConsequence.evaluate(MVELConsequence.java:104)
                        at 
org.drools.common.DefaultAgenda.fireActivation(DefaultAgenda.java:1287)
                        at 
org.drools.common.DefaultAgenda.fireNextItem(DefaultAgenda.java:1221)
-- 
                Imesh Gunaratne

                Technical Lead, WSO2

                Committer & PMC Member, Apache Stratos
-- Imesh Gunaratne 

            Technical Lead, WSO2
            Committer & PMC Member, Apache Stratos
-- 
        Udara Liyanage
        Software Engineer
        WSO2, Inc.: http://wso2.com <http://wso2.com/>
        lean. enterprise. middleware

        web: http://udaraliyanage.wordpress.com
        phone: +94 71 443 6897
-- Imesh Gunaratne 

    Technical Lead, WSO2
    Committer & PMC Member, Apache Stratos





--
Imesh Gunaratne

Technical Lead, WSO2
Committer & PMC Member, Apache Stratos

Reply via email to