Hi,
I added forcefull undeployment which can be invoked as below.
curl -X POST -H "Content-Type: application/json" -k -v -u admin:admin
https://
${host_ip}:${host_port}/api/applications/${application_name}/undeploy?
*force=true*
This is supposed to be executed when graceful undeployment does not
complete successfully. This will try to terminate the instances once and
send MemberTerminatedEvent regardless of termination process is successful
or not. At the end Stratos should cleanup its internal memory and
application should become CREATED. If stratos could not terminate instance,
they should be manually terminated.
Forceful termination can be invoked even without invoking graceful
undeployment.
On Tue, Mar 31, 2015 at 9:15 AM, Lakmal Warusawithana <lak...@wso2.com>
wrote:
>
>
> On Mon, Mar 30, 2015 at 7:18 PM, Vanson Lim <v...@cisco.com> wrote:
>
>> On 3/30/15, 9:18 AM, Lakmal Warusawithana wrote:
>>
>> IMO forceful un deploy is enough no need to have forceful delete. If un
>> deployment success delete should work without any issues
>>
>>
>> I agree, forceful undeploy should be enough. I assume this means that
>> if I call undeploy (force=false) the first time and it doesn't work, I can
>> call undeploy(force=true).
>>
>> In order for these api's to be robust, the create/deploy APIs should
>> perform strict error checking and clean up of partial state when an
>> exceptions are thrown. The code seems to do sufficient error checking but
>> we keep encountering partial state left over when an exception is thrown.
>>
>> The undeploy force and/or delete APIs should ideally always cleanup even
>> when detecting inconsistencies. We seem to be voting to handle this as a
>> part of undeploy (force=true). I hope this means that an app MUST be
>> cleanly transitioned back to "Created" state when we use (undeploy=force)
>> and gracefully handle a subsequent deploy command.
>>
>>
> Yes, this is exactly what we need.
>
>
>> -Vanson
>>
>>
>>
>>
>>
>> On Monday, March 30, 2015, Imesh Gunaratne <im...@apache.org> wrote:
>>
>>> Thanks Vanson for pointing this scenario! Yes we would need to
>>> forcefully delete an application in certain scenarios.
>>>
>>> Udara: +1 For the suggestion!
>>>
>>> On Monday, March 30, 2015, Udara Liyanage <ud...@wso2.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> Sorry for the typo. We can add the flag as a query parameter.
>>>>
>>>> curl -X POST applications/single-cartridge-app/undeploy?*force=true*
>>>>
>>>> On Mon, Mar 30, 2015 at 10:04 AM, Udara Liyanage <ud...@wso2.com>
>>>> wrote:
>>>>
>>>> Hi Imesh,
>>>>
>>>> How about the blow addition to the API?
>>>>
>>>> curl -X POST applications/single-cartridge-app/undeploy/*force=true*
>>>>
>>>> On Mon, Mar 30, 2015 at 9:48 AM, Udara Liyanage <ud...@wso2.com> wrote:
>>>>
>>>>
>>>>
>>>> On Sun, Mar 29, 2015 at 10:34 PM, Vanson Lim <v...@cisco.com> wrote:
>>>>
>>>> On 3/27/15, 5:57 PM, Vanson Lim wrote:
>>>>
>>>> On 3/27/15, 5:52 PM, Imesh Gunaratne wrote:
>>>>
>>>> Hi Vanson,
>>>>
>>>> I added a validation in Autoscaler to avoid applications being
>>>> deleted while they are being undeployed in the background.
>>>>
>>>> Now the following error would be raised if this condition is met:
>>>> "Application cannot be deleted, undeployment process is still in progress:
>>>> [application-id] <application-id>".
>>>>
>>>> In addition I have implemented your suggestion to set the application
>>>> status to CREATED once the application is completely undeployed. Now we can
>>>> check from the API whether the application is undeployed properly. Please
>>>> take a pull and verify.
>>>>
>>>>
>>>>
>>>>
>>>> Took me a while to experiment with the failure cases.
>>>>
>>>> Some observations:
>>>>
>>>> 1) The change to check that the Application state changes to CREATED
>>>> works. I am able to check that application status returns to Created before
>>>> deleting.
>>>>
>>>> 2) Application delete throws tracebacks to the wso2carbon.log. IMO,
>>>> this should only be informational rather then throw exceptions.
>>>>
>>>> I guess the reason for throwing an RuntimeException is to notify SM
>>>> that an problem has occured while deleting the application. I will
>>>> investigate firther.
>>>>
>>>>
>>>> ie
>>>>
>>>> org.apache.stratos.autoscaler.exception.AutoScalerException:
>>>> Application is in deployed state, please undeploy it before deleting:
>>>> [application-id] cisco-sample-vm
>>>> at
>>>> org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl.deleteApplication(AutoscalerServiceImpl.java:411)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>> at
>>>> org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:212)
>>>> at
>>>> org.apache.axis2.rpc.receivers.RPCInOnlyMessageReceiver.invokeBusinessLogic(RPCInOnlyMessageReceiver.java:66)
>>>> at
>>>> org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:110)
>>>> at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
>>>> at
>>>> org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:172)
>>>> at
>>>> org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:146)
>>>> at
>>>> org.wso2.carbon.core.transports.CarbonServlet.doPost(CarbonServlet.java:231)
>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:755)
>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
>>>> at
>>>> org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
>>>> at
>>>> org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128)
>>>> at
>>>> org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68)
>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
>>>> at
>>>> org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68)
>>>> at
>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
>>>> at
>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>>>> at
>>>> org.wso2.carbon.tomcat.ext.filter.CharacterSetFilter.doFilter(CharacterSetFilter.java:61)
>>>> at
>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
>>>> at
>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>>>> at
>>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
>>>> at
>>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
>>>> at
>>>> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
>>>> at
>>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
>>>> at
>>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
>>>> at
>>>> org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:178)
>>>> at
>>>> org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
>>>> at
>>>> org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:56)
>>>> at
>>>> org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
>>>> at
>>>> org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:141)
>>>> at
>>>> org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:156)
>>>> at
>>>> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
>>>> at
>>>> org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:52)
>>>> at
>>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
>>>> at
>>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
>>>> at
>>>> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
>>>> at
>>>> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
>>>> at
>>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1653)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> TID: [0] [STRATOS] [2015-03-28 00:17:52,952] ERROR
>>>> {org.apache.axis2.rpc.receivers.RPCInOnlyMessageReceiver} - Could not
>>>> delete application: [application-id] cisco-sample-vm
>>>> java.lang.RuntimeException: Could not delete application:
>>>> [application-id] cisco-sample-vm
>>>> at
>>>> org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl.deleteApplication(AutoscalerServiceImpl.java:426)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>> at
>>>> org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:212)
>>>> at
>>>> org.apache.axis2.rpc.receivers.RPCInOnlyMessageReceiver.invokeBusinessLogic(RPCInOnlyMessageReceiver.java:66)
>>>> at
>>>> org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:110)
>>>> at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
>>>> at
>>>> org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:172)
>>>> at
>>>> org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:146)
>>>> at
>>>> org.wso2.carbon.core.transports.CarbonServlet.doPost(CarbonServlet.java:231)
>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:755)
>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
>>>> at
>>>> org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
>>>> at
>>>> org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128)
>>>> at
>>>> org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68)
>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
>>>> at
>>>> org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68)
>>>> at
>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
>>>> at
>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>>>> at
>>>> org.wso2.carbon.tomcat.ext.filter.CharacterSetFilter.doFilter(CharacterSetFilter.java:61)
>>>> at
>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
>>>> at
>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>>>> at
>>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
>>>> at
>>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
>>>> at
>>>> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
>>>> at
>>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
>>>> at
>>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
>>>> at
>>>> org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:178)
>>>> at
>>>> org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
>>>> at
>>>> org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:56)
>>>> at
>>>> org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
>>>> at
>>>> org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:141)
>>>> at
>>>> org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:156)
>>>> at
>>>> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
>>>> at
>>>> org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:52)
>>>> at
>>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
>>>> at
>>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
>>>> at
>>>> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
>>>> at
>>>> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
>>>> at
>>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1653)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> Caused by: org.apache.stratos.autoscaler.exception.AutoScalerException:
>>>> Application is in deployed state, please undeploy it before deleting:
>>>> [application-id] cisco-sample-vm
>>>> at
>>>> org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl.deleteApplication(AutoscalerServiceImpl.java:411)
>>>> ... 44 more
>>>>
>>>>
>>>> 3) I am hitting the "redeploy application issue" that was fixed
>>>> yesterday as a part of testing this fix and found that the application is
>>>> in a broken state after an application redeploy such that application
>>>> delete doesn't work anymore because the partially deployed application is
>>>> in stuck in the "Deployed" state.
>>>>
>>>> This is similar to a valid use case which application delete doesn't
>>>> currently handle. Our current implementation of application delete only
>>>> works gracefully if the application is well behaved. If we have a single
>>>> VM that is unresponsive to a terminate event, won't the application
>>>> permanently be stuck in a "Deployed" state? This leads me to think that
>>>> application delete needs to able to clean up all the state in the various
>>>> components that make up stratos instead of preventing the delete requests
>>>> regardless of the cluster instance counts or application status. In other
>>>> words, the request should always be honored, otherwise we have no way of
>>>> forcing the undeploy and deletion of an application. If the user wants to
>>>> be graceful, they have the option to poll the application state changes
>>>> using the application GET api before calling the application delete API.
>>>>
>>>>
>>>> Your point seems to be valid since with the above fix, application
>>>> will not be able to be deleted if it doesn't become created for some
>>>> reason. One solution is to let the user explicitly specify to delete
>>>> forcefully which will delete the application regardless of the state. I
>>>> will do a further analysis and get back to you.
>>>>
>>>>
>>>> -Vanson
>>>>
>>>> okay, will give it a try.
>>>>
>>>> -Vanson
>>>>
>>>> Thanks
>>>>
>>>> On Sat, Mar 28, 2015 at 2:40 AM, Vanson Lim <v...@cisco.com> wrote:
>>>>
>>>> On 3/27/15, 2:26 PM, Imesh Gunaratne wrote:
>>>>
>>>> Hi Vanson,
>>>>
>>>> I analyzed this issue. Logically we cannot delete an application soon
>>>> after un-deploying it. The reason is that the un-deployment process
>>>> executes gracefully on each member in all the clusters in an application.
>>>> Depending on the complexity of the application, it may take some time to
>>>> completely terminate all the members, remove the clusters and then
>>>> un-deploy the application.
>>>>
>>>> IMO what we are missing here is a validation in the delete
>>>> application API method, to tell the API client that an application cannot
>>>> be deleted until it is completely un-deployed.
>>>>
>>>> Imesh,
>>>>
>>>> Thanks for looking at this.
>>>>
>>>> It sounds reasonable to hold off the removing to application and also
>>>> have the remove rest api call return that it's not ready. For
>>>> applications consisting of a lot of groups and cartridges the time to clean
>>>> up could be variable. Could we use the Rest api to get the application
>>>> status, ie check that status=Created before deleting.
>>>>
>>>> ie this api:
>>>>
>>>>
>>>> https://cwiki.apache.org/confluence/display/STRATOS/4.1.0+Getting+Details+of+an+Application+via+REST+API
>>>>
>>>> It looks like an application has two states: "Created" and
>>>> "Deployed". If we had a Undeploying state, then we could check and hold
>>>> off
>>>> requesting the remove-application until it returns to the Created state.
>>>>
>>>> -Vanson
>>>>
>>>>
>>>> Thanks
>>>> Imesh
>>>>
>>>> On Wed, Mar 25, 2015 at 12:10 PM, Udara Liyanage <ud...@wso2.com>
>>>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> This arises when the application is deleted immediately after the
>>>> application underemployment. I guess CC does not have the member context
>>>> when the application is removed. I will have a look.
>>>>
>>>> A simple workaround it to wait sometime to remove application after
>>>> undeployment.
>>>>
>>>> On Tue, Mar 24, 2015 at 9:06 AM, Imesh Gunaratne <im...@apache.org>
>>>> wrote:
>>>>
>>>> Thanks Martin! We will have a look at this.
>>>>
>>>> On Tue, Mar 24, 2015 at 1:21 AM, Martin Eppel (meppel) <
>>>> mep...@cisco.com> wrote:
>>>>
>>>> I opened a jira https://issues.apache.org/jira/browse/STRATOS-1281 to
>>>> track the issue,
>>>>
>>>>
>>>>
>>>> Thanks
>>>>
>>>>
>>>>
>>>> Martin
>>>>
>>>>
>>>>
>>>> *From:* Imesh Gunaratne [mailto:im...@apache.org]
>>>> *Sent:* Friday, March 20, 2015 3:27 AM
>>>> *To:* dev
>>>> *Subject:* Re: Stratos 4.1.0 - tracebacks seen when issuing
>>>> application undeploy/remove
>>>>
>>>>
>>>>
>>>> Hi Vanson,
>>>>
>>>>
>>>>
>>>> Thanks for reporting this problem. According to the logs stratos is
>>>> trying to remove the same member twice in this flow:
>>>>
>>>>
>>>>
>>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,952] INFO
>>>> {org.apache.stratos.messaging.message.processor.application.ApplicationDeletedMessageProcessor}
>>>> - [Application] cisco-sample-vm has been successfully removed
>>>>
>>>> TID: [0] [STRATOS] [2015-03-19 19:45:44,786] INFO
>>>> {org.apache.stratos.cloud.controller.iaases.JcloudsIaas} - Member
>>>> terminated: [member-id]
>>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
>>>>
>>>>
>>>>
>>>> TID: [0] [STRATOS] [2015-03-19 19:45:50,012] INFO
>>>> {org.apache.stratos.common.client.CloudControllerServiceClient} -
>>>> Terminating instance via cloud controller: [member]
>>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
>>>>
>>>> TID: [0] [STRATOS] [2015-03-19 19:45:50,017] ERROR
>>>> {org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl}
>>>> - Could not terminate instance, member context not found: [member-id]
>>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
>>>>
>>>>
>>>>
>>>> We might need to investigate and see what's causing this.
>>>>
>>>>
>>>>
>>>> Thanks
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, Mar 20, 2015 at 6:55 AM, Vanson Lim <v...@cisco.com> wrote:
>>>>
>>>> Hi,
>>>>
>>>> We are testing the behavior of stratos 4.1.0 rest api's and found that
>>>> if we issue an undeploy-application followed immediately with
>>>> application-remove, that we see the following traceback.
>>>>
>>>> The VM successfully get's deleted but not sure what kind of side
>>>> effects this has on the system.
>>>>
>>>> Here's the snippet from the wso2carbon.log file. I've also attached
>>>> the entire log.
>>>>
>>>> Steps to reproduce this:
>>>>
>>>> 1) deploy an application which leads to startup a single instance of a
>>>> cartridge.
>>>> 2) wait for it to become active
>>>> 3) issue the undeploy application and application remove rest api calls.
>>>>
>>>>
>>>> -Vanson
>>>>
>>>>
>>>>
>>>> TID: [0] [STRATOS] [2015-03-19 19:45:08,681] INFO
>>>> {org.wso2.carbon.databridge.core.DataBridge} - admin connected
>>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,023] INFO
>>>> {org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl} -
>>>> Starting to undeploy application: [application-id] cisco-sample-vm
>>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,024] INFO
>>>> {org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl} -
>>>> Removing application signup: [application-id] cisco-sample-vm
>>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,051] INFO
>>>> {org.apache.stratos.manager.components.ApplicationSignUpHandler} - Removing
>>>> application signup: [application-id] cisco-sample-vm [tenant-id] -1234
>>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,084] INFO
>>>> {org.apache.stratos.manager.components.ApplicationSignUpHandler} -
>>>> Application signup removed successfully: [application-id] cisco-sample-vm
>>>> [tenant-id] -1234
>>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,091] INFO
>>>> {org.apache.stratos.autoscaler.context.AutoscalerContext} - Network
>>>> partition algorithm context is removed successfully: [id] cisco-sample-vm
>>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,092] INFO
>>>> {org.apache.stratos.autoscaler.monitor.cluster.ClusterMonitor} - Publishing
>>>> Cluster terminating event for [application] cisco-sample-vm [cluster]
>>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain [instance]
>>>> cisco-sample-vm-1
>>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,109] INFO
>>>> {org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder} -
>>>> Cluster Terminating adding status started
>>>> forcisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain
>>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,117] INFO
>>>> {org.apache.stratos.autoscaler.applications.topic.ApplicationsEventPublisher}
>>>> - Publishing application inactivated event: [application] cisco-sample-vm
>>>> [instance] cisco-sample-vm-1
>>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,120] INFO
>>>> {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher}
>>>> - Publishing Cluster terminating event: [application-id] cisco-sample-vm
>>>> [cluster id] cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain
>>>> [instance-id] cisco-sample-vm-1
>>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,144] INFO
>>>> {org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl} -
>>>> Application undeployed successfully: [application-id] cisco-sample-vm
>>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,154] INFO
>>>> {org.apache.stratos.autoscaler.event.receiver.topology.AutoscalerTopologyEventReceiver}
>>>> - [ClusterTerminatingEvent] Received: class
>>>> org.apache.stratos.messaging.event.topology.ClusterInstanceTerminatingEvent
>>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,155] INFO
>>>> {org.apache.stratos.autoscaler.event.publisher.InstanceNotificationPublisher}
>>>> - Publishing Instance Cleanup Event: [cluster]
>>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain
>>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,171] WARN
>>>> {org.apache.stratos.autoscaler.status.processor.cluster.ClusterStatusActiveProcessor}
>>>> - No possible state change found for [type] [cluster]
>>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain [instance]
>>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,192] INFO
>>>> {org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder} -
>>>> member maintenance mode event adding status started
>>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,204] INFO
>>>> {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher}
>>>> - Publishing member in maintenance mode event: [service-name]
>>>> cisco-sample-vm [cluster-id]
>>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain
>>>> [cluster-instance-id] cisco-sample-vm-1 [member-id]
>>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
>>>> [network-partition-id] RegionOne [partition-id] whole-region
>>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,230] INFO
>>>> {org.apache.stratos.messaging.message.processor.topology.MemberMaintenanceModeProcessor}
>>>> - Member updated as In_Maintenance: [service] cisco-sample-vm [cluster]
>>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain [member]
>>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
>>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,231] INFO
>>>> {org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder} -
>>>> Member Ready to shut down event adding status started
>>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,242] INFO
>>>> {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher}
>>>> - Publishing member ready to shut down event: [service-name]
>>>> cisco-sample-vm [cluster-id]
>>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain
>>>> [cluster-instance-id] cisco-sample-vm-1 [member-id]
>>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
>>>> [network-partition-id] RegionOne [partition-id] whole-region
>>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,262] INFO
>>>> {org.apache.stratos.messaging.message.processor.topology.MemberReadyToShutdownMessageProcessor}
>>>> - Member updated as Ready to shutdown: [service] cisco-sample-vm [cluster]
>>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain [member]
>>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
>>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,264] INFO
>>>> {org.apache.stratos.autoscaler.event.receiver.topology.AutoscalerTopologyEventReceiver}
>>>> - [MemberReadyToShutdownEvent] Received: class
>>>> org.apache.stratos.messaging.event.topology.MemberReadyToShutdownEvent
>>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,264] INFO
>>>> {org.apache.stratos.autoscaler.context.partition.ClusterLevelPartitionContext}
>>>> - Starting the moving of termination pending to obsolete for [member]
>>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
>>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,264] INFO
>>>> {org.apache.stratos.autoscaler.context.partition.ClusterLevelPartitionContext}
>>>> - Found termination pending member and trying to move [member]
>>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
>>>> to obsolete list
>>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,266] INFO
>>>> {org.apache.stratos.common.client.CloudControllerServiceClient} -
>>>> Terminating instance via cloud controller: [member]
>>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
>>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,275] INFO
>>>> {org.apache.stratos.cloud.controller.iaases.JcloudsIaas} - Starting to
>>>> terminate member: [cartridge-type] cisco-sample-vm [member-id]
>>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
>>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,883] INFO
>>>> {org.apache.stratos.messaging.domain.application.locking.ApplicationLockHierarchy}
>>>> - Removed lock for application: [application-id] cisco-sample-vm
>>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,909] INFO
>>>> {org.apache.stratos.cloud.controller.messaging.receiver.application.ApplicationEventReceiver}
>>>> - Application deleted event received: [application-id] cisco-sample-vm
>>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,909] INFO
>>>> {org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder} -
>>>> Removed application [ cisco-sample-vm ]'s Cluster [
>>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain ] from the topology
>>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,910] INFO
>>>> {org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl} -
>>>> Application deleted successfully: [application-id]
>>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,930] INFO
>>>> {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher}
>>>> - Publishing application clusters removed event: [application-id]
>>>> cisco-sample-vm
>>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,950] INFO
>>>> {org.apache.stratos.messaging.message.processor.topology.ApplicationClustersRemovedMessageProcessor}
>>>> - Cluster cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain removed
>>>> from topology for application cisco-sample-vm
>>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,952] INFO
>>>> {org.apache.stratos.messaging.message.processor.application.ApplicationDeletedMessageProcessor}
>>>> - [Application] cisco-sample-vm has been successfully removed
>>>> TID: [0] [STRATOS] [2015-03-19 19:45:44,786] INFO
>>>> {org.apache.stratos.cloud.controller.iaases.JcloudsIaas} - Member
>>>> terminated: [member-id]
>>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
>>>> TID: [0] [STRATOS] [2015-03-19 19:45:44,786] WARN
>>>> {org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder} -
>>>> Cluster cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain does not
>>>> exist
>>>> TID: [0] [STRATOS] [2015-03-19 19:45:50,012] INFO
>>>> {org.apache.stratos.common.client.CloudControllerServiceClient} -
>>>> Terminating instance via cloud controller: [member]
>>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
>>>> TID: [0] [STRATOS] [2015-03-19 19:45:50,017] ERROR
>>>> {org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl}
>>>> - Could not terminate instance, member context not found: [member-id]
>>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
>>>> TID: [0] [STRATOS] [2015-03-19 19:45:50,018] ERROR
>>>> {org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl}
>>>> - Could not terminate instance: [member-id]
>>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
>>>> org.apache.stratos.cloud.controller.exception.InvalidMemberException:
>>>> Could not terminate instance, member context not found: [member-id]
>>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
>>>> at
>>>> org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl.terminateInstance(CloudControllerServiceImpl.java:553)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>> at
>>>> org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:212)
>>>> at
>>>> org.apache.axis2.rpc.receivers.RPCInOnlyMessageReceiver.invokeBusinessLogic(RPCInOnlyMessageReceiver.java:66)
>>>> at
>>>> org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:110)
>>>> at
>>>> org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
>>>> at
>>>> org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:172)
>>>> at
>>>> org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:146)
>>>> at
>>>> org.wso2.carbon.core.transports.CarbonServlet.doPost(CarbonServlet.java:231)
>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:755)
>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
>>>> at
>>>> org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
>>>> at
>>>> org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128)
>>>> at
>>>> org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68)
>>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
>>>> at
>>>> org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68)
>>>> at
>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
>>>> at
>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>>>> at
>>>> org.wso2.carbon.tomcat.ext.filter.CharacterSetFilter.doFilter(CharacterSetFilter.java:61)
>>>> at
>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
>>>> at
>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>>>> at
>>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
>>>> at
>>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
>>>> at
>>>> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
>>>> at
>>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
>>>> at
>>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
>>>> at
>>>> org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:178)
>>>> at
>>>> org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
>>>> at
>>>> org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:56)
>>>> at
>>>> org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
>>>> at
>>>> org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:141)
>>>> at
>>>> org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:156)
>>>> at
>>>> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
>>>> at
>>>> org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:52)
>>>> at
>>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
>>>>
>>>>
>>>
>>> --
>>> Imesh Gunaratne
>>>
>>> Technical Lead, WSO2
>>> Committer & PMC Member, Apache Stratos
>>>
>>>
>>
>> --
>> Sent from Gmail Mobile
>>
>>
>>
>
>
> --
> Lakmal Warusawithana
> Vice President, Apache Stratos
> Director - Cloud Architecture; WSO2 Inc.
> Mobile : +94714289692
> Blog : http://lakmalsview.blogspot.com/
>
>
--
Udara Liyanage
Software Engineer
WSO2, Inc.: http://wso2.com
lean. enterprise. middleware
web: http://udaraliyanage.wordpress.com
phone: +94 71 443 6897
