On Mon, Mar 30, 2015 at 7:18 PM, Vanson Lim <v...@cisco.com> wrote:
> On 3/30/15, 9:18 AM, Lakmal Warusawithana wrote:
>
> IMO forceful un deploy is enough no need to have forceful delete. If un
> deployment success delete should work without any issues
>
>
> I agree, forceful undeploy should be enough. I assume this means that
> if I call undeploy (force=false) the first time and it doesn't work, I can
> call undeploy(force=true).
>
> In order for these api's to be robust, the create/deploy APIs should
> perform strict error checking and clean up of partial state when an
> exceptions are thrown. The code seems to do sufficient error checking but
> we keep encountering partial state left over when an exception is thrown.
>
> The undeploy force and/or delete APIs should ideally always cleanup even
> when detecting inconsistencies. We seem to be voting to handle this as a
> part of undeploy (force=true). I hope this means that an app MUST be
> cleanly transitioned back to "Created" state when we use (undeploy=force)
> and gracefully handle a subsequent deploy command.
>
>
Yes, this is exactly what we need.
> -Vanson
>
>
>
>
>
> On Monday, March 30, 2015, Imesh Gunaratne <im...@apache.org> wrote:
>
>> Thanks Vanson for pointing this scenario! Yes we would need to forcefully
>> delete an application in certain scenarios.
>>
>> Udara: +1 For the suggestion!
>>
>> On Monday, March 30, 2015, Udara Liyanage <ud...@wso2.com> wrote:
>>
>>> Hi,
>>>
>>> Sorry for the typo. We can add the flag as a query parameter.
>>>
>>> curl -X POST applications/single-cartridge-app/undeploy?*force=true*
>>>
>>> On Mon, Mar 30, 2015 at 10:04 AM, Udara Liyanage <ud...@wso2.com> wrote:
>>>
>>> Hi Imesh,
>>>
>>> How about the blow addition to the API?
>>>
>>> curl -X POST applications/single-cartridge-app/undeploy/*force=true*
>>>
>>> On Mon, Mar 30, 2015 at 9:48 AM, Udara Liyanage <ud...@wso2.com> wrote:
>>>
>>>
>>>
>>> On Sun, Mar 29, 2015 at 10:34 PM, Vanson Lim <v...@cisco.com> wrote:
>>>
>>> On 3/27/15, 5:57 PM, Vanson Lim wrote:
>>>
>>> On 3/27/15, 5:52 PM, Imesh Gunaratne wrote:
>>>
>>> Hi Vanson,
>>>
>>> I added a validation in Autoscaler to avoid applications being deleted
>>> while they are being undeployed in the background.
>>>
>>> Now the following error would be raised if this condition is met:
>>> "Application cannot be deleted, undeployment process is still in progress:
>>> [application-id] <application-id>".
>>>
>>> In addition I have implemented your suggestion to set the application
>>> status to CREATED once the application is completely undeployed. Now we can
>>> check from the API whether the application is undeployed properly. Please
>>> take a pull and verify.
>>>
>>>
>>>
>>>
>>> Took me a while to experiment with the failure cases.
>>>
>>> Some observations:
>>>
>>> 1) The change to check that the Application state changes to CREATED
>>> works. I am able to check that application status returns to Created before
>>> deleting.
>>>
>>> 2) Application delete throws tracebacks to the wso2carbon.log. IMO,
>>> this should only be informational rather then throw exceptions.
>>>
>>> I guess the reason for throwing an RuntimeException is to notify SM that
>>> an problem has occured while deleting the application. I will investigate
>>> firther.
>>>
>>>
>>> ie
>>>
>>> org.apache.stratos.autoscaler.exception.AutoScalerException: Application
>>> is in deployed state, please undeploy it before deleting: [application-id]
>>> cisco-sample-vm
>>> at
>>> org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl.deleteApplication(AutoscalerServiceImpl.java:411)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>> at
>>> org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:212)
>>> at
>>> org.apache.axis2.rpc.receivers.RPCInOnlyMessageReceiver.invokeBusinessLogic(RPCInOnlyMessageReceiver.java:66)
>>> at
>>> org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:110)
>>> at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
>>> at
>>> org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:172)
>>> at
>>> org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:146)
>>> at
>>> org.wso2.carbon.core.transports.CarbonServlet.doPost(CarbonServlet.java:231)
>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:755)
>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
>>> at
>>> org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
>>> at
>>> org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128)
>>> at
>>> org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68)
>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
>>> at
>>> org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68)
>>> at
>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
>>> at
>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>>> at
>>> org.wso2.carbon.tomcat.ext.filter.CharacterSetFilter.doFilter(CharacterSetFilter.java:61)
>>> at
>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
>>> at
>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>>> at
>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
>>> at
>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
>>> at
>>> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
>>> at
>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
>>> at
>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
>>> at
>>> org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:178)
>>> at
>>> org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
>>> at
>>> org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:56)
>>> at
>>> org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
>>> at
>>> org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:141)
>>> at
>>> org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:156)
>>> at
>>> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
>>> at
>>> org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:52)
>>> at
>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
>>> at
>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
>>> at
>>> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
>>> at
>>> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
>>> at
>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1653)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> at java.lang.Thread.run(Thread.java:745)
>>> TID: [0] [STRATOS] [2015-03-28 00:17:52,952] ERROR
>>> {org.apache.axis2.rpc.receivers.RPCInOnlyMessageReceiver} - Could not
>>> delete application: [application-id] cisco-sample-vm
>>> java.lang.RuntimeException: Could not delete application:
>>> [application-id] cisco-sample-vm
>>> at
>>> org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl.deleteApplication(AutoscalerServiceImpl.java:426)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>> at
>>> org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:212)
>>> at
>>> org.apache.axis2.rpc.receivers.RPCInOnlyMessageReceiver.invokeBusinessLogic(RPCInOnlyMessageReceiver.java:66)
>>> at
>>> org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:110)
>>> at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
>>> at
>>> org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:172)
>>> at
>>> org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:146)
>>> at
>>> org.wso2.carbon.core.transports.CarbonServlet.doPost(CarbonServlet.java:231)
>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:755)
>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
>>> at
>>> org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
>>> at
>>> org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128)
>>> at
>>> org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68)
>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
>>> at
>>> org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68)
>>> at
>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
>>> at
>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>>> at
>>> org.wso2.carbon.tomcat.ext.filter.CharacterSetFilter.doFilter(CharacterSetFilter.java:61)
>>> at
>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
>>> at
>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>>> at
>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
>>> at
>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
>>> at
>>> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
>>> at
>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
>>> at
>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
>>> at
>>> org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:178)
>>> at
>>> org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
>>> at
>>> org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:56)
>>> at
>>> org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
>>> at
>>> org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:141)
>>> at
>>> org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:156)
>>> at
>>> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
>>> at
>>> org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:52)
>>> at
>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
>>> at
>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
>>> at
>>> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
>>> at
>>> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
>>> at
>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1653)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>> at java.lang.Thread.run(Thread.java:745)
>>> Caused by: org.apache.stratos.autoscaler.exception.AutoScalerException:
>>> Application is in deployed state, please undeploy it before deleting:
>>> [application-id] cisco-sample-vm
>>> at
>>> org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl.deleteApplication(AutoscalerServiceImpl.java:411)
>>> ... 44 more
>>>
>>>
>>> 3) I am hitting the "redeploy application issue" that was fixed
>>> yesterday as a part of testing this fix and found that the application is
>>> in a broken state after an application redeploy such that application
>>> delete doesn't work anymore because the partially deployed application is
>>> in stuck in the "Deployed" state.
>>>
>>> This is similar to a valid use case which application delete doesn't
>>> currently handle. Our current implementation of application delete only
>>> works gracefully if the application is well behaved. If we have a single
>>> VM that is unresponsive to a terminate event, won't the application
>>> permanently be stuck in a "Deployed" state? This leads me to think that
>>> application delete needs to able to clean up all the state in the various
>>> components that make up stratos instead of preventing the delete requests
>>> regardless of the cluster instance counts or application status. In other
>>> words, the request should always be honored, otherwise we have no way of
>>> forcing the undeploy and deletion of an application. If the user wants to
>>> be graceful, they have the option to poll the application state changes
>>> using the application GET api before calling the application delete API.
>>>
>>>
>>> Your point seems to be valid since with the above fix, application
>>> will not be able to be deleted if it doesn't become created for some
>>> reason. One solution is to let the user explicitly specify to delete
>>> forcefully which will delete the application regardless of the state. I
>>> will do a further analysis and get back to you.
>>>
>>>
>>> -Vanson
>>>
>>> okay, will give it a try.
>>>
>>> -Vanson
>>>
>>> Thanks
>>>
>>> On Sat, Mar 28, 2015 at 2:40 AM, Vanson Lim <v...@cisco.com> wrote:
>>>
>>> On 3/27/15, 2:26 PM, Imesh Gunaratne wrote:
>>>
>>> Hi Vanson,
>>>
>>> I analyzed this issue. Logically we cannot delete an application soon
>>> after un-deploying it. The reason is that the un-deployment process
>>> executes gracefully on each member in all the clusters in an application.
>>> Depending on the complexity of the application, it may take some time to
>>> completely terminate all the members, remove the clusters and then
>>> un-deploy the application.
>>>
>>> IMO what we are missing here is a validation in the delete application
>>> API method, to tell the API client that an application cannot be deleted
>>> until it is completely un-deployed.
>>>
>>> Imesh,
>>>
>>> Thanks for looking at this.
>>>
>>> It sounds reasonable to hold off the removing to application and also
>>> have the remove rest api call return that it's not ready. For
>>> applications consisting of a lot of groups and cartridges the time to clean
>>> up could be variable. Could we use the Rest api to get the application
>>> status, ie check that status=Created before deleting.
>>>
>>> ie this api:
>>>
>>>
>>> https://cwiki.apache.org/confluence/display/STRATOS/4.1.0+Getting+Details+of+an+Application+via+REST+API
>>>
>>> It looks like an application has two states: "Created" and
>>> "Deployed". If we had a Undeploying state, then we could check and hold
>>> off
>>> requesting the remove-application until it returns to the Created state.
>>>
>>> -Vanson
>>>
>>>
>>> Thanks
>>> Imesh
>>>
>>> On Wed, Mar 25, 2015 at 12:10 PM, Udara Liyanage <ud...@wso2.com> wrote:
>>>
>>> Hi,
>>>
>>> This arises when the application is deleted immediately after the
>>> application underemployment. I guess CC does not have the member context
>>> when the application is removed. I will have a look.
>>>
>>> A simple workaround it to wait sometime to remove application after
>>> undeployment.
>>>
>>> On Tue, Mar 24, 2015 at 9:06 AM, Imesh Gunaratne <im...@apache.org>
>>> wrote:
>>>
>>> Thanks Martin! We will have a look at this.
>>>
>>> On Tue, Mar 24, 2015 at 1:21 AM, Martin Eppel (meppel) <mep...@cisco.com
>>> > wrote:
>>>
>>> I opened a jira https://issues.apache.org/jira/browse/STRATOS-1281 to
>>> track the issue,
>>>
>>>
>>>
>>> Thanks
>>>
>>>
>>>
>>> Martin
>>>
>>>
>>>
>>> *From:* Imesh Gunaratne [mailto:im...@apache.org]
>>> *Sent:* Friday, March 20, 2015 3:27 AM
>>> *To:* dev
>>> *Subject:* Re: Stratos 4.1.0 - tracebacks seen when issuing application
>>> undeploy/remove
>>>
>>>
>>>
>>> Hi Vanson,
>>>
>>>
>>>
>>> Thanks for reporting this problem. According to the logs stratos is
>>> trying to remove the same member twice in this flow:
>>>
>>>
>>>
>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,952] INFO
>>> {org.apache.stratos.messaging.message.processor.application.ApplicationDeletedMessageProcessor}
>>> - [Application] cisco-sample-vm has been successfully removed
>>>
>>> TID: [0] [STRATOS] [2015-03-19 19:45:44,786] INFO
>>> {org.apache.stratos.cloud.controller.iaases.JcloudsIaas} - Member
>>> terminated: [member-id]
>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
>>>
>>>
>>>
>>> TID: [0] [STRATOS] [2015-03-19 19:45:50,012] INFO
>>> {org.apache.stratos.common.client.CloudControllerServiceClient} -
>>> Terminating instance via cloud controller: [member]
>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
>>>
>>> TID: [0] [STRATOS] [2015-03-19 19:45:50,017] ERROR
>>> {org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl}
>>> - Could not terminate instance, member context not found: [member-id]
>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
>>>
>>>
>>>
>>> We might need to investigate and see what's causing this.
>>>
>>>
>>>
>>> Thanks
>>>
>>>
>>>
>>>
>>>
>>> On Fri, Mar 20, 2015 at 6:55 AM, Vanson Lim <v...@cisco.com> wrote:
>>>
>>> Hi,
>>>
>>> We are testing the behavior of stratos 4.1.0 rest api's and found that
>>> if we issue an undeploy-application followed immediately with
>>> application-remove, that we see the following traceback.
>>>
>>> The VM successfully get's deleted but not sure what kind of side effects
>>> this has on the system.
>>>
>>> Here's the snippet from the wso2carbon.log file. I've also attached the
>>> entire log.
>>>
>>> Steps to reproduce this:
>>>
>>> 1) deploy an application which leads to startup a single instance of a
>>> cartridge.
>>> 2) wait for it to become active
>>> 3) issue the undeploy application and application remove rest api calls.
>>>
>>>
>>> -Vanson
>>>
>>>
>>>
>>> TID: [0] [STRATOS] [2015-03-19 19:45:08,681] INFO
>>> {org.wso2.carbon.databridge.core.DataBridge} - admin connected
>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,023] INFO
>>> {org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl} -
>>> Starting to undeploy application: [application-id] cisco-sample-vm
>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,024] INFO
>>> {org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl} -
>>> Removing application signup: [application-id] cisco-sample-vm
>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,051] INFO
>>> {org.apache.stratos.manager.components.ApplicationSignUpHandler} - Removing
>>> application signup: [application-id] cisco-sample-vm [tenant-id] -1234
>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,084] INFO
>>> {org.apache.stratos.manager.components.ApplicationSignUpHandler} -
>>> Application signup removed successfully: [application-id] cisco-sample-vm
>>> [tenant-id] -1234
>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,091] INFO
>>> {org.apache.stratos.autoscaler.context.AutoscalerContext} - Network
>>> partition algorithm context is removed successfully: [id] cisco-sample-vm
>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,092] INFO
>>> {org.apache.stratos.autoscaler.monitor.cluster.ClusterMonitor} - Publishing
>>> Cluster terminating event for [application] cisco-sample-vm [cluster]
>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain [instance]
>>> cisco-sample-vm-1
>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,109] INFO
>>> {org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder} -
>>> Cluster Terminating adding status started
>>> forcisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain
>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,117] INFO
>>> {org.apache.stratos.autoscaler.applications.topic.ApplicationsEventPublisher}
>>> - Publishing application inactivated event: [application] cisco-sample-vm
>>> [instance] cisco-sample-vm-1
>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,120] INFO
>>> {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher}
>>> - Publishing Cluster terminating event: [application-id] cisco-sample-vm
>>> [cluster id] cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain
>>> [instance-id] cisco-sample-vm-1
>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,144] INFO
>>> {org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl} -
>>> Application undeployed successfully: [application-id] cisco-sample-vm
>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,154] INFO
>>> {org.apache.stratos.autoscaler.event.receiver.topology.AutoscalerTopologyEventReceiver}
>>> - [ClusterTerminatingEvent] Received: class
>>> org.apache.stratos.messaging.event.topology.ClusterInstanceTerminatingEvent
>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,155] INFO
>>> {org.apache.stratos.autoscaler.event.publisher.InstanceNotificationPublisher}
>>> - Publishing Instance Cleanup Event: [cluster]
>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain
>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,171] WARN
>>> {org.apache.stratos.autoscaler.status.processor.cluster.ClusterStatusActiveProcessor}
>>> - No possible state change found for [type] [cluster]
>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain [instance]
>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,192] INFO
>>> {org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder} -
>>> member maintenance mode event adding status started
>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,204] INFO
>>> {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher}
>>> - Publishing member in maintenance mode event: [service-name]
>>> cisco-sample-vm [cluster-id]
>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain
>>> [cluster-instance-id] cisco-sample-vm-1 [member-id]
>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
>>> [network-partition-id] RegionOne [partition-id] whole-region
>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,230] INFO
>>> {org.apache.stratos.messaging.message.processor.topology.MemberMaintenanceModeProcessor}
>>> - Member updated as In_Maintenance: [service] cisco-sample-vm [cluster]
>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain [member]
>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,231] INFO
>>> {org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder} -
>>> Member Ready to shut down event adding status started
>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,242] INFO
>>> {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher}
>>> - Publishing member ready to shut down event: [service-name]
>>> cisco-sample-vm [cluster-id]
>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain
>>> [cluster-instance-id] cisco-sample-vm-1 [member-id]
>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
>>> [network-partition-id] RegionOne [partition-id] whole-region
>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,262] INFO
>>> {org.apache.stratos.messaging.message.processor.topology.MemberReadyToShutdownMessageProcessor}
>>> - Member updated as Ready to shutdown: [service] cisco-sample-vm [cluster]
>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain [member]
>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,264] INFO
>>> {org.apache.stratos.autoscaler.event.receiver.topology.AutoscalerTopologyEventReceiver}
>>> - [MemberReadyToShutdownEvent] Received: class
>>> org.apache.stratos.messaging.event.topology.MemberReadyToShutdownEvent
>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,264] INFO
>>> {org.apache.stratos.autoscaler.context.partition.ClusterLevelPartitionContext}
>>> - Starting the moving of termination pending to obsolete for [member]
>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,264] INFO
>>> {org.apache.stratos.autoscaler.context.partition.ClusterLevelPartitionContext}
>>> - Found termination pending member and trying to move [member]
>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
>>> to obsolete list
>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,266] INFO
>>> {org.apache.stratos.common.client.CloudControllerServiceClient} -
>>> Terminating instance via cloud controller: [member]
>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,275] INFO
>>> {org.apache.stratos.cloud.controller.iaases.JcloudsIaas} - Starting to
>>> terminate member: [cartridge-type] cisco-sample-vm [member-id]
>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,883] INFO
>>> {org.apache.stratos.messaging.domain.application.locking.ApplicationLockHierarchy}
>>> - Removed lock for application: [application-id] cisco-sample-vm
>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,909] INFO
>>> {org.apache.stratos.cloud.controller.messaging.receiver.application.ApplicationEventReceiver}
>>> - Application deleted event received: [application-id] cisco-sample-vm
>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,909] INFO
>>> {org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder} -
>>> Removed application [ cisco-sample-vm ]'s Cluster [
>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain ] from the topology
>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,910] INFO
>>> {org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl} -
>>> Application deleted successfully: [application-id]
>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,930] INFO
>>> {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher}
>>> - Publishing application clusters removed event: [application-id]
>>> cisco-sample-vm
>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,950] INFO
>>> {org.apache.stratos.messaging.message.processor.topology.ApplicationClustersRemovedMessageProcessor}
>>> - Cluster cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain removed
>>> from topology for application cisco-sample-vm
>>> TID: [0] [STRATOS] [2015-03-19 19:45:43,952] INFO
>>> {org.apache.stratos.messaging.message.processor.application.ApplicationDeletedMessageProcessor}
>>> - [Application] cisco-sample-vm has been successfully removed
>>> TID: [0] [STRATOS] [2015-03-19 19:45:44,786] INFO
>>> {org.apache.stratos.cloud.controller.iaases.JcloudsIaas} - Member
>>> terminated: [member-id]
>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
>>> TID: [0] [STRATOS] [2015-03-19 19:45:44,786] WARN
>>> {org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder} -
>>> Cluster cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain does not
>>> exist
>>> TID: [0] [STRATOS] [2015-03-19 19:45:50,012] INFO
>>> {org.apache.stratos.common.client.CloudControllerServiceClient} -
>>> Terminating instance via cloud controller: [member]
>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
>>> TID: [0] [STRATOS] [2015-03-19 19:45:50,017] ERROR
>>> {org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl}
>>> - Could not terminate instance, member context not found: [member-id]
>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
>>> TID: [0] [STRATOS] [2015-03-19 19:45:50,018] ERROR
>>> {org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl}
>>> - Could not terminate instance: [member-id]
>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
>>> org.apache.stratos.cloud.controller.exception.InvalidMemberException:
>>> Could not terminate instance, member context not found: [member-id]
>>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
>>> at
>>> org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl.terminateInstance(CloudControllerServiceImpl.java:553)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>> at
>>> org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:212)
>>> at
>>> org.apache.axis2.rpc.receivers.RPCInOnlyMessageReceiver.invokeBusinessLogic(RPCInOnlyMessageReceiver.java:66)
>>> at
>>> org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:110)
>>> at
>>> org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
>>> at
>>> org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:172)
>>> at
>>> org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:146)
>>> at
>>> org.wso2.carbon.core.transports.CarbonServlet.doPost(CarbonServlet.java:231)
>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:755)
>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
>>> at
>>> org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
>>> at
>>> org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128)
>>> at
>>> org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68)
>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
>>> at
>>> org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68)
>>> at
>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
>>> at
>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>>> at
>>> org.wso2.carbon.tomcat.ext.filter.CharacterSetFilter.doFilter(CharacterSetFilter.java:61)
>>> at
>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
>>> at
>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
>>> at
>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
>>> at
>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
>>> at
>>> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
>>> at
>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
>>> at
>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
>>> at
>>> org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:178)
>>> at
>>> org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
>>> at
>>> org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:56)
>>> at
>>> org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
>>> at
>>> org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:141)
>>> at
>>> org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:156)
>>> at
>>> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
>>> at
>>> org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:52)
>>> at
>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
>>>
>>>
>>
>> --
>> Imesh Gunaratne
>>
>> Technical Lead, WSO2
>> Committer & PMC Member, Apache Stratos
>>
>>
>
> --
> Sent from Gmail Mobile
>
>
>
--
Lakmal Warusawithana
Vice President, Apache Stratos
Director - Cloud Architecture; WSO2 Inc.
Mobile : +94714289692
Blog : http://lakmalsview.blogspot.com/
