+1 Yes Lakmal, a good point! We might not need a forceful application delete, a forceful application un-deployment would be sufficient.
Thanks On Mon, Mar 30, 2015 at 6:48 PM, Lakmal Warusawithana <lak...@wso2.com> wrote: > IMO forceful un deploy is enough no need to have forceful delete. If un > deployment success delete should work without any issues > > > On Monday, March 30, 2015, Imesh Gunaratne <im...@apache.org> wrote: > >> Thanks Vanson for pointing this scenario! Yes we would need to forcefully >> delete an application in certain scenarios. >> >> Udara: +1 For the suggestion! >> >> On Monday, March 30, 2015, Udara Liyanage <ud...@wso2.com> wrote: >> >>> Hi, >>> >>> Sorry for the typo. We can add the flag as a query parameter. >>> >>> curl -X POST applications/single-cartridge-app/undeploy?*force=true* >>> >>> On Mon, Mar 30, 2015 at 10:04 AM, Udara Liyanage <ud...@wso2.com> wrote: >>> >>> Hi Imesh, >>> >>> How about the blow addition to the API? >>> >>> curl -X POST applications/single-cartridge-app/undeploy/*force=true* >>> >>> On Mon, Mar 30, 2015 at 9:48 AM, Udara Liyanage <ud...@wso2.com> wrote: >>> >>> >>> >>> On Sun, Mar 29, 2015 at 10:34 PM, Vanson Lim <v...@cisco.com> wrote: >>> >>> On 3/27/15, 5:57 PM, Vanson Lim wrote: >>> >>> On 3/27/15, 5:52 PM, Imesh Gunaratne wrote: >>> >>> Hi Vanson, >>> >>> I added a validation in Autoscaler to avoid applications being deleted >>> while they are being undeployed in the background. >>> >>> Now the following error would be raised if this condition is met: >>> "Application cannot be deleted, undeployment process is still in progress: >>> [application-id] <application-id>". >>> >>> In addition I have implemented your suggestion to set the application >>> status to CREATED once the application is completely undeployed. Now we can >>> check from the API whether the application is undeployed properly. Please >>> take a pull and verify. >>> >>> >>> >>> >>> Took me a while to experiment with the failure cases. >>> >>> Some observations: >>> >>> 1) The change to check that the Application state changes to CREATED >>> works. I am able to check that application status returns to Created before >>> deleting. >>> >>> 2) Application delete throws tracebacks to the wso2carbon.log. IMO, >>> this should only be informational rather then throw exceptions. >>> >>> I guess the reason for throwing an RuntimeException is to notify SM that >>> an problem has occured while deleting the application. I will investigate >>> firther. >>> >>> >>> ie >>> >>> org.apache.stratos.autoscaler.exception.AutoScalerException: Application >>> is in deployed state, please undeploy it before deleting: [application-id] >>> cisco-sample-vm >>> at >>> org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl.deleteApplication(AutoscalerServiceImpl.java:411) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:606) >>> at >>> org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:212) >>> at >>> org.apache.axis2.rpc.receivers.RPCInOnlyMessageReceiver.invokeBusinessLogic(RPCInOnlyMessageReceiver.java:66) >>> at >>> org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:110) >>> at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180) >>> at >>> org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:172) >>> at >>> org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:146) >>> at >>> org.wso2.carbon.core.transports.CarbonServlet.doPost(CarbonServlet.java:231) >>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:755) >>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) >>> at >>> org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61) >>> at >>> org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128) >>> at >>> org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68) >>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) >>> at >>> org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68) >>> at >>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) >>> at >>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) >>> at >>> org.wso2.carbon.tomcat.ext.filter.CharacterSetFilter.doFilter(CharacterSetFilter.java:61) >>> at >>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) >>> at >>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) >>> at >>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222) >>> at >>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) >>> at >>> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472) >>> at >>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) >>> at >>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99) >>> at >>> org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:178) >>> at >>> org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47) >>> at >>> org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:56) >>> at >>> org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47) >>> at >>> org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:141) >>> at >>> org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:156) >>> at >>> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936) >>> at >>> org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:52) >>> at >>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) >>> at >>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407) >>> at >>> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004) >>> at >>> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589) >>> at >>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1653) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>> at java.lang.Thread.run(Thread.java:745) >>> TID: [0] [STRATOS] [2015-03-28 00:17:52,952] ERROR >>> {org.apache.axis2.rpc.receivers.RPCInOnlyMessageReceiver} - Could not >>> delete application: [application-id] cisco-sample-vm >>> java.lang.RuntimeException: Could not delete application: >>> [application-id] cisco-sample-vm >>> at >>> org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl.deleteApplication(AutoscalerServiceImpl.java:426) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:606) >>> at >>> org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:212) >>> at >>> org.apache.axis2.rpc.receivers.RPCInOnlyMessageReceiver.invokeBusinessLogic(RPCInOnlyMessageReceiver.java:66) >>> at >>> org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:110) >>> at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180) >>> at >>> org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:172) >>> at >>> org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:146) >>> at >>> org.wso2.carbon.core.transports.CarbonServlet.doPost(CarbonServlet.java:231) >>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:755) >>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) >>> at >>> org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61) >>> at >>> org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128) >>> at >>> org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68) >>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) >>> at >>> org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68) >>> at >>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) >>> at >>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) >>> at >>> org.wso2.carbon.tomcat.ext.filter.CharacterSetFilter.doFilter(CharacterSetFilter.java:61) >>> at >>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) >>> at >>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) >>> at >>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222) >>> at >>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) >>> at >>> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472) >>> at >>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) >>> at >>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99) >>> at >>> org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:178) >>> at >>> org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47) >>> at >>> org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:56) >>> at >>> org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47) >>> at >>> org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:141) >>> at >>> org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:156) >>> at >>> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936) >>> at >>> org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:52) >>> at >>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) >>> at >>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407) >>> at >>> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004) >>> at >>> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589) >>> at >>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1653) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>> at java.lang.Thread.run(Thread.java:745) >>> Caused by: org.apache.stratos.autoscaler.exception.AutoScalerException: >>> Application is in deployed state, please undeploy it before deleting: >>> [application-id] cisco-sample-vm >>> at >>> org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl.deleteApplication(AutoscalerServiceImpl.java:411) >>> ... 44 more >>> >>> >>> 3) I am hitting the "redeploy application issue" that was fixed >>> yesterday as a part of testing this fix and found that the application is >>> in a broken state after an application redeploy such that application >>> delete doesn't work anymore because the partially deployed application is >>> in stuck in the "Deployed" state. >>> >>> This is similar to a valid use case which application delete doesn't >>> currently handle. Our current implementation of application delete only >>> works gracefully if the application is well behaved. If we have a single >>> VM that is unresponsive to a terminate event, won't the application >>> permanently be stuck in a "Deployed" state? This leads me to think that >>> application delete needs to able to clean up all the state in the various >>> components that make up stratos instead of preventing the delete requests >>> regardless of the cluster instance counts or application status. In other >>> words, the request should always be honored, otherwise we have no way of >>> forcing the undeploy and deletion of an application. If the user wants to >>> be graceful, they have the option to poll the application state changes >>> using the application GET api before calling the application delete API. >>> >>> >>> Your point seems to be valid since with the above fix, application will >>> not be able to be deleted if it doesn't become created for some reason. One >>> solution is to let the user explicitly specify to delete forcefully which >>> will delete the application regardless of the state. I will do a further >>> analysis and get back to you. >>> >>> >>> -Vanson >>> >>> okay, will give it a try. >>> >>> -Vanson >>> >>> Thanks >>> >>> On Sat, Mar 28, 2015 at 2:40 AM, Vanson Lim <v...@cisco.com> wrote: >>> >>> On 3/27/15, 2:26 PM, Imesh Gunaratne wrote: >>> >>> Hi Vanson, >>> >>> I analyzed this issue. Logically we cannot delete an application soon >>> after un-deploying it. The reason is that the un-deployment process >>> executes gracefully on each member in all the clusters in an application. >>> Depending on the complexity of the application, it may take some time to >>> completely terminate all the members, remove the clusters and then >>> un-deploy the application. >>> >>> IMO what we are missing here is a validation in the delete application >>> API method, to tell the API client that an application cannot be deleted >>> until it is completely un-deployed. >>> >>> Imesh, >>> >>> Thanks for looking at this. >>> >>> It sounds reasonable to hold off the removing to application and also >>> have the remove rest api call return that it's not ready. For >>> applications consisting of a lot of groups and cartridges the time to clean >>> up could be variable. Could we use the Rest api to get the application >>> status, ie check that status=Created before deleting. >>> >>> ie this api: >>> >>> >>> https://cwiki.apache.org/confluence/display/STRATOS/4.1.0+Getting+Details+of+an+Application+via+REST+API >>> >>> It looks like an application has two states: "Created" and >>> "Deployed". If we had a Undeploying state, then we could check and hold >>> off >>> requesting the remove-application until it returns to the Created state. >>> >>> -Vanson >>> >>> >>> Thanks >>> Imesh >>> >>> On Wed, Mar 25, 2015 at 12:10 PM, Udara Liyanage <ud...@wso2.com> wrote: >>> >>> Hi, >>> >>> This arises when the application is deleted immediately after the >>> application underemployment. I guess CC does not have the member context >>> when the application is removed. I will have a look. >>> >>> A simple workaround it to wait sometime to remove application after >>> undeployment. >>> >>> On Tue, Mar 24, 2015 at 9:06 AM, Imesh Gunaratne <im...@apache.org> >>> wrote: >>> >>> Thanks Martin! We will have a look at this. >>> >>> On Tue, Mar 24, 2015 at 1:21 AM, Martin Eppel (meppel) <mep...@cisco.com >>> > wrote: >>> >>> I opened a jira https://issues.apache.org/jira/browse/STRATOS-1281 to >>> track the issue, >>> >>> >>> >>> Thanks >>> >>> >>> >>> Martin >>> >>> >>> >>> *From:* Imesh Gunaratne [mailto:im...@apache.org] >>> *Sent:* Friday, March 20, 2015 3:27 AM >>> *To:* dev >>> *Subject:* Re: Stratos 4.1.0 - tracebacks seen when issuing application >>> undeploy/remove >>> >>> >>> >>> Hi Vanson, >>> >>> >>> >>> Thanks for reporting this problem. According to the logs stratos is >>> trying to remove the same member twice in this flow: >>> >>> >>> >>> TID: [0] [STRATOS] [2015-03-19 19:45:43,952] INFO >>> {org.apache.stratos.messaging.message.processor.application.ApplicationDeletedMessageProcessor} >>> - [Application] cisco-sample-vm has been successfully removed >>> >>> TID: [0] [STRATOS] [2015-03-19 19:45:44,786] INFO >>> {org.apache.stratos.cloud.controller.iaases.JcloudsIaas} - Member >>> terminated: [member-id] >>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d >>> >>> >>> >>> TID: [0] [STRATOS] [2015-03-19 19:45:50,012] INFO >>> {org.apache.stratos.common.client.CloudControllerServiceClient} - >>> Terminating instance via cloud controller: [member] >>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d >>> >>> TID: [0] [STRATOS] [2015-03-19 19:45:50,017] ERROR >>> {org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl} >>> - Could not terminate instance, member context not found: [member-id] >>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d >>> >>> >>> >>> We might need to investigate and see what's causing this. >>> >>> >>> >>> Thanks >>> >>> >>> >>> >>> >>> On Fri, Mar 20, 2015 at 6:55 AM, Vanson Lim <v...@cisco.com> wrote: >>> >>> Hi, >>> >>> We are testing the behavior of stratos 4.1.0 rest api's and found that >>> if we issue an undeploy-application followed immediately with >>> application-remove, that we see the following traceback. >>> >>> The VM successfully get's deleted but not sure what kind of side effects >>> this has on the system. >>> >>> Here's the snippet from the wso2carbon.log file. I've also attached the >>> entire log. >>> >>> Steps to reproduce this: >>> >>> 1) deploy an application which leads to startup a single instance of a >>> cartridge. >>> 2) wait for it to become active >>> 3) issue the undeploy application and application remove rest api calls. >>> >>> >>> -Vanson >>> >>> >>> >>> TID: [0] [STRATOS] [2015-03-19 19:45:08,681] INFO >>> {org.wso2.carbon.databridge.core.DataBridge} - admin connected >>> TID: [0] [STRATOS] [2015-03-19 19:45:43,023] INFO >>> {org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl} - >>> Starting to undeploy application: [application-id] cisco-sample-vm >>> TID: [0] [STRATOS] [2015-03-19 19:45:43,024] INFO >>> {org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl} - >>> Removing application signup: [application-id] cisco-sample-vm >>> TID: [0] [STRATOS] [2015-03-19 19:45:43,051] INFO >>> {org.apache.stratos.manager.components.ApplicationSignUpHandler} - Removing >>> application signup: [application-id] cisco-sample-vm [tenant-id] -1234 >>> TID: [0] [STRATOS] [2015-03-19 19:45:43,084] INFO >>> {org.apache.stratos.manager.components.ApplicationSignUpHandler} - >>> Application signup removed successfully: [application-id] cisco-sample-vm >>> [tenant-id] -1234 >>> TID: [0] [STRATOS] [2015-03-19 19:45:43,091] INFO >>> {org.apache.stratos.autoscaler.context.AutoscalerContext} - Network >>> partition algorithm context is removed successfully: [id] cisco-sample-vm >>> TID: [0] [STRATOS] [2015-03-19 19:45:43,092] INFO >>> {org.apache.stratos.autoscaler.monitor.cluster.ClusterMonitor} - Publishing >>> Cluster terminating event for [application] cisco-sample-vm [cluster] >>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain [instance] >>> cisco-sample-vm-1 >>> TID: [0] [STRATOS] [2015-03-19 19:45:43,109] INFO >>> {org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder} - >>> Cluster Terminating adding status started >>> forcisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain >>> TID: [0] [STRATOS] [2015-03-19 19:45:43,117] INFO >>> {org.apache.stratos.autoscaler.applications.topic.ApplicationsEventPublisher} >>> - Publishing application inactivated event: [application] cisco-sample-vm >>> [instance] cisco-sample-vm-1 >>> TID: [0] [STRATOS] [2015-03-19 19:45:43,120] INFO >>> {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher} >>> - Publishing Cluster terminating event: [application-id] cisco-sample-vm >>> [cluster id] cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain >>> [instance-id] cisco-sample-vm-1 >>> TID: [0] [STRATOS] [2015-03-19 19:45:43,144] INFO >>> {org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl} - >>> Application undeployed successfully: [application-id] cisco-sample-vm >>> TID: [0] [STRATOS] [2015-03-19 19:45:43,154] INFO >>> {org.apache.stratos.autoscaler.event.receiver.topology.AutoscalerTopologyEventReceiver} >>> - [ClusterTerminatingEvent] Received: class >>> org.apache.stratos.messaging.event.topology.ClusterInstanceTerminatingEvent >>> TID: [0] [STRATOS] [2015-03-19 19:45:43,155] INFO >>> {org.apache.stratos.autoscaler.event.publisher.InstanceNotificationPublisher} >>> - Publishing Instance Cleanup Event: [cluster] >>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain >>> TID: [0] [STRATOS] [2015-03-19 19:45:43,171] WARN >>> {org.apache.stratos.autoscaler.status.processor.cluster.ClusterStatusActiveProcessor} >>> - No possible state change found for [type] [cluster] >>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain [instance] >>> TID: [0] [STRATOS] [2015-03-19 19:45:43,192] INFO >>> {org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder} - >>> member maintenance mode event adding status started >>> TID: [0] [STRATOS] [2015-03-19 19:45:43,204] INFO >>> {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher} >>> - Publishing member in maintenance mode event: [service-name] >>> cisco-sample-vm [cluster-id] >>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain >>> [cluster-instance-id] cisco-sample-vm-1 [member-id] >>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d >>> [network-partition-id] RegionOne [partition-id] whole-region >>> TID: [0] [STRATOS] [2015-03-19 19:45:43,230] INFO >>> {org.apache.stratos.messaging.message.processor.topology.MemberMaintenanceModeProcessor} >>> - Member updated as In_Maintenance: [service] cisco-sample-vm [cluster] >>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain [member] >>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d >>> TID: [0] [STRATOS] [2015-03-19 19:45:43,231] INFO >>> {org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder} - >>> Member Ready to shut down event adding status started >>> TID: [0] [STRATOS] [2015-03-19 19:45:43,242] INFO >>> {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher} >>> - Publishing member ready to shut down event: [service-name] >>> cisco-sample-vm [cluster-id] >>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain >>> [cluster-instance-id] cisco-sample-vm-1 [member-id] >>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d >>> [network-partition-id] RegionOne [partition-id] whole-region >>> TID: [0] [STRATOS] [2015-03-19 19:45:43,262] INFO >>> {org.apache.stratos.messaging.message.processor.topology.MemberReadyToShutdownMessageProcessor} >>> - Member updated as Ready to shutdown: [service] cisco-sample-vm [cluster] >>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain [member] >>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d >>> TID: [0] [STRATOS] [2015-03-19 19:45:43,264] INFO >>> {org.apache.stratos.autoscaler.event.receiver.topology.AutoscalerTopologyEventReceiver} >>> - [MemberReadyToShutdownEvent] Received: class >>> org.apache.stratos.messaging.event.topology.MemberReadyToShutdownEvent >>> TID: [0] [STRATOS] [2015-03-19 19:45:43,264] INFO >>> {org.apache.stratos.autoscaler.context.partition.ClusterLevelPartitionContext} >>> - Starting the moving of termination pending to obsolete for [member] >>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d >>> TID: [0] [STRATOS] [2015-03-19 19:45:43,264] INFO >>> {org.apache.stratos.autoscaler.context.partition.ClusterLevelPartitionContext} >>> - Found termination pending member and trying to move [member] >>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d >>> to obsolete list >>> TID: [0] [STRATOS] [2015-03-19 19:45:43,266] INFO >>> {org.apache.stratos.common.client.CloudControllerServiceClient} - >>> Terminating instance via cloud controller: [member] >>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d >>> TID: [0] [STRATOS] [2015-03-19 19:45:43,275] INFO >>> {org.apache.stratos.cloud.controller.iaases.JcloudsIaas} - Starting to >>> terminate member: [cartridge-type] cisco-sample-vm [member-id] >>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d >>> TID: [0] [STRATOS] [2015-03-19 19:45:43,883] INFO >>> {org.apache.stratos.messaging.domain.application.locking.ApplicationLockHierarchy} >>> - Removed lock for application: [application-id] cisco-sample-vm >>> TID: [0] [STRATOS] [2015-03-19 19:45:43,909] INFO >>> {org.apache.stratos.cloud.controller.messaging.receiver.application.ApplicationEventReceiver} >>> - Application deleted event received: [application-id] cisco-sample-vm >>> TID: [0] [STRATOS] [2015-03-19 19:45:43,909] INFO >>> {org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder} - >>> Removed application [ cisco-sample-vm ]'s Cluster [ >>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain ] from the topology >>> TID: [0] [STRATOS] [2015-03-19 19:45:43,910] INFO >>> {org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl} - >>> Application deleted successfully: [application-id] >>> TID: [0] [STRATOS] [2015-03-19 19:45:43,930] INFO >>> {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher} >>> - Publishing application clusters removed event: [application-id] >>> cisco-sample-vm >>> TID: [0] [STRATOS] [2015-03-19 19:45:43,950] INFO >>> {org.apache.stratos.messaging.message.processor.topology.ApplicationClustersRemovedMessageProcessor} >>> - Cluster cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain removed >>> from topology for application cisco-sample-vm >>> TID: [0] [STRATOS] [2015-03-19 19:45:43,952] INFO >>> {org.apache.stratos.messaging.message.processor.application.ApplicationDeletedMessageProcessor} >>> - [Application] cisco-sample-vm has been successfully removed >>> TID: [0] [STRATOS] [2015-03-19 19:45:44,786] INFO >>> {org.apache.stratos.cloud.controller.iaases.JcloudsIaas} - Member >>> terminated: [member-id] >>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d >>> TID: [0] [STRATOS] [2015-03-19 19:45:44,786] WARN >>> {org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder} - >>> Cluster cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain does not >>> exist >>> TID: [0] [STRATOS] [2015-03-19 19:45:50,012] INFO >>> {org.apache.stratos.common.client.CloudControllerServiceClient} - >>> Terminating instance via cloud controller: [member] >>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d >>> TID: [0] [STRATOS] [2015-03-19 19:45:50,017] ERROR >>> {org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl} >>> - Could not terminate instance, member context not found: [member-id] >>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d >>> TID: [0] [STRATOS] [2015-03-19 19:45:50,018] ERROR >>> {org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl} >>> - Could not terminate instance: [member-id] >>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d >>> org.apache.stratos.cloud.controller.exception.InvalidMemberException: >>> Could not terminate instance, member context not found: [member-id] >>> cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d >>> at >>> org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl.terminateInstance(CloudControllerServiceImpl.java:553) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:606) >>> at >>> org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:212) >>> at >>> org.apache.axis2.rpc.receivers.RPCInOnlyMessageReceiver.invokeBusinessLogic(RPCInOnlyMessageReceiver.java:66) >>> at >>> org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:110) >>> at >>> org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180) >>> at >>> org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:172) >>> at >>> org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:146) >>> at >>> org.wso2.carbon.core.transports.CarbonServlet.doPost(CarbonServlet.java:231) >>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:755) >>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) >>> at >>> org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61) >>> at >>> org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128) >>> at >>> org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68) >>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) >>> at >>> org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68) >>> at >>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) >>> at >>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) >>> at >>> org.wso2.carbon.tomcat.ext.filter.CharacterSetFilter.doFilter(CharacterSetFilter.java:61) >>> at >>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) >>> at >>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) >>> at >>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222) >>> at >>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) >>> at >>> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472) >>> at >>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) >>> at >>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99) >>> at >>> org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:178) >>> at >>> org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47) >>> at >>> org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:56) >>> at >>> org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47) >>> at >>> org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:141) >>> at >>> org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:156) >>> at >>> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936) >>> at >>> org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:52) >>> at >>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) >>> >>> >> >> -- >> Imesh Gunaratne >> >> Technical Lead, WSO2 >> Committer & PMC Member, Apache Stratos >> >> > > -- > Sent from Gmail Mobile > -- Imesh Gunaratne Technical Lead, WSO2 Committer & PMC Member, Apache Stratos