On 3/30/15, 9:18 AM, Lakmal Warusawithana wrote:
IMO forceful un deploy is enough no need to have forceful delete. If un 
deployment success delete should work without any issues

I agree, forceful undeploy should be enough. I assume this means that if I call undeploy (force=false) the first time and it doesn't work, I can call undeploy(force=true).

In order for these api's to be robust, the create/deploy APIs should perform strict error checking and clean up of partial state when an exceptions are thrown. The code seems to do sufficient error checking but we keep encountering partial state left over when an exception is thrown.

The undeploy force and/or delete APIs should ideally always cleanup even when detecting inconsistencies. We seem to be voting to handle this as a part of undeploy (force=true). I hope this means that an app MUST be cleanly transitioned back to "Created" state when we use (undeploy=force) and gracefully handle a subsequent deploy command.

-Vanson




On Monday, March 30, 2015, Imesh Gunaratne <im...@apache.org 
<mailto:im...@apache.org>> wrote:

    Thanks Vanson for pointing this scenario! Yes we would need to forcefully 
delete an application in certain scenarios.

    Udara: +1 For the suggestion!

    On Monday, March 30, 2015, Udara Liyanage <ud...@wso2.com> wrote:

        Hi,

        Sorry for the typo. We can add the flag as a query parameter.

         curl -X POST applications/single-cartridge-app/undeploy?*force=true*

        On Mon, Mar 30, 2015 at 10:04 AM, Udara Liyanage <ud...@wso2.com> wrote:

            Hi Imesh,

            How about the blow addition to the API?

             curl -X POST 
applications/single-cartridge-app/undeploy/*force=true*

            On Mon, Mar 30, 2015 at 9:48 AM, Udara Liyanage <ud...@wso2.com> 
wrote:



                On Sun, Mar 29, 2015 at 10:34 PM, Vanson Lim <v...@cisco.com> 
wrote:

                    On 3/27/15, 5:57 PM, Vanson Lim wrote:
                    On 3/27/15, 5:52 PM, Imesh Gunaratne wrote:
                    Hi Vanson,

                    I added a validation in Autoscaler to avoid applications 
being deleted while they are being undeployed in the
                    background.

                    Now the following error would be raised if this condition is 
met: "Application cannot be deleted, undeployment
                    process is still in progress: [application-id] 
<application-id>".

                    In addition I have implemented your suggestion to set the 
application status to CREATED once the application is
                    completely undeployed. Now we can check from the API 
whether the application is undeployed properly. Please take a
                    pull and verify.




                    Took me a while to experiment with the failure cases.

                    Some observations:

                    1) The change to check that the Application state changes 
to CREATED works. I am able to check that application
                    status returns to Created before deleting.

                    2) Application delete throws tracebacks to the 
wso2carbon.log. IMO, this should only be informational rather then
                    throw exceptions.

                I guess the reason for throwing an RuntimeException is to 
notify SM that  an problem has occured while deleting the
                application. I will investigate firther.


                    ie

                    
org.apache.stratos.autoscaler.exception.AutoScalerException: Application is in 
deployed state, please undeploy it
                    before deleting: [application-id] cisco-sample-vm
                        at
                    
org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl.deleteApplication(AutoscalerServiceImpl.java:411)
                        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
                        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
                        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                        at java.lang.reflect.Method.invoke(Method.java:606)
                        at 
org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:212)
                        at 
org.apache.axis2.rpc.receivers.RPCInOnlyMessageReceiver.invokeBusinessLogic(RPCInOnlyMessageReceiver.java:66)
                        at 
org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:110)
                        at 
org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
                        at 
org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:172)
                        at 
org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:146)
                        at 
org.wso2.carbon.core.transports.CarbonServlet.doPost(CarbonServlet.java:231)
                        at 
javax.servlet.http.HttpServlet.service(HttpServlet.java:755)
                        at 
javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
                        at 
org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
                        at 
org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128)
                        at 
org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68)
                        at 
javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
                        at 
org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68)
                        at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
                        at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
                        at 
org.wso2.carbon.tomcat.ext.filter.CharacterSetFilter.doFilter(CharacterSetFilter.java:61)
                        at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
                        at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
                        at 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
                        at 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
                        at 
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
                        at 
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
                        at 
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
                        at 
org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:178)
                        at 
org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
                        at 
org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:56)
                        at 
org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
                        at 
org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:141)
                        at 
org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:156)
                        at 
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
                        at 
org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:52)
                        at 
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
                        at 
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
                        at 
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
                        at 
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
                        at 
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1653)
                        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
                        at java.lang.Thread.run(Thread.java:745)
                    TID: [0] [STRATOS] [2015-03-28 00:17:52,952] ERROR 
{org.apache.axis2.rpc.receivers.RPCInOnlyMessageReceiver} -  Could
                    not delete application: [application-id] cisco-sample-vm
                    java.lang.RuntimeException: Could not delete application: 
[application-id] cisco-sample-vm
                        at
                    
org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl.deleteApplication(AutoscalerServiceImpl.java:426)
                        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
                        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
                        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                        at java.lang.reflect.Method.invoke(Method.java:606)
                        at 
org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:212)
                        at 
org.apache.axis2.rpc.receivers.RPCInOnlyMessageReceiver.invokeBusinessLogic(RPCInOnlyMessageReceiver.java:66)
                        at 
org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:110)
                        at 
org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
                        at 
org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:172)
                        at 
org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:146)
                        at 
org.wso2.carbon.core.transports.CarbonServlet.doPost(CarbonServlet.java:231)
                        at 
javax.servlet.http.HttpServlet.service(HttpServlet.java:755)
                        at 
javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
                        at 
org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
                        at 
org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128)
                        at 
org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68)
                        at 
javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
                        at 
org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68)
                        at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
                        at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
                        at 
org.wso2.carbon.tomcat.ext.filter.CharacterSetFilter.doFilter(CharacterSetFilter.java:61)
                        at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
                        at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
                        at 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
                        at 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
                        at 
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
                        at 
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
                        at 
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
                        at 
org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:178)
                        at 
org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
                        at 
org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:56)
                        at 
org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
                        at 
org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:141)
                        at 
org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:156)
                        at 
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
                        at 
org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:52)
                        at 
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
                        at 
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
                        at 
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
                        at 
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
                        at 
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1653)
                        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
                        at java.lang.Thread.run(Thread.java:745)
                    Caused by: 
org.apache.stratos.autoscaler.exception.AutoScalerException: Application is in 
deployed state, please
                    undeploy it before deleting: [application-id] 
cisco-sample-vm
                        at
                    
org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl.deleteApplication(AutoscalerServiceImpl.java:411)
                        ... 44 more


                    3) I am hitting the "redeploy application issue" that was 
fixed yesterday as a part of testing this fix and found
                    that the application is in a  broken state after an 
application redeploy such that application delete doesn't work
                    anymore because the partially deployed application is in stuck in the 
"Deployed" state.

                    This is similar to a valid use case which application 
delete doesn't currently handle.   Our current implementation
                    of application delete only works gracefully if the 
application is well behaved.  If we have a single VM that is
                    unresponsive to a terminate event,  won't the application permanently 
be stuck in a "Deployed" state?   This leads me
                    to think that application delete needs to able to clean up 
all the state in the various components that make up
stratos instead of preventing the delete requests regardless of the cluster instance counts or application status. In other words, the request should always be honored, otherwise we have no way of forcing the undeploy and deletion
                    of an application.   If the user wants to be graceful, they 
have the option to poll the application state changes
                    using the application GET api before calling the 
application delete API.


                Your point seems to be  valid since with the above fix, 
application will not be able to be deleted if it doesn't become
                created for some reason. One solution is to let the user 
explicitly specify to delete forcefully which will delete the
                application  regardless of the state. I will do a further 
analysis and get back to you.


                    -Vanson

                    okay, will give it a try.

                    -Vanson

                    Thanks

                    On Sat, Mar 28, 2015 at 2:40 AM, Vanson Lim 
<v...@cisco.com> wrote:

                        On 3/27/15, 2:26 PM, Imesh Gunaratne wrote:
                        Hi Vanson,

                        I analyzed this issue. Logically we cannot delete an 
application soon after un-deploying it. The reason is
                        that the un-deployment process executes gracefully on 
each member in all the clusters in an application.
                        Depending on the complexity of the application, it may 
take some time to completely terminate all the members,
                        remove the clusters and then un-deploy the application.

                        IMO what we are missing here is a validation in the 
delete application API method, to tell the API client that
                        an application cannot be deleted until it is completely 
un-deployed.

                        Imesh,

                        Thanks for looking at this.

                        It sounds reasonable to hold off the removing to 
application and also have the remove rest api call return that
                        it's not ready.   For applications consisting of a lot 
of groups and cartridges the time to clean up could be
                        variable.   Could we use the Rest api to get the 
application status, ie check that status=Created before deleting.

                        ie this api:

                        
https://cwiki.apache.org/confluence/display/STRATOS/4.1.0+Getting+Details+of+an+Application+via+REST+API

                        It looks like an application has two states:   "Created" and 
"Deployed".   If we had a Undeploying state, then
                        we could check and hold off
                        requesting the remove-application until it returns to 
the Created state.

                        -Vanson


                        Thanks
                        Imesh

                        On Wed, Mar 25, 2015 at 12:10 PM, Udara Liyanage 
<ud...@wso2.com> wrote:

                            Hi,

                            This arises when the application is deleted 
immediately after the application underemployment. I guess CC
                            does not have the member context when the 
application is removed. I will have a look.

                            A simple workaround it to wait sometime to remove 
application after undeployment.

                            On Tue, Mar 24, 2015 at 9:06 AM, Imesh Gunaratne 
<im...@apache.org> wrote:

                                Thanks Martin! We will have a look at this.

                                On Tue, Mar 24, 2015 at 1:21 AM, Martin Eppel 
(meppel) <mep...@cisco.com> wrote:

                                    I opened a jira 
https://issues.apache.org/jira/browse/STRATOS-1281 to track the issue,

                                    Thanks

                                    Martin

                                    *From:*Imesh Gunaratne 
[mailto:im...@apache.org]
                                    *Sent:* Friday, March 20, 2015 3:27 AM
                                    *To:* dev
                                    *Subject:* Re: Stratos 4.1.0 - tracebacks 
seen when issuing application undeploy/remove

                                    Hi Vanson,

                                    Thanks for reporting this problem. 
According to the logs stratos is trying to remove the same
                                    member twice in this flow:

                                    TID: [0] [STRATOS] [2015-03-19 
19:45:43,952]  INFO
                                    
{org.apache.stratos.messaging.message.processor.application.ApplicationDeletedMessageProcessor}
 -
                                     [Application] cisco-sample-vm has been 
successfully removed

                                    TID: [0] [STRATOS] [2015-03-19 
19:45:44,786]  INFO
                                    
{org.apache.stratos.cloud.controller.iaases.JcloudsIaas} -  Member terminated: 
[member-id]
                                    
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d

                                    TID: [0] [STRATOS] [2015-03-19 
19:45:50,012]  INFO
                                    
{org.apache.stratos.common.client.CloudControllerServiceClient} - Terminating 
instance via cloud
                                    controller: [member]
                                    
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d

                                    TID: [0] [STRATOS] [2015-03-19 
19:45:50,017] ERROR
                                    
{org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl} 
-  Could not
                                    terminate instance, member context not 
found: [member-id]
                                    
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d

                                    We might need to investigate and see what's 
causing this.

                                    Thanks

                                    On Fri, Mar 20, 2015 at 6:55 AM, Vanson Lim 
<v...@cisco.com> wrote:

                                    Hi,

                                    We are testing the behavior of stratos 
4.1.0 rest api's  and found that if we issue an
                                    undeploy-application followed immediately 
with application-remove, that we see the following
                                    traceback.

                                    The VM successfully get's deleted but not 
sure what kind of side effects this has on the system.

                                    Here's the snippet from the wso2carbon.log 
file.  I've also attached the entire log.

                                    Steps to reproduce this:

                                    1) deploy an application which leads to 
startup a single instance of a cartridge.
                                    2) wait for it to become active
                                    3) issue the undeploy application and 
application remove rest api calls.


                                    -Vanson



TID: [0] [STRATOS] [2015-03-19 19:45:08,681] INFO {org.wso2.carbon.databridge.core.DataBridge} - admin connected
                                    TID: [0] [STRATOS] [2015-03-19 
19:45:43,023] INFO
                                    
{org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl} -  Starting 
to undeploy
                                    application: [application-id] 
cisco-sample-vm
                                    TID: [0] [STRATOS] [2015-03-19 
19:45:43,024] INFO
                                    
{org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl} -  Removing 
application
                                    signup: [application-id] cisco-sample-vm
                                    TID: [0] [STRATOS] [2015-03-19 
19:45:43,051] INFO
                                    
{org.apache.stratos.manager.components.ApplicationSignUpHandler} - Removing 
application signup:
                                    [application-id] cisco-sample-vm 
[tenant-id] -1234
                                    TID: [0] [STRATOS] [2015-03-19 
19:45:43,084] INFO
                                    
{org.apache.stratos.manager.components.ApplicationSignUpHandler} - Application 
signup removed
                                    successfully: [application-id] 
cisco-sample-vm [tenant-id] -1234
                                    TID: [0] [STRATOS] [2015-03-19 
19:45:43,091] INFO
                                    
{org.apache.stratos.autoscaler.context.AutoscalerContext} -  Network partition 
algorithm context
                                    is removed successfully: [id] 
cisco-sample-vm
                                    TID: [0] [STRATOS] [2015-03-19 
19:45:43,092] INFO
                                    
{org.apache.stratos.autoscaler.monitor.cluster.ClusterMonitor} - Publishing 
Cluster terminating
                                    event for [application] cisco-sample-vm 
[cluster]
                                    
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain [instance] 
cisco-sample-vm-1
                                    TID: [0] [STRATOS] [2015-03-19 
19:45:43,109] INFO
                                    
{org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder} -  
Cluster Terminating
                                    adding status started 
forcisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain
                                    TID: [0] [STRATOS] [2015-03-19 
19:45:43,117] INFO
                                    
{org.apache.stratos.autoscaler.applications.topic.ApplicationsEventPublisher} - 
 Publishing
                                    application inactivated event: 
[application] cisco-sample-vm [instance] cisco-sample-vm-1
                                    TID: [0] [STRATOS] [2015-03-19 
19:45:43,120] INFO
                                    
{org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher}
 -  Publishing
                                    Cluster terminating event: [application-id] 
cisco-sample-vm [cluster id]
                                    
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain [instance-id] 
cisco-sample-vm-1
                                    TID: [0] [STRATOS] [2015-03-19 
19:45:43,144] INFO
                                    
{org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl} -  
Application undeployed
                                    successfully: [application-id] 
cisco-sample-vm
                                    TID: [0] [STRATOS] [2015-03-19 
19:45:43,154] INFO
                                    
{org.apache.stratos.autoscaler.event.receiver.topology.AutoscalerTopologyEventReceiver}
 -
                                    [ClusterTerminatingEvent] Received: class
                                    
org.apache.stratos.messaging.event.topology.ClusterInstanceTerminatingEvent
                                    TID: [0] [STRATOS] [2015-03-19 
19:45:43,155] INFO
                                    
{org.apache.stratos.autoscaler.event.publisher.InstanceNotificationPublisher} - 
 Publishing
                                    Instance Cleanup Event: [cluster] 
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain
                                    TID: [0] [STRATOS] [2015-03-19 
19:45:43,171] WARN
                                    
{org.apache.stratos.autoscaler.status.processor.cluster.ClusterStatusActiveProcessor}
 -  No
                                    possible state change found for [type] 
[cluster]
                                    
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain [instance]
                                    TID: [0] [STRATOS] [2015-03-19 
19:45:43,192] INFO
                                    
{org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder} -  
member maintenance
                                    mode event adding status started
                                    TID: [0] [STRATOS] [2015-03-19 
19:45:43,204] INFO
                                    
{org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher}
 -  Publishing
                                    member in maintenance mode event: 
[service-name] cisco-sample-vm [cluster-id]
                                    
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain [cluster-instance-id] 
cisco-sample-vm-1
                                    [member-id]
                                    
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
                                    [network-partition-id] RegionOne 
[partition-id] whole-region
                                    TID: [0] [STRATOS] [2015-03-19 
19:45:43,230] INFO
                                    
{org.apache.stratos.messaging.message.processor.topology.MemberMaintenanceModeProcessor}
 -  Member
                                    updated as In_Maintenance: [service] 
cisco-sample-vm [cluster]
                                    
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain [member]
                                    
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
                                    TID: [0] [STRATOS] [2015-03-19 
19:45:43,231] INFO
                                    
{org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder} -  
Member Ready to shut
                                    down event adding status started
                                    TID: [0] [STRATOS] [2015-03-19 
19:45:43,242] INFO
                                    
{org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher}
 -  Publishing
                                    member ready to shut down event: 
[service-name] cisco-sample-vm [cluster-id]
                                    
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain [cluster-instance-id] 
cisco-sample-vm-1
                                    [member-id]
                                    
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
                                    [network-partition-id] RegionOne 
[partition-id] whole-region
                                    TID: [0] [STRATOS] [2015-03-19 
19:45:43,262] INFO
{org.apache.stratos.messaging.message.processor.topology.MemberReadyToShutdownMessageProcessor} - Member updated as Ready to shutdown: [service] cisco-sample-vm [cluster]
                                    
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain [member]
                                    
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
                                    TID: [0] [STRATOS] [2015-03-19 
19:45:43,264] INFO
                                    
{org.apache.stratos.autoscaler.event.receiver.topology.AutoscalerTopologyEventReceiver}
 -
                                    [MemberReadyToShutdownEvent] Received: class
                                    
org.apache.stratos.messaging.event.topology.MemberReadyToShutdownEvent
                                    TID: [0] [STRATOS] [2015-03-19 
19:45:43,264] INFO
                                    
{org.apache.stratos.autoscaler.context.partition.ClusterLevelPartitionContext} 
-  Starting the
                                    moving of termination pending to obsolete 
for [member]
                                    
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
                                    TID: [0] [STRATOS] [2015-03-19 
19:45:43,264] INFO
                                    
{org.apache.stratos.autoscaler.context.partition.ClusterLevelPartitionContext} 
-  Found
                                    termination pending member and trying to 
move [member]
                                    
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
 to
                                    obsolete list
                                    TID: [0] [STRATOS] [2015-03-19 
19:45:43,266] INFO
                                    
{org.apache.stratos.common.client.CloudControllerServiceClient} - Terminating 
instance via cloud
                                    controller: [member]
                                    
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
                                    TID: [0] [STRATOS] [2015-03-19 
19:45:43,275] INFO
                                    
{org.apache.stratos.cloud.controller.iaases.JcloudsIaas} -  Starting to 
terminate member:
                                    [cartridge-type] cisco-sample-vm [member-id]
                                    
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
                                    TID: [0] [STRATOS] [2015-03-19 
19:45:43,883] INFO
                                    
{org.apache.stratos.messaging.domain.application.locking.ApplicationLockHierarchy}
 -  Removed lock
                                    for application: [application-id] 
cisco-sample-vm
                                    TID: [0] [STRATOS] [2015-03-19 
19:45:43,909] INFO
{org.apache.stratos.cloud.controller.messaging.receiver.application.ApplicationEventReceiver} - Application deleted event received: [application-id] cisco-sample-vm
                                    TID: [0] [STRATOS] [2015-03-19 
19:45:43,909] INFO
                                    
{org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder} -  
Removed application [
                                    cisco-sample-vm ]'s Cluster [ 
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain ] from the
                                    topology
                                    TID: [0] [STRATOS] [2015-03-19 
19:45:43,910] INFO
                                    
{org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl} -  
Application deleted
                                    successfully: [application-id]
                                    TID: [0] [STRATOS] [2015-03-19 
19:45:43,930] INFO
                                    
{org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher}
 -  Publishing
                                    application clusters removed event: 
[application-id] cisco-sample-vm
                                    TID: [0] [STRATOS] [2015-03-19 
19:45:43,950] INFO
                                    
{org.apache.stratos.messaging.message.processor.topology.ApplicationClustersRemovedMessageProcessor}
                                    -  Cluster 
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain removed from topology for
                                    application cisco-sample-vm
                                    TID: [0] [STRATOS] [2015-03-19 
19:45:43,952] INFO
                                    
{org.apache.stratos.messaging.message.processor.application.ApplicationDeletedMessageProcessor}
 -
                                    [Application] cisco-sample-vm has been 
successfully removed
                                    TID: [0] [STRATOS] [2015-03-19 
19:45:44,786] INFO
                                    
{org.apache.stratos.cloud.controller.iaases.JcloudsIaas} -  Member terminated: 
[member-id]
                                    
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
                                    TID: [0] [STRATOS] [2015-03-19 
19:45:44,786] WARN
                                    
{org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder} -  
Cluster
                                    
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain does not exist
                                    TID: [0] [STRATOS] [2015-03-19 
19:45:50,012] INFO
                                    
{org.apache.stratos.common.client.CloudControllerServiceClient} - Terminating 
instance via cloud
                                    controller: [member]
                                    
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
                                    TID: [0] [STRATOS] [2015-03-19 
19:45:50,017] ERROR
                                    
{org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl} 
-  Could not
                                    terminate instance, member context not 
found: [member-id]
                                    
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
                                    TID: [0] [STRATOS] [2015-03-19 
19:45:50,018] ERROR
                                    
{org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl} 
-  Could not
                                    terminate instance: [member-id]
                                    
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
                                    
org.apache.stratos.cloud.controller.exception.InvalidMemberException: Could not 
terminate
                                    instance, member context not found: 
[member-id]
                                    
cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
                                            at
                                    
org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl.terminateInstance(CloudControllerServiceImpl.java:553)
                                            at 
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                                            at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
                                            at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                                            at 
java.lang.reflect.Method.invoke(Method.java:606)
                                            at 
org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:212)
                                            at
                                    
org.apache.axis2.rpc.receivers.RPCInOnlyMessageReceiver.invokeBusinessLogic(RPCInOnlyMessageReceiver.java:66)
                                            at
                                    
org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:110)
                                            at 
org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
                                            at
                                    
org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:172)
                                            at 
org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:146)
                                            at 
org.wso2.carbon.core.transports.CarbonServlet.doPost(CarbonServlet.java:231)
                                            at 
javax.servlet.http.HttpServlet.service(HttpServlet.java:755)
                                            at 
javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
                                            at
                                    
org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
                                            at 
org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128)
                                            at 
org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68)
                                            at 
javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
                                            at 
org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68)
                                            at
                                    
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
                                            at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
                                            at 
org.wso2.carbon.tomcat.ext.filter.CharacterSetFilter.doFilter(CharacterSetFilter.java:61)
                                            at
                                    
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
                                            at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
                                            at 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
                                            at 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
                                            at 
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
                                            at 
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
                                            at 
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
                                            at
                                    
org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:178)
                                            at 
org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
                                            at 
org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:56)
                                            at
                                    
org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
                                            at 
org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:141)
                                            at
                                    
org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:156)
                                            at 
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
                                            at
                                    
org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:52)
                                            at 
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)



-- Imesh Gunaratne

    Technical Lead, WSO2
    Committer & PMC Member, Apache Stratos



--
Sent from Gmail Mobile

Reply via email to