Hi Reka, Just a heads up, the issue turned out to be an issue with the clean up script. Thanks for the pointers,
Regards Martin From: Reka Thirunavukkarasu [mailto:[email protected]] Sent: Wednesday, July 01, 2015 2:08 AM To: Martin Eppel (meppel) Cc: [email protected]; Ryan Du Plessis (rdupless) Subject: Re: application removal issue Hi Martin, As i have gone through your log, please see below with my findings. There seems to be not anything to do with undeployment process for this particular scenario as it is not a bug in undeployment. We can fix it by changing a configuration. Also, we will need to investigate at cartridge agent side as well. 1. After you invoke the undeploy, member sent the in-maintenance event. 2. Autoscaler received it and mark the member as terminatingPending 3. Stratos is waiting for ReadToShutdown event from the member. But member didn't send this event due to network issue or some cleanup process issue 4. Autoscaler has a pendingTerminationMemberExpiryTimeout in autoscaler.xml(SRTATOS-HOME/repository/conf). I believe that this timeout is 30min(according to the log below) in your setup. After this timeout, autoscaler will move this member to obsolete list. So that the member will get terminated immediately. Please see the log as i found: TID: [0] [STRATOS] [2015-06-30 16:22:56,505] INFO {org.apache.stratos.autoscaler.context.partition.ClusterLevelPartitionContext$TerminationPendingMemberWatcher} - Termination pending state of member is expired, member will be moved to obsolete list [termination pending member] di-000-002.di-000-002.cisco-qvpc-cf-02-0.domain2b83bd51-a85a-473b-b143-adc2c8feac94 [expiry time] 1800000 [cluster] di-000-002.di-000-002.cisco-qvpc-cf-02-0.domain [cluster instance] di-000-002-1 5. Member got terminated successfully. Hence cluster and then application also got terminated successfully. As the solution, can you try to reduce pendingTerminationMemberExpiryTimeout according to your suitable value? Also, if you detect this issue again, it is better to get the cartridge agent logs as well. So that we can try to find out why cartridge agent failed to send the ReadyToShutdown event. Thanks, Reka On Wed, Jul 1, 2015 at 2:27 AM, Martin Eppel (meppel) <[email protected]<mailto:[email protected]>> wrote: Hi Reka, Here is the issue with application removal in one of our use cases: In the scenario there are 5 applications deployed, cartridge-proxy, di-000-001, di-000-002, di-000-003, di-000-004 The scenario tries to remove application , di-000-001, di-000-002, di-000-003, di-000-004. Removing , di-000-001 succeeds but, at least at first sight, removing di-000-002 ends up with the application di-000-002 stuck in “undeploying” mode and the cartridge in maintenance mode. However, as it turned out, di-000-002 is being removed successfully but it took 32 minutes. I think the question is why does it take that long, is it expected and what are the conditions for it to take that long (in comparison, removing di-000-002 only took a few seconds). Also, the application were removed using the force flag. Shouldn’t force remove go quick in any case ? Removing di-000-001: “ TID: [0] [STRATOS] [2015-06-30 15:52:09,073] INFO {org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl} - Application undeployment process started: [application-id] di-000-001 … TID: [0] [STRATOS] [2015-06-30 15:52:12,982] INFO {org.apache.stratos.autoscaler.applications.topic.ApplicationBuilder} - Application un-deployed successfully: [application-id] di-000-001 “ Removing di-000-002 “ … TID: [0] [STRATOS] [2015-06-30 15:52:50,383] INFO {org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl} - Application undeployment process started: [application-id] di-000-002 … TID: [0] [STRATOS] [2015-06-30 16:24:04,083] INFO {org.apache.stratos.autoscaler.applications.topic.ApplicationBuilder} - Application un-deployed successfully: [application-id] di-000-002 … “ Application jsons are listed below, wso2carbon.log is attached. Thanks Martin stratos> list-applications Applications found: +-----------------+-----------------+-------------+ | Application ID | Alias | Status | +-----------------+-----------------+-------------+ | cartridge-proxy | cartridge-proxy | Deployed | +-----------------+-----------------+-------------+ | di-000-002 | di-000-002 | Undeploying | +-----------------+-----------------+-------------+ | di-000-003 | di-000-003 | Deployed | +-----------------+-----------------+-------------+ | di-000-004 | di-000-004 | Deployed | Waiting for a long time: +-----------------+-----------------+----------+ | Application ID | Alias | Status | +-----------------+-----------------+----------+ | cartridge-proxy | cartridge-proxy | Deployed | +-----------------+-----------------+----------+ | di-000-002 | di-000-002 | Created | +-----------------+-----------------+----------+ | di-000-003 | di-000-003 | Deployed | +-----------------+-----------------+----------+ | di-000-004 | di-000-004 | Deployed | +-----------------+-----------------+----------+ See below json files for the applications: stratos> describe-application di-000-001 Application not found: di-000-001 stratos> describe-application di-000-002 Application: di-000-002 { "applicationId": "di-000-002", "multiTenant": false, "alias": "di-000-002", "status": "Undeploying", "components": { "cartridges": [ { "type": "cisco-qvpc-cf-02-0", "cartridgeMin": 1, "cartridgeMax": 1, "subscribableInfo": { "alias": "di-000-002", "deploymentPolicy": "static-1", "autoscalingPolicy": "economyPolicy", "maxMembers": 0, "minMembers": 0, "artifactRepository": { "alias": "di-000-002", "privateRepo": true, "repoUrl": "http://octl.qmog.cisco.com:10080/git/default.git";, "repoUsername": "user", "repoPassword": "password" }, "property": [ { "name": "payload_parameter.VOLUME_INFO", "value": "di-000-002:ca6bbb1b-47a5-44c5-af41-6264eb2bdcfd" } ], "persistence": { "isRequired": false } } } ] } } //// Application: di-000-003 { "applicationId": "di-000-003", "multiTenant": false, "alias": "di-000-003", "status": "Deployed", "components": { "cartridges": [ { "type": "cisco-qvpc-sf-0", "cartridgeMin": 1, "cartridgeMax": 1, "subscribableInfo": { "alias": "di-000-003", "deploymentPolicy": "static-1", "autoscalingPolicy": "economyPolicy", "maxMembers": 0, "minMembers": 0, "artifactRepository": { "alias": "di-000-003", "privateRepo": true, "repoUrl": "http://octl.qmog.cisco.com:10080/git/default.git";, "repoUsername": "user", "repoPassword": "password" } } } ] } } ///// stratos> describe-application di-000-004 Application: di-000-004 { "applicationId": "di-000-004", "multiTenant": false, "alias": "di-000-004", "status": "Deployed", "components": { "cartridges": [ { "type": "cisco-qvpc-sf-0", "cartridgeMin": 1, "cartridgeMax": 1, "subscribableInfo": { "alias": "di-000-004", "deploymentPolicy": "static-1", "autoscalingPolicy": "economyPolicy", "maxMembers": 0, "minMembers": 0, "artifactRepository": { "alias": "di-000-004", "privateRepo": true, "repoUrl": "http://octl.qmog.cisco.com:10080/git/default.git";, "repoUsername": "user", "repoPassword": "password" } } } ] } } //// Application: cartridge-proxy { "applicationId": "cartridge-proxy", "multiTenant": false, "alias": "cartridge-proxy", "status": "Deployed", "components": { "cartridges": [ { "type": "cartridge-proxy", "cartridgeMin": 1, "cartridgeMax": 1, "subscribableInfo": { "alias": "cartridge-proxy", "deploymentPolicy": "static-1", "autoscalingPolicy": "economyPolicy", "maxMembers": 0, "minMembers": 0, "artifactRepository": { "alias": "cartridge-proxy", "privateRepo": true, "repoUrl": "http://octl.qmog.cisco.com:10080/git/default.git";, "repoUsername": "user", "repoPassword": "password" } } } ] } } -- Reka Thirunavukkarasu Senior Software Engineer, WSO2, Inc.:http://wso2.com, Mobile: +94776442007
