Thanks for the update Martin..It's good that the issue is identified.! Thanks, Reka
On Thu, Jul 2, 2015 at 8:40 PM, Martin Eppel (meppel) <[email protected]> wrote: > Hi Reka, > > > > Just a heads up, the issue turned out to be an issue with the clean up > script. Thanks for the pointers, > > > > Regards > > > > Martin > > > > *From:* Reka Thirunavukkarasu [mailto:[email protected]] > *Sent:* Wednesday, July 01, 2015 2:08 AM > *To:* Martin Eppel (meppel) > *Cc:* [email protected]; Ryan Du Plessis (rdupless) > *Subject:* Re: application removal issue > > > > Hi Martin, > > As i have gone through your log, please see below with my findings. There > seems to be not anything to do with undeployment process for this > particular scenario as it is not a bug in undeployment. We can fix it by > changing a configuration. Also, we will need to investigate at cartridge > agent side as well. > > 1. After you invoke the undeploy, member sent the in-maintenance event. > > 2. Autoscaler received it and mark the member as terminatingPending > > 3. Stratos is waiting for ReadToShutdown event from the member. But member > didn't send this event due to network issue or some cleanup process issue > > 4. Autoscaler has a pendingTerminationMemberExpiryTimeout in > autoscaler.xml(SRTATOS-HOME/repository/conf). I believe that this timeout > is 30min(according to the log below) in your setup. After this timeout, > autoscaler will move this member to obsolete list. So that the member will > get terminated immediately. Please see the log as i found: > > TID: [0] [STRATOS] [2015-06-30 16:22:56,505] INFO > {org.apache.stratos.autoscaler.context.partition.ClusterLevelPartitionContext$TerminationPendingMemberWatcher} > - * Termination pending state of member is expired, member will be moved > to obsolete list* [termination pending member] > di-000-002.di-000-002.cisco-qvpc-cf-02-0.domain2b83bd51-a85a-473b-b143-adc2c8feac94 > [expiry time] 1800000 [cluster] > di-000-002.di-000-002.cisco-qvpc-cf-02-0.domain [cluster instance] > di-000-002-1 > > 5. Member got terminated successfully. Hence cluster and then application > also got terminated successfully. > > As the solution, can you try to reduce > pendingTerminationMemberExpiryTimeout according to your suitable value? > Also, if you detect this issue again, it is better to get the cartridge > agent logs as well. So that we can try to find out why cartridge agent > failed to send the ReadyToShutdown event. > > Thanks, > > Reka > > > > On Wed, Jul 1, 2015 at 2:27 AM, Martin Eppel (meppel) <[email protected]> > wrote: > > Hi Reka, > > > > Here is the issue with application removal in one of our use cases: > > > > In the scenario there are 5 applications deployed, cartridge-proxy, > di-000-001, di-000-002, di-000-003, di-000-004 > > > > The scenario tries to remove application , di-000-001, di-000-002, > di-000-003, di-000-004. > > > > Removing , di-000-001 succeeds but, at least at first sight, removing > di-000-002 ends up with the application di-000-002 stuck in “undeploying” > mode and the cartridge in maintenance mode. However, as it turned out, > di-000-002 is being removed successfully but it took *32 minutes*. > > I think the question is why does it take that long, is it expected and > what are the conditions for it to take that long (in comparison, removing > di-000-002 only took a few seconds). > > Also, the application were removed using the force flag. Shouldn’t force > remove go quick in any case ? > > > > Removing di-000-001: > > “ > > *TID: [0] [STRATOS] [2015-06-30 15:52:09,073] INFO > {org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl} - > Application undeployment process started: [application-id] di-000-001* > > *…* > > *TID: [0] [STRATOS] [2015-06-30 15:52:12,982] INFO > {org.apache.stratos.autoscaler.applications.topic.ApplicationBuilder} - > Application un-deployed successfully: [application-id] di-000-001* > > “ > > > > Removing di-000-002 > > “ > > … > > *TID: [0] [STRATOS] [2015-06-30 15:52:50,383] INFO > {org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl} - > Application undeployment process started: [application-id] di-000-002* > > *…* > > *TID: [0] [STRATOS] [2015-06-30 16:24:04,083] INFO > {org.apache.stratos.autoscaler.applications.topic.ApplicationBuilder} - > Application un-deployed successfully: [application-id] di-000-002* > > *…* > > “ > > > > Application jsons are listed below, wso2carbon.log is attached. > > > > Thanks > > > > Martin > > > > > > stratos> list-applications > > Applications found: > > +-----------------+-----------------+-------------+ > > | Application ID | Alias | Status | > > +-----------------+-----------------+-------------+ > > | cartridge-proxy | cartridge-proxy | Deployed | > > +-----------------+-----------------+-------------+ > > | di-000-002 | di-000-002 | Undeploying | > > +-----------------+-----------------+-------------+ > > | di-000-003 | di-000-003 | Deployed | > > +-----------------+-----------------+-------------+ > > | di-000-004 | di-000-004 | Deployed | > > > > > > > > Waiting for a long time: > > > > +-----------------+-----------------+----------+ > > | Application ID | Alias | Status | > > +-----------------+-----------------+----------+ > > | cartridge-proxy | cartridge-proxy | Deployed | > > +-----------------+-----------------+----------+ > > | di-000-002 | di-000-002 | Created | > > +-----------------+-----------------+----------+ > > | di-000-003 | di-000-003 | Deployed | > > +-----------------+-----------------+----------+ > > | di-000-004 | di-000-004 | Deployed | > > +-----------------+-----------------+----------+ > > > > > > See below json files for the applications: > > > > stratos> describe-application di-000-001 > > Application not found: di-000-001 > > > > > > stratos> describe-application di-000-002 > > Application: di-000-002 > > { > > "applicationId": "di-000-002", > > "multiTenant": false, > > "alias": "di-000-002", > > "status": "Undeploying", > > "components": { > > "cartridges": [ > > { > > "type": "cisco-qvpc-cf-02-0", > > "cartridgeMin": 1, > > "cartridgeMax": 1, > > "subscribableInfo": { > > "alias": "di-000-002", > > "deploymentPolicy": "static-1", > > "autoscalingPolicy": "economyPolicy", > > "maxMembers": 0, > > "minMembers": 0, > > "artifactRepository": { > > "alias": "di-000-002", > > "privateRepo": true, > > "repoUrl": "http://octl.qmog.cisco.com:10080/git/default.git";, > > "repoUsername": "user", > > "repoPassword": "password" > > }, > > "property": [ > > { > > "name": "payload_parameter.VOLUME_INFO", > > "value": "di-000-002:ca6bbb1b-47a5-44c5-af41-6264eb2bdcfd" > > } > > ], > > "persistence": { > > "isRequired": false > > } > > } > > } > > ] > > } > > } > > > > //// > > Application: di-000-003 > > { > > "applicationId": "di-000-003", > > "multiTenant": false, > > "alias": "di-000-003", > > "status": "Deployed", > > "components": { > > "cartridges": [ > > { > > "type": "cisco-qvpc-sf-0", > > "cartridgeMin": 1, > > "cartridgeMax": 1, > > "subscribableInfo": { > > "alias": "di-000-003", > > "deploymentPolicy": "static-1", > > "autoscalingPolicy": "economyPolicy", > > "maxMembers": 0, > > "minMembers": 0, > > "artifactRepository": { > > "alias": "di-000-003", > > "privateRepo": true, > > "repoUrl": "http://octl.qmog.cisco.com:10080/git/default.git";, > > "repoUsername": "user", > > "repoPassword": "password" > > } > > } > > } > > ] > > } > > } > > > > ///// > > stratos> describe-application di-000-004 > > Application: di-000-004 > > { > > "applicationId": "di-000-004", > > "multiTenant": false, > > "alias": "di-000-004", > > "status": "Deployed", > > "components": { > > "cartridges": [ > > { > > "type": "cisco-qvpc-sf-0", > > "cartridgeMin": 1, > > "cartridgeMax": 1, > > "subscribableInfo": { > > "alias": "di-000-004", > > "deploymentPolicy": "static-1", > > "autoscalingPolicy": "economyPolicy", > > "maxMembers": 0, > > "minMembers": 0, > > "artifactRepository": { > > "alias": "di-000-004", > > "privateRepo": true, > > "repoUrl": "http://octl.qmog.cisco.com:10080/git/default.git";, > > "repoUsername": "user", > > "repoPassword": "password" > > } > > } > > } > > ] > > } > > } > > > > //// > > > > Application: cartridge-proxy > > { > > "applicationId": "cartridge-proxy", > > "multiTenant": false, > > "alias": "cartridge-proxy", > > "status": "Deployed", > > "components": { > > "cartridges": [ > > { > > "type": "cartridge-proxy", > > "cartridgeMin": 1, > > "cartridgeMax": 1, > > "subscribableInfo": { > > "alias": "cartridge-proxy", > > "deploymentPolicy": "static-1", > > "autoscalingPolicy": "economyPolicy", > > "maxMembers": 0, > > "minMembers": 0, > > "artifactRepository": { > > "alias": "cartridge-proxy", > > "privateRepo": true, > > "repoUrl": "http://octl.qmog.cisco.com:10080/git/default.git";, > > "repoUsername": "user", > > "repoPassword": "password" > > } > > } > > } > > ] > > } > > } > > > > > > > > > > > > > -- > > Reka Thirunavukkarasu > Senior Software Engineer, > WSO2, Inc.:http://wso2.com, > > Mobile: +94776442007 > > > -- Reka Thirunavukkarasu Senior Software Engineer, WSO2, Inc.:http://wso2.com, Mobile: +94776442007
