[
https://issues.apache.org/jira/browse/CLOUDSTACK-7799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14200036#comment-14200036
]
Sanjay Tripathi commented on CLOUDSTACK-7799:
---------------------------------------------
Hi Sanjeev,
If one entity fails to delete then CS set the accountCleanUpNeeded to true and
deletes the account entry from DB. Later account cleanup thread picks up this
account for cleaning and deletes all the entities associated with this account.
Admin can set the "account.cleanup.interval" gc parameter to cleanup the
accounts frequently.
> [Accounts] Account deletion failed to remove all the entities related to it
> in case of failure in deleting one entity
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: CLOUDSTACK-7799
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7799
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Management Server
> Affects Versions: 4.5.0
> Environment: Latest build from 4.5 with commit [root@BPKxDmS ~]#
> cloudstack-sccs
> 385c4f673dfbd1fd326e539625e2c06db4cdc27d
> Reporter: Sanjeev N
> Assignee: Sanjay Tripathi
> Priority: Critical
> Fix For: 4.5.0
>
> Attachments: cloud.dmp, management-server.rar
>
>
> [Critical][Accounts] Account deletion failed to remove all the entities
> related to it in case of failure in deleting one entity
> Steps to Reproduce:
> ================
> 1.Bring up CS in advanced zone with one vmware cluster
> 2.Create a user account
> 3.With the user account deploy few (4-5) vms
> 4.Take snapshots on all of the vms root disks
> 5.Deploy another vm and simulate snapshot failure operation(In my case I
> tried snapshot operation with quiesce option set to true) so that snapshot
> will be in "Allocated" state
> 6.Delete this user account
> Expected Behavior:
> ==============
> Whatever may the state of the snapshot account deletion shall clean all the
> objects related that account
> Actual Behavior:
> =============
> Account deletion started deleting snapshots. Since one of the snapshots is in
> Allocated state it failed to delete that snapshot and the Account clean up
> job ended there. So vms and networks related to the accounts are still there
> but the account was deleted.
> Here is the log snippet:
> =================
> 2014-10-28 15:16:38,822 DEBUG [c.c.a.ApiServlet]
> (catalina-exec-5:ctx-1c16bc72) ===START=== 10.252.193.8 -- GET
> command=deleteAccount&response=json&sessionkey=EpI1EuP5ZFF0VKispQyNb9AxWF4%3D&id=83f26103-9d83-4e4e-8bcc-c282c2acb498&_=1414470667367
> 2014-10-28 15:16:38,974 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (catalina-exec-5:ctx-1c16bc72 ctx-51140d28) submit async job-53, details:
> AsyncJobVO {id:53, userId: 2, accountId: 2, instanceType: Account,
> instanceId: null, cmd:
> org.apache.cloudstack.api.command.admin.account.DeleteAccountCmd, cmdInfo:
> {"response":"json","id":"83f26103-9d83-4e4e-8bcc-c282c2acb498","sessionkey":"EpI1EuP5ZFF0VKispQyNb9AxWF4\u003d","ctxDetails":"{\"com.cloud.user.Account\":\"83f26103-9d83-4e4e-8bcc-c282c2acb498\"}","cmdEventType":"ACCOUNT.DELETE","ctxUserId":"2","httpmethod":"GET","_":"1414470667367","uuid":"83f26103-9d83-4e4e-8bcc-c282c2acb498","ctxAccountId":"2","ctxStartEventId":"118"},
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
> null, initMsid: 6637838401571, completeMsid: null, lastUpdated: null,
> lastPolled: null, created: null}
> 2014-10-28 15:16:38,978 DEBUG [c.c.a.ApiServlet]
> (catalina-exec-5:ctx-1c16bc72 ctx-51140d28) ===END=== 10.252.193.8 -- GET
> command=deleteAccount&response=json&sessionkey=EpI1EuP5ZFF0VKispQyNb9AxWF4%3D&id=83f26103-9d83-4e4e-8bcc-c282c2acb498&_=1414470667367
> 2014-10-28 15:16:38,996 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (API-Job-Executor-30:ctx-0b9151e2 job-53) Add job-53 into job monitoring
> 2014-10-28 15:16:38,996 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-30:ctx-0b9151e2 job-53) Executing AsyncJobVO {id:53,
> userId: 2, accountId: 2, instanceType: Account, instanceId: null, cmd:
> org.apache.cloudstack.api.command.admin.account.DeleteAccountCmd, cmdInfo:
> {"response":"json","id":"83f26103-9d83-4e4e-8bcc-c282c2acb498","sessionkey":"EpI1EuP5ZFF0VKispQyNb9AxWF4\u003d","ctxDetails":"{\"com.cloud.user.Account\":\"83f26103-9d83-4e4e-8bcc-c282c2acb498\"}","cmdEventType":"ACCOUNT.DELETE","ctxUserId":"2","httpmethod":"GET","_":"1414470667367","uuid":"83f26103-9d83-4e4e-8bcc-c282c2acb498","ctxAccountId":"2","ctxStartEventId":"118"},
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
> null, initMsid: 6637838401571, completeMsid: null, lastUpdated: null,
> lastPolled: null, created: null}
> 2014-10-28 15:16:39,153 DEBUG [c.c.u.AccountManagerImpl]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Removed account 4
> 2014-10-28 15:16:39,287 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
> 4-4574812796478292948: Sending { Cmd , MgmtId: 6637838401571, via:
> 4(s-2-QA), Ver: v1, Flags: 100111,
> [{"com.cloud.agent.api.DeleteSnapshotsDirCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"directory":"snapshots/4/5","wait":0}}]
> }
> 2014-10-28 15:16:40,556 DEBUG [c.c.a.t.Request]
> (AgentManager-Handler-12:null) Seq 4-4574812796478292948: Processing: { Ans:
> , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 110,
> [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }
> 2014-10-28 15:16:40,556 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
> 4-4574812796478292948: Received: { Ans: , MgmtId: 6637838401571, via: 4,
> Ver: v1, Flags: 110, { Answer } }
> 2014-10-28 15:16:40,557 DEBUG [c.c.s.s.SnapshotManagerImpl]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Deleted all snapshots
> for volume: 5 under account: 4
> 2014-10-28 15:16:40,561 DEBUG [c.c.a.m.AgentAttache]
> (AgentManager-Handler-12:null) Seq 4-4574812796478292948: No more commands
> found
> 2014-10-28 15:16:40,628 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) delete snapshot chain
> for snapshot: 1
> 2014-10-28 15:16:40,637 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Snapshot: 1 doesn't
> have children, so it's ok to delete it and its parents
> 2014-10-28 15:16:40,713 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
> 4-4574812796478292949: Sending { Cmd , MgmtId: 6637838401571, via:
> 4(s-2-QA), Ver: v1, Flags: 100011,
> [{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/4/5/a2b6fc7f-c086-4494-aa89-d271f0b96d36/a2b6fc7f-c086-4494-aa89-d271f0b96d36","volume":{"uuid":"280c4f37-87ff-4b62-b1f5-5c610e90531d","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"adc69295-88b2-3db8-9e55-fe7db3847af9","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_cluster","port":2049,"url":"NetworkFilesystem://10.147.28.7/export/home/sanjeev/pri_cluster/?ROLE=Primary&STOREUUID=adc69295-88b2-3db8-9e55-fe7db3847af9"}},"name":"ROOT-5","size":2147483648,"path":"ROOT-5","volumeId":5,"vmName":"i-4-5-QA","accountId":4,"chainInfo":"{\"diskDeviceBusName\":\"ide0:1\",\"diskChain\":[\"[adc6929588b23db89e55fe7db3847af9]
> i-4-5-QA/ROOT-5.vmdk\",\"[adc6929588b23db89e55fe7db3847af9]
> 0bc482b12d62345e9fca3cbce554086c/0bc482b12d62345e9fca3cbce554086c.vmdk\"]}","format":"OVA","provisioningType":"THIN","id":5,"deviceId":0,"hypervisorType":"VMware"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"vmName":"i-4-5-QA","name":"v1_ROOT-5_20141027213127","hypervisorType":"VMware","id":1,"quiescevm":false,"physicalSize":0}},"wait":0}}]
> }
> 2014-10-28 15:16:40,768 DEBUG [c.c.a.t.Request]
> (AgentManager-Handler-17:null) Seq 4-4574812796478292949: Processing: { Ans:
> , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 10,
> [{"com.cloud.agent.api.Answer":{"result":true,"details":"snapshot directory
> a2b6fc7f-c086-4494-aa89-d271f0b96d36 doesn't exist","wait":0}}] }
> 2014-10-28 15:16:40,768 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
> 4-4574812796478292949: Received: { Ans: , MgmtId: 6637838401571, via: 4,
> Ver: v1, Flags: 10, { Answer } }
> 2014-10-28 15:16:40,874 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
> 4-4574812796478292950: Sending { Cmd , MgmtId: 6637838401571, via:
> 4(s-2-QA), Ver: v1, Flags: 100111,
> [{"com.cloud.agent.api.DeleteSnapshotsDirCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"directory":"snapshots/4/7","wait":0}}]
> }
> 2014-10-28 15:16:42,048 DEBUG [c.c.a.ApiServlet]
> (catalina-exec-6:ctx-452a9eca) ===START=== 10.252.193.8 -- GET
> command=queryAsyncJobResult&jobId=2c5b0fa0-ef16-4a61-bd64-7cae6acfdaa4&response=json&sessionkey=EpI1EuP5ZFF0VKispQyNb9AxWF4%3D&_=1414470670595
> 2014-10-28 15:16:42,100 DEBUG [c.c.a.ApiServlet]
> (catalina-exec-6:ctx-452a9eca ctx-9e36e958) ===END=== 10.252.193.8 -- GET
> command=queryAsyncJobResult&jobId=2c5b0fa0-ef16-4a61-bd64-7cae6acfdaa4&response=json&sessionkey=EpI1EuP5ZFF0VKispQyNb9AxWF4%3D&_=1414470670595
> 2014-10-28 15:16:42,267 DEBUG [c.c.a.t.Request]
> (AgentManager-Handler-15:null) Seq 4-4574812796478292950: Processing: { Ans:
> , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 110,
> [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }
> 2014-10-28 15:16:42,268 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
> 4-4574812796478292950: Received: { Ans: , MgmtId: 6637838401571, via: 4,
> Ver: v1, Flags: 110, { Answer } }
> 2014-10-28 15:16:42,268 DEBUG [c.c.s.s.SnapshotManagerImpl]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Deleted all snapshots
> for volume: 7 under account: 4
> 2014-10-28 15:16:42,271 DEBUG [c.c.a.m.AgentAttache]
> (AgentManager-Handler-15:null) Seq 4-4574812796478292950: No more commands
> found
> 2014-10-28 15:16:42,311 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) delete snapshot chain
> for snapshot: 2
> 2014-10-28 15:16:42,314 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Snapshot: 2 doesn't
> have children, so it's ok to delete it and its parents
> 2014-10-28 15:16:42,347 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
> 4-4574812796478292951: Sending { Cmd , MgmtId: 6637838401571, via:
> 4(s-2-QA), Ver: v1, Flags: 100011,
> [{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/4/7/ecb858aa-1656-4594-996a-869f1695bce2/ecb858aa-1656-4594-996a-869f1695bce2","volume":{"uuid":"5b639b64-a539-4651-ac4e-f06b69f37d97","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"adc69295-88b2-3db8-9e55-fe7db3847af9","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_cluster","port":2049,"url":"NetworkFilesystem://10.147.28.7/export/home/sanjeev/pri_cluster/?ROLE=Primary&STOREUUID=adc69295-88b2-3db8-9e55-fe7db3847af9"}},"name":"ROOT-7","size":2147483648,"path":"ROOT-7","volumeId":7,"vmName":"i-4-7-QA","accountId":4,"chainInfo":"{\"diskDeviceBusName\":\"ide0:1\",\"diskChain\":[\"[adc6929588b23db89e55fe7db3847af9]
> i-4-7-QA/ROOT-7.vmdk\",\"[adc6929588b23db89e55fe7db3847af9]
> 0bc482b12d62345e9fca3cbce554086c/0bc482b12d62345e9fca3cbce554086c.vmdk\"]}","format":"OVA","provisioningType":"THIN","id":7,"deviceId":0,"hypervisorType":"VMware"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"vmName":"i-4-7-QA","name":"v2_ROOT-7_20141027213212","hypervisorType":"VMware","id":2,"quiescevm":false,"physicalSize":0}},"wait":0}}]
> }
> 2014-10-28 15:16:42,359 DEBUG [c.c.a.t.Request]
> (AgentManager-Handler-16:null) Seq 4-4574812796478292951: Processing: { Ans:
> , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 10,
> [{"com.cloud.agent.api.Answer":{"result":true,"details":"snapshot directory
> ecb858aa-1656-4594-996a-869f1695bce2 doesn't exist","wait":0}}] }
> 2014-10-28 15:16:42,360 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
> 4-4574812796478292951: Received: { Ans: , MgmtId: 6637838401571, via: 4,
> Ver: v1, Flags: 10, { Answer } }
> 2014-10-28 15:16:42,434 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
> 4-4574812796478292952: Sending { Cmd , MgmtId: 6637838401571, via:
> 4(s-2-QA), Ver: v1, Flags: 100111,
> [{"com.cloud.agent.api.DeleteSnapshotsDirCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"directory":"snapshots/4/8","wait":0}}]
> }
> 2014-10-28 15:16:43,338 DEBUG [c.c.a.t.Request]
> (AgentManager-Handler-19:null) Seq 4-4574812796478292952: Processing: { Ans:
> , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 110,
> [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }
> 2014-10-28 15:16:43,339 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
> 4-4574812796478292952: Received: { Ans: , MgmtId: 6637838401571, via: 4,
> Ver: v1, Flags: 110, { Answer } }
> 2014-10-28 15:16:43,339 DEBUG [c.c.s.s.SnapshotManagerImpl]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Deleted all snapshots
> for volume: 8 under account: 4
> 2014-10-28 15:16:43,342 DEBUG [c.c.a.m.AgentAttache]
> (AgentManager-Handler-19:null) Seq 4-4574812796478292952: No more commands
> found
> 2014-10-28 15:16:43,354 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) delete snapshot chain
> for snapshot: 3
> 2014-10-28 15:16:43,356 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Snapshot: 3 doesn't
> have children, so it's ok to delete it and its parents
> 2014-10-28 15:16:43,389 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
> 4-4574812796478292953: Sending { Cmd , MgmtId: 6637838401571, via:
> 4(s-2-QA), Ver: v1, Flags: 100011,
> [{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/4/8/657a763c-4fc6-49e0-8e7b-cb1ad2471a41/657a763c-4fc6-49e0-8e7b-cb1ad2471a41","volume":{"uuid":"1aebff64-f4c6-45bb-add1-249d54eca270","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"adc69295-88b2-3db8-9e55-fe7db3847af9","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_cluster","port":2049,"url":"NetworkFilesystem://10.147.28.7/export/home/sanjeev/pri_cluster/?ROLE=Primary&STOREUUID=adc69295-88b2-3db8-9e55-fe7db3847af9"}},"name":"ROOT-8","size":2147483648,"path":"ROOT-8","volumeId":8,"vmName":"i-4-8-QA","accountId":4,"chainInfo":"{\"diskDeviceBusName\":\"ide0:1\",\"diskChain\":[\"[adc6929588b23db89e55fe7db3847af9]
> i-4-8-QA/ROOT-8.vmdk\",\"[adc6929588b23db89e55fe7db3847af9]
> 0bc482b12d62345e9fca3cbce554086c/0bc482b12d62345e9fca3cbce554086c.vmdk\"]}","format":"OVA","provisioningType":"THIN","id":8,"deviceId":0,"hypervisorType":"VMware"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"vmName":"i-4-8-QA","name":"v3_ROOT-8_20141027213237","hypervisorType":"VMware","id":3,"quiescevm":false,"physicalSize":0}},"wait":0}}]
> }
> 2014-10-28 15:16:43,400 DEBUG [c.c.a.t.Request]
> (AgentManager-Handler-20:null) Seq 4-4574812796478292953: Processing: { Ans:
> , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 10,
> [{"com.cloud.agent.api.Answer":{"result":true,"details":"snapshot directory
> 657a763c-4fc6-49e0-8e7b-cb1ad2471a41 doesn't exist","wait":0}}] }
> 2014-10-28 15:16:43,400 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
> 4-4574812796478292953: Received: { Ans: , MgmtId: 6637838401571, via: 4,
> Ver: v1, Flags: 10, { Answer } }
> 2014-10-28 15:16:43,483 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
> 4-4574812796478292954: Sending { Cmd , MgmtId: 6637838401571, via:
> 4(s-2-QA), Ver: v1, Flags: 100111,
> [{"com.cloud.agent.api.DeleteSnapshotsDirCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"directory":"snapshots/4/9","wait":0}}]
> }
> 2014-10-28 15:16:43,758 DEBUG [c.c.s.StatsCollector]
> (StatsCollector-4:ctx-6ecc0a29) AutoScaling Monitor is running...
> 2014-10-28 15:16:44,555 DEBUG [c.c.a.t.Request] (AgentManager-Handler-2:null)
> Seq 4-4574812796478292954: Processing: { Ans: , MgmtId: 6637838401571, via:
> 4, Ver: v1, Flags: 110,
> [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }
> 2014-10-28 15:16:44,556 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
> 4-4574812796478292954: Received: { Ans: , MgmtId: 6637838401571, via: 4,
> Ver: v1, Flags: 110, { Answer } }
> 2014-10-28 15:16:44,556 DEBUG [c.c.s.s.SnapshotManagerImpl]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Deleted all snapshots
> for volume: 9 under account: 4
> 2014-10-28 15:16:44,559 DEBUG [c.c.a.m.AgentAttache]
> (AgentManager-Handler-2:null) Seq 4-4574812796478292954: No more commands
> found
> 2014-10-28 15:16:44,579 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) delete snapshot chain
> for snapshot: 4
> 2014-10-28 15:16:44,582 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Snapshot: 4 doesn't
> have children, so it's ok to delete it and its parents
> 2014-10-28 15:16:44,616 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
> 4-4574812796478292955: Sending { Cmd , MgmtId: 6637838401571, via:
> 4(s-2-QA), Ver: v1, Flags: 100011,
> [{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/4/9/e7d15b83-06be-4596-9572-09478415e776/e7d15b83-06be-4596-9572-09478415e776","volume":{"uuid":"e88d9c15-e81b-44fd-99be-4d14f3d19c06","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"adc69295-88b2-3db8-9e55-fe7db3847af9","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_cluster","port":2049,"url":"NetworkFilesystem://10.147.28.7/export/home/sanjeev/pri_cluster/?ROLE=Primary&STOREUUID=adc69295-88b2-3db8-9e55-fe7db3847af9"}},"name":"ROOT-9","size":2147483648,"path":"ROOT-9","volumeId":9,"vmName":"i-4-9-QA","accountId":4,"chainInfo":"{\"diskDeviceBusName\":\"ide0:1\",\"diskChain\":[\"[adc6929588b23db89e55fe7db3847af9]
> i-4-9-QA/ROOT-9.vmdk\",\"[adc6929588b23db89e55fe7db3847af9]
> 0bc482b12d62345e9fca3cbce554086c/0bc482b12d62345e9fca3cbce554086c.vmdk\"]}","format":"OVA","provisioningType":"THIN","id":9,"deviceId":0,"hypervisorType":"VMware"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"vmName":"i-4-9-QA","name":"v4_ROOT-9_20141027213330","hypervisorType":"VMware","id":4,"quiescevm":false,"physicalSize":0}},"wait":0}}]
> }
> 2014-10-28 15:16:44,628 DEBUG [c.c.a.t.Request] (AgentManager-Handler-1:null)
> Seq 4-4574812796478292955: Processing: { Ans: , MgmtId: 6637838401571, via:
> 4, Ver: v1, Flags: 10,
> [{"com.cloud.agent.api.Answer":{"result":true,"details":"snapshot directory
> e7d15b83-06be-4596-9572-09478415e776 doesn't exist","wait":0}}] }
> 2014-10-28 15:16:44,628 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
> 4-4574812796478292955: Received: { Ans: , MgmtId: 6637838401571, via: 4,
> Ver: v1, Flags: 10, { Answer } }
> 2014-10-28 15:16:44,692 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
> 4-4574812796478292956: Sending { Cmd , MgmtId: 6637838401571, via:
> 4(s-2-QA), Ver: v1, Flags: 100111,
> [{"com.cloud.agent.api.DeleteSnapshotsDirCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"directory":"snapshots/4/10","wait":0}}]
> }
> 2014-10-28 15:16:44,805 DEBUG [c.c.a.t.Request] (AgentManager-Handler-3:null)
> Seq 4-4574812796478292956: Processing: { Ans: , MgmtId: 6637838401571, via:
> 4, Ver: v1, Flags: 110,
> [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }
> 2014-10-28 15:16:44,805 DEBUG [c.c.a.m.AgentAttache]
> (AgentManager-Handler-3:null) Seq 4-4574812796478292956: No more commands
> found
> 2014-10-28 15:16:44,805 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
> 4-4574812796478292956: Received: { Ans: , MgmtId: 6637838401571, via: 4,
> Ver: v1, Flags: 110, { Answer } }
> 2014-10-28 15:16:44,805 DEBUG [c.c.s.s.SnapshotManagerImpl]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Deleted all snapshots
> for volume: 10 under account: 4
> 2014-10-28 15:16:44,820 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) delete snapshot chain
> for snapshot: 5
> 2014-10-28 15:16:44,822 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Snapshot: 5 doesn't
> have children, so it's ok to delete it and its parents
> 2014-10-28 15:16:44,849 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
> 4-4574812796478292957: Sending { Cmd , MgmtId: 6637838401571, via:
> 4(s-2-QA), Ver: v1, Flags: 100011,
> [{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/4/10/1c4b5c4a-b14e-4e42-9a0e-698215db758a/1c4b5c4a-b14e-4e42-9a0e-698215db758a","volume":{"uuid":"cb40dae3-6c47-40b1-8f07-f7ef6d095995","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"adc69295-88b2-3db8-9e55-fe7db3847af9","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_cluster","port":2049,"url":"NetworkFilesystem://10.147.28.7/export/home/sanjeev/pri_cluster/?ROLE=Primary&STOREUUID=adc69295-88b2-3db8-9e55-fe7db3847af9"}},"name":"Data-10","size":5368709120,"path":"9bd9c800f78545db87fc805e9f3fec1d","volumeId":10,"accountId":4,"format":"OVA","provisioningType":"THIN","id":10,"hypervisorType":"VMware"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"name":"detached_Data-10_20141027214535","hypervisorType":"VMware","id":5,"quiescevm":false,"physicalSize":0}},"wait":0}}]
> }
> 2014-10-28 15:16:44,860 DEBUG [c.c.a.t.Request] (AgentManager-Handler-5:null)
> Seq 4-4574812796478292957: Processing: { Ans: , MgmtId: 6637838401571, via:
> 4, Ver: v1, Flags: 10,
> [{"com.cloud.agent.api.Answer":{"result":true,"details":"snapshot directory
> 1c4b5c4a-b14e-4e42-9a0e-698215db758a doesn't exist","wait":0}}] }
> 2014-10-28 15:16:44,860 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
> 4-4574812796478292957: Received: { Ans: , MgmtId: 6637838401571, via: 4,
> Ver: v1, Flags: 10, { Answer } }
> 2014-10-28 15:16:44,928 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
> 4-4574812796478292958: Sending { Cmd , MgmtId: 6637838401571, via:
> 4(s-2-QA), Ver: v1, Flags: 100111,
> [{"com.cloud.agent.api.DeleteSnapshotsDirCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"directory":"snapshots/4/11","wait":0}}]
> }
> 2014-10-28 15:16:44,975 DEBUG [c.c.a.t.Request] (AgentManager-Handler-4:null)
> Seq 4-4574812796478292958: Processing: { Ans: , MgmtId: 6637838401571, via:
> 4, Ver: v1, Flags: 110,
> [{"com.cloud.agent.api.Answer":{"result":true,"details":"snapshot directory
> 11 doesn't exist","wait":0}}] }
> 2014-10-28 15:16:44,975 DEBUG [c.c.a.m.AgentAttache]
> (AgentManager-Handler-4:null) Seq 4-4574812796478292958: No more commands
> found
> 2014-10-28 15:16:44,975 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
> 4-4574812796478292958: Received: { Ans: , MgmtId: 6637838401571, via: 4,
> Ver: v1, Flags: 110, { Answer } }
> 2014-10-28 15:16:44,975 DEBUG [c.c.s.s.SnapshotManagerImpl]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Deleted all snapshots
> for volume: 11 under account: 4
> 2014-10-28 15:16:44,985 WARN [c.c.u.AccountManagerImpl]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Failed to cleanup
> account Acct[83f26103-9d83-4e4e-8bcc-c282c2acb498-acc] due to
> com.cloud.exception.InvalidParameterValueException: Can't delete snapshotshot
> 6 due to it is in Allocated Status
> at
> org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.deleteSnapshot(XenserverSnapshotStrategy.java:213)
> at
> com.cloud.storage.snapshot.SnapshotManagerImpl.deleteSnapshotDirsForAccount(SnapshotManagerImpl.java:623)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at $Proxy180.deleteSnapshotDirsForAccount(Unknown Source)
> at
> com.cloud.user.AccountManagerImpl.cleanupAccount(AccountManagerImpl.java:706)
> at
> com.cloud.user.AccountManagerImpl.deleteAccount(AccountManagerImpl.java:665)
> at
> com.cloud.user.AccountManagerImpl.deleteUserAccount(AccountManagerImpl.java:1444)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at
> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
> at
> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at $Proxy103.deleteUserAccount(Unknown Source)
> at
> org.apache.cloudstack.region.RegionManagerImpl.deleteUserAccount(RegionManagerImpl.java:187)
> at
> org.apache.cloudstack.region.RegionServiceImpl.deleteUserAccount(RegionServiceImpl.java:121)
> at
> org.apache.cloudstack.api.command.admin.account.DeleteAccountCmd.execute(DeleteAccountCmd.java:104)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141)
> at
> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:513)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:470)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> 2014-10-28 15:16:45,008 INFO [c.c.u.AccountManagerImpl]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Cleanup for account 4
> is needed.
> 2014-10-28 15:16:45,047 DEBUG [c.c.a.ApiServlet]
> (catalina-exec-24:ctx-bcb44ca9) ===START=== 10.252.193.8 -- GET
> command=queryAsyncJobResult&jobId=2c5b0fa0-ef16-4a61-bd64-7cae6acfdaa4&response=json&sessionkey=EpI1EuP5ZFF0VKispQyNb9AxWF4%3D&_=1414470673593
> 2014-10-28 15:16:45,082 DEBUG [c.c.a.ApiServlet]
> (catalina-exec-24:ctx-bcb44ca9 ctx-82b3efc5) ===END=== 10.252.193.8 -- GET
> command=queryAsyncJobResult&jobId=2c5b0fa0-ef16-4a61-bd64-7cae6acfdaa4&response=json&sessionkey=EpI1EuP5ZFF0VKispQyNb9AxWF4%3D&_=1414470673593
> 2014-10-28 15:16:45,230 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Complete async job-53,
> jobStatus: SUCCEEDED, resultCode: 0, result:
> org.apache.cloudstack.api.response.SuccessResponse/null/{"success":true}
> 2014-10-28 15:16:45,232 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Publish async job-53
> complete on message bus
> 2014-10-28 15:16:45,233 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Wake up jobs related
> to job- 53
> 2014-10-28 15:16:45,233 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Update db status for
> job- 53
> 2014-10-28 15:16:45,236 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Wake up jobs joined
> with job- 53 and disjoin all subjobs created from job- 53
> 2014-10-28 15:16:45,241 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-30:ctx-0b9151e2 job-53) Done executing
> org.apache.cloudstack.api.command.admin.account.DeleteAccountCmd for job-53
> 2014-10-28 15:16:45,248 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (API-Job-Executor-30:ctx-0b9151e2 job-53) Remove job-53 from job monitoring
> Even the account cleanup thread could not clean any of the remained artifacts
> like vms,networks
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
