[ https://issues.apache.org/jira/browse/CLOUDSTACK-3876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Likitha Shetty reassigned CLOUDSTACK-3876: ------------------------------------------ Assignee: Likitha Shetty > Runtime Exceptions while deleting account which has snapshots created from > volume > --------------------------------------------------------------------------------- > > Key: CLOUDSTACK-3876 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-3876 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Storage Controller > Affects Versions: 4.2.0 > Reporter: Sailaja Mada > Assignee: Likitha Shetty > Fix For: 4.2.0 > > > Steps: > 1. Configure AdvZone with Xenserver > 2. Create new account > 3. Deploy VM > 4. Add new volume and attach to this instance > 5. Create snapshots to this volume ( Total4 ) > 6. Delete the account > 7. Account got deleted with no issues but there were Runtime exceptions in > the log while deleting the snapshots of DATA volume. > 2013-07-27 14:24:55,363 DEBUG [cloud.api.ApiServlet] (catalina-exec-2:null) > ===START=== 10.104.255.13 -- GET > command=deleteAccount&response=json&sessionkey=C%2FrzjdA6B1wbWrgmfiasGDl6A6Q%3D&id=7cf7f1c8-b1f2-4754-9049-10506322e037&_=1374915066313 > 2013-07-27 14:24:55,398 DEBUG [cloud.async.AsyncJobManagerImpl] > (catalina-exec-2:null) submit async job-45 = [ > e87af4d1-54e2-46b4-bbc9-3b020bb364ba ], details: AsyncJobVO {id:45, userId: > 2, accountId: 2, sessionKey: null, instanceType: Account, instanceId: null, > cmd: org.apache.cloudstack.api.command.admin.account.DeleteAccountCmd, > cmdOriginator: null, cmdInfo: > {"response":"json","id":"7cf7f1c8-b1f2-4754-9049-10506322e037","sessionkey":"C/rzjdA6B1wbWrgmfiasGDl6A6Q\u003d","cmdEventType":"ACCOUNT.DELETE","ctxUserId":"2","httpmethod":"GET","_":"1374915066313","ctxAccountId":"2","ctxStartEventId":"163"}, > cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0, > processStatus: 0, resultCode: 0, result: null, initMsid: 226870599129537, > completeMsid: null, lastUpdated: null, lastPolled: null, created: null} > 2013-07-27 14:24:55,401 DEBUG [cloud.api.ApiServlet] (catalina-exec-2:null) > ===END=== 10.104.255.13 -- GET > command=deleteAccount&response=json&sessionkey=C%2FrzjdA6B1wbWrgmfiasGDl6A6Q%3D&id=7cf7f1c8-b1f2-4754-9049-10506322e037&_=1374915066313 > 2013-07-27 14:24:55,403 DEBUG [cloud.async.AsyncJobManagerImpl] > (Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Executing > org.apache.cloudstack.api.command.admin.account.DeleteAccountCmd for job-45 = > [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ] > 2013-07-27 14:24:55,434 DEBUG [cloud.user.AccountManagerImpl] > (Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Removed > account 4 > 2013-07-27 14:24:55,459 DEBUG [agent.transport.Request] > (Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Seq > 5-1160774981: Sending { Cmd , MgmtId: 226870599129537, via: 5, Ver: v1, > Flags: 100111, > [{"com.cloud.agent.api.DeleteSnapshotsDirCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.102.192.100/cpg_vol/abhinav/xen-sec-old","_role":"Image"}},"directory":"snapshots/4/16","wait":0}}] > } > 2013-07-27 14:24:55,551 DEBUG [agent.transport.Request] > (AgentManager-Handler-2:null) Seq 5-1160774981: Processing: { Ans: , MgmtId: > 226870599129537, via: 5, Ver: v1, Flags: 110, > [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] } > 2013-07-27 14:24:55,551 DEBUG [agent.manager.AgentAttache] > (AgentManager-Handler-2:null) Seq 5-1160774981: No more commands found > 2013-07-27 14:24:55,551 DEBUG [agent.transport.Request] > (Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Seq > 5-1160774981: Received: { Ans: , MgmtId: 226870599129537, via: 5, Ver: v1, > Flags: 110, { Answer } } > 2013-07-27 14:24:55,551 DEBUG [storage.snapshot.SnapshotManagerImpl] > (Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Deleted > all snapshots for volume: 16 under account: 4 > 2013-07-27 14:24:55,567 DEBUG [storage.snapshot.XenserverSnapshotStrategy] > (Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) delete > snapshot chain for snapshot: 4 > 2013-07-27 14:24:55,569 DEBUG [storage.snapshot.XenserverSnapshotStrategy] > (Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Snapshot: > 4 doesn't have children, so it's ok to delete it and its parents > 2013-07-27 14:24:55,616 DEBUG [agent.transport.Request] > (Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Seq > 5-1160774982: Sending { Cmd , MgmtId: 226870599129537, via: 5, Ver: v1, > Flags: 100011, > [{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/4/16/6671a6a2-565f-47ec-96cf-f804665c600d","volume":{"uuid":"bb9c0beb-ba86-4d03-9cbe-66dad892cd57","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"c6bb64ae-9919-3068-97a9-fe8ca71acbe0","id":200,"poolType":"NetworkFilesystem","host":"10.102.192.100","path":"/cpg_vol/abhinav/xen-pri1","port":2049}},"name":"ROOT-11","size":21474836480,"path":"12bb3e53-db3b-4d47-ab01-a4d8bbd5bbc3","volumeId":16,"vmName":"i-4-11-VM","accountId":4,"format":"VHD","id":16},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.102.192.100/cpg_vol/abhinav/xen-sec-old","_role":"Image"}},"vmName":"i-4-11-VM","name":"instance1_ROOT-11_20130727072147","hypervisorType":"XenServer","id":4}},"wait":0}}] > } > 2013-07-27 14:24:55,641 DEBUG [agent.transport.Request] > (AgentManager-Handler-4:null) Seq 5-1160774982: Processing: { Ans: , MgmtId: > 226870599129537, via: 5, Ver: v1, Flags: 10, > [{"com.cloud.agent.api.Answer":{"result":false,"details":"snapshot directory > 16 doesn't exist","wait":0}}] } > 2013-07-27 14:24:55,642 DEBUG [agent.transport.Request] > (Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Seq > 5-1160774982: Received: { Ans: , MgmtId: 226870599129537, via: 5, Ver: v1, > Flags: 10, { Answer } } > 2013-07-27 14:24:55,642 DEBUG [storage.snapshot.SnapshotServiceImpl] > (Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) delete > snapshot failedsnapshot directory 16 doesn't exist > 2013-07-27 14:24:55,673 DEBUG [storage.snapshot.XenserverSnapshotStrategy] > (Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Failed to > delete snapshot: > com.cloud.utils.exception.CloudRuntimeException: snapshot directory 16 > doesn't exist > at > org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.deleteSnapshot(SnapshotServiceImpl.java:375) > at > org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.deleteSnapshotChain(XenserverSnapshotStrategy.java:143) > at > org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.deleteSnapshot(XenserverSnapshotStrategy.java:187) > at > com.cloud.storage.snapshot.SnapshotManagerImpl.deleteSnapshotDirsForAccount(SnapshotManagerImpl.java:689) > at > com.cloud.user.AccountManagerImpl.cleanupAccount(AccountManagerImpl.java:581) > at > com.cloud.user.AccountManagerImpl.deleteAccount(AccountManagerImpl.java:544) > at > com.cloud.user.AccountManagerImpl.deleteUserAccount(AccountManagerImpl.java:1273) > at > com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) > at > org.apache.cloudstack.region.RegionManagerImpl.deleteUserAccount(RegionManagerImpl.java:177) > at > org.apache.cloudstack.region.RegionServiceImpl.deleteUserAccount(RegionServiceImpl.java:118) > at > org.apache.cloudstack.api.command.admin.account.DeleteAccountCmd.execute(DeleteAccountCmd.java:100) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158) > at > com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:636) > 2013-07-27 14:24:55,736 DEBUG [agent.transport.Request] > (Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Seq > 5-1160774983: Sending { Cmd , MgmtId: 226870599129537, via: 5, Ver: v1, > Flags: 100111, > [{"com.cloud.agent.api.DeleteSnapshotsDirCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.102.192.100/cpg_vol/abhinav/xen-sec-old","_role":"Image"}},"directory":"snapshots/4/18","wait":0}}] > } > 2013-07-27 14:24:55,866 DEBUG [agent.transport.Request] > (AgentManager-Handler-3:null) Seq 5-1160774983: Processing: { Ans: , MgmtId: > 226870599129537, via: 5, Ver: v1, Flags: 110, > [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] } > 2013-07-27 14:24:55,866 DEBUG [agent.manager.AgentAttache] > (AgentManager-Handler-3:null) Seq 5-1160774983: No more commands found > 2013-07-27 14:24:55,866 DEBUG [agent.transport.Request] > (Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Seq > 5-1160774983: Received: { Ans: , MgmtId: 226870599129537, via: 5, Ver: v1, > Flags: 110, { Answer } } > 2013-07-27 14:24:55,866 DEBUG [storage.snapshot.SnapshotManagerImpl] > (Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Deleted > all snapshots for volume: 18 under account: 4 > 2013-07-27 14:24:55,882 DEBUG [storage.snapshot.XenserverSnapshotStrategy] > (Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) delete > snapshot chain for snapshot: 8 > 2013-07-27 14:24:55,885 DEBUG [storage.snapshot.XenserverSnapshotStrategy] > (Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) the > snapshot has child, can't delete it on the storage > 2013-07-27 14:24:55,944 DEBUG [storage.snapshot.XenserverSnapshotStrategy] > (Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) delete > snapshot chain for snapshot: 9 > 2013-07-27 14:24:55,947 DEBUG [storage.snapshot.XenserverSnapshotStrategy] > (Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) the > snapshot has child, can't delete it on the storage > 2013-07-27 14:24:56,011 DEBUG [storage.snapshot.XenserverSnapshotStrategy] > (Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) delete > snapshot chain for snapshot: 10 > 2013-07-27 14:24:56,014 DEBUG [storage.snapshot.XenserverSnapshotStrategy] > (Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) the > snapshot has child, can't delete it on the storage > 2013-07-27 14:24:56,075 DEBUG [storage.snapshot.XenserverSnapshotStrategy] > (Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) delete > snapshot chain for snapshot: 11 > 2013-07-27 14:24:56,077 DEBUG [storage.snapshot.XenserverSnapshotStrategy] > (Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Snapshot: > 11 doesn't have children, so it's ok to delete it and its parents > 2013-07-27 14:24:56,129 DEBUG [agent.transport.Request] > (Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Seq > 5-1160774984: Sending { Cmd , MgmtId: 226870599129537, via: 5, Ver: v1, > Flags: 100011, > [{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/4/18/c44a1894-0142-444a-9c75-868358893a6a","volume":{"uuid":"f4e0e5fa-1a98-4dbb-b7b4-10e4ddf28465","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"c6bb64ae-9919-3068-97a9-fe8ca71acbe0","id":200,"poolType":"NetworkFilesystem","host":"10.102.192.100","path":"/cpg_vol/abhinav/xen-pri1","port":2049}},"name":"Volumenew1","size":5368709120,"path":"a06f2cde-31f2-41e3-bdaa-82d224996856","volumeId":18,"vmName":"i-4-11-VM","accountId":4,"format":"VHD","id":18},"parentSnapshotPath":"snapshots/4/18/c44a1894-0142-444a-9c75-868358893a6a","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.102.192.100/cpg_vol/abhinav/xen-sec-old","_role":"Image"}},"vmName":"i-4-11-VM","name":"instance1_Volumenew1_20130727085422","hypervisorType":"XenServer","id":11}},"wait":0}}] > } > 2013-07-27 14:24:56,174 DEBUG [agent.transport.Request] > (AgentManager-Handler-7:null) Seq 5-1160774984: Processing: { Ans: , MgmtId: > 226870599129537, via: 5, Ver: v1, Flags: 10, > [{"com.cloud.agent.api.Answer":{"result":false,"details":"snapshot directory > 18 doesn't exist","wait":0}}] } > 2013-07-27 14:24:56,175 DEBUG [agent.transport.Request] > (Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Seq > 5-1160774984: Received: { Ans: , MgmtId: 226870599129537, via: 5, Ver: v1, > Flags: 10, { Answer } } > 2013-07-27 14:24:56,175 DEBUG [storage.snapshot.SnapshotServiceImpl] > (Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) delete > snapshot failedsnapshot directory 18 doesn't exist > 2013-07-27 14:24:56,192 DEBUG [storage.snapshot.XenserverSnapshotStrategy] > (Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Failed to > delete snapshot: > com.cloud.utils.exception.CloudRuntimeException: snapshot directory 18 > doesn't exist > at > org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.deleteSnapshot(SnapshotServiceImpl.java:375) > at > org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.deleteSnapshotChain(XenserverSnapshotStrategy.java:143) > at > org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.deleteSnapshot(XenserverSnapshotStrategy.java:187) > at > com.cloud.storage.snapshot.SnapshotManagerImpl.deleteSnapshotDirsForAccount(SnapshotManagerImpl.java:689) > at > com.cloud.user.AccountManagerImpl.cleanupAccount(AccountManagerImpl.java:581) > at > com.cloud.user.AccountManagerImpl.deleteAccount(AccountManagerImpl.java:544) > at > com.cloud.user.AccountManagerImpl.deleteUserAccount(AccountManagerImpl.java:1273) > at > com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) > at > org.apache.cloudstack.region.RegionManagerImpl.deleteUserAccount(RegionManagerImpl.java:177) > at > org.apache.cloudstack.region.RegionServiceImpl.deleteUserAccount(RegionServiceImpl.java:118) > at > org.apache.cloudstack.api.command.admin.account.DeleteAccountCmd.execute(DeleteAccountCmd.java:100) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158) > at > com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:636) > 2013-07-27 14:24:56,234 DEBUG [cloud.user.AccountManagerImpl] > (Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) > Successfully deleted snapshots directories for all volumes under account 4 > across all zones > 2013-07-27 14:24:56,238 DEBUG [cloud.user.AccountManagerImpl] > (Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Expunging > # of vms (accountId=4): 1 > 2013-07-27 14:24:56,261 DEBUG [cloud.capacity.CapacityManagerImpl] > (Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) VM state > transitted from :Running to Stopping with event: StopRequestedvm's original > host id: 1 new host id: 1 host id before state transition: 1 > 2013-07-27 14:24:56,265 DEBUG [agent.transport.Request] > (Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Seq > 1-1592922337: Sending { Cmd , MgmtId: 226870599129537, via: 1, Ver: v1, > Flags: 100011, > [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"i-4-11-VM","wait":0}}] > } -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira