Anyone any hints on this? I've installed it a few times and always have
the same issue. Wondering if it's some configuration problem or a known
bug.
Thanks,
Tom.
On Thu, 2014-03-27 at 11:55 +0900, Thomas O'Dowd wrote:
> I remember this used to work on Cloudstack 4.2 when I was testing it
> with devcloud but with CloudPlatform 4.2.1 and KVM I can't get snapshots
> to work.
>
> I've setup a single node using KVM and management server, db and agent.
>
> I can start an instance using a template stored on NFS secondary
> storage. If I stop the instance and goto storage, I can see it's disk
> and then I can select "take snapshot". This always seems to fail for me
> and I'm not sure why.
>
> The error seems to be:
> Failed to create snapshot
> com.cloud.utils.exception.CloudRuntimeException: Failed to
> backup 4d3ea6fa-6902-4037-be67-166e34575ee9 for
> disk
> /mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4
> to /mnt/e5c41d36-90d2-3d5e-bdbf-8a0140dd330e/snapshots/2/3
>
> /mnt/8a218... is mounted fine if I look at df.
> /mnt/e5c41... is not mounted.
>
> If I browse to /mnt, the e5c41... directory (used for mounting) exists.
> If I manually mount it, I can see the
> directory /mnt/e5c41.../snapshots/2/3 so it exists. If I leave it
> mounted and try to take a snapshot again, it fails again with the same
> error and again when I check the /mnt/e5c41... directory is not mounted.
>
> As I mentioned above, the NFS server I'm using is actually the same
> node. I'm using KVM but I don't think that is causing the issue? I'm
> using CentOS 6.4.
>
> If I setup S3 as secondary storage and use NFS as the staging server, I
> get the same error. Any hints on what I am doing wrong is greatly
> appreciated.
>
> Here is more of the exception log:
> 2014-03-24 16:12:33,450 DEBUG [cloud.api.ApiServlet]
> (catalina-exec-10:null) ===START=== 10.181.164.198 -- GET
> command=createSnapshot&volumeid=96f59602-8a84-4731-91a5-218d4f45ee70&response=json&sessionkey=TnWYMJmMIZRly5DfGPNX%2BpZeT%2BI%3D&_=1395645153401
> 2014-03-24 16:12:33,819 DEBUG [cloud.async.AsyncJobManagerImpl]
> (catalina-exec-10:null) submit async job-10 = [
> 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ], details: AsyncJobVO {id:10, userId:
> 2, accountId: 2, sessionKey: null, instanceType: Snapshot, instanceId: 1,
> cmd: org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd,
> cmdOriginator: null, cmdInfo:
> {"id":"1","response":"json","sessionkey":"TnWYMJmMIZRly5DfGPNX+pZeT+I\u003d","cmdEventType":"SNAPSHOT.CREATE","ctxUserId":"2","httpmethod":"GET","_":"1395645153401","volumeid":"96f59602-8a84-4731-91a5-218d4f45ee70","ctxAccountId":"2","ctxStartEventId":"45"},
> cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0,
> processStatus: 0, resultCode: 0, result: null, initMsid: 143618408992,
> completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
> 2014-03-24 16:12:33,820 DEBUG [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Executing
> org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for job-10
> = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]
> 2014-03-24 16:12:33,821 DEBUG [cloud.api.ApiServlet]
> (catalina-exec-10:null) ===END=== 10.181.164.198 -- GET
> command=createSnapshot&volumeid=96f59602-8a84-4731-91a5-218d4f45ee70&response=json&sessionkey=TnWYMJmMIZRly5DfGPNX%2BpZeT%2BI%3D&_=1395645153401
> 2014-03-24 16:12:33,847 INFO [user.snapshot.CreateSnapshotCmd]
> (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) VOLSS:
> createSnapshotCmd starts:1395645153847
> 2014-03-24 16:12:34,179 DEBUG [agent.transport.Request]
> (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Seq
> 1-249561236: Sending { Cmd , MgmtId: 143618408992, via: 1, Ver: v1, Flags:
> 100011,
> [{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"96f59602-8a84-4731-91a5-218d4f45ee70","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8a2181f7-86a0-3d7d-a5be-10caa89b3d27","id":1,"poolType":"NetworkFilesystem","host":"10.181.168.230","path":"/home/primary","port":2049}},"name":"ROOT-3","size":13147648,"path":"f0615600-d806-486e-bbea-d408de0225e4","volumeId":3,"vmName":"i-2-3-VM","accountId":2,"format":"QCOW2","id":3,"hypervisorType":"KVM"},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8a2181f7-86a0-3d7d-a5be-10caa89b3d27","id":1,"poolType":"NetworkFilesystem","host":"10.181.168.230","path":"/home/primary","port":2049}},"vmName":"i-2-3-VM","name":"test1_ROOT-3_20140324071233","hypervisorType":"KVM","id":1}},"wait":0}}]
> }
> 2014-03-24 16:12:34,374 DEBUG [agent.transport.Request]
> (AgentManager-Handler-8:null) Seq 1-249561236: Processing: { Ans: , MgmtId:
> 143618408992, via: 1, Ver: v1, Flags: 10,
> [{"org.apache.cloudstack.storage.command.CreateObjectAnswer":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"/mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4/4d3ea6fa-6902-4037-be67-166e34575ee9","id":0}},"result":true,"wait":0}}]
> }
> 2014-03-24 16:12:34,374 DEBUG [agent.transport.Request]
> (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Seq
> 1-249561236: Received: { Ans: , MgmtId: 143618408992, via: 1, Ver: v1,
> Flags: 10, { CreateObjectAnswer } }
> 2014-03-24 16:12:34,784 DEBUG
> [storage.motion.AncientDataMotionStrategy] (Job-Executor-10:job-10 = [
> 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) copyAsync inspecting src type
> SNAPSHOT copyAsync inspecting dest type SNAPSHOT
> 2014-03-24 16:12:34,802 DEBUG
> [storage.motion.AncientDataMotionStrategy] (Job-Executor-10:job-10 = [
> 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) needCacheStorage true, dest at
> snapshots/2/3 dest role
> Image/mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4/4d3ea6fa-6902-4037-be67-166e34575ee9
> src role Primary
> 2014-03-24 16:12:34,938 DEBUG [agent.transport.Request]
> (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Seq
> 1-249561237: Sending { Cmd , MgmtId: 143618408992, via: 1, Ver: v1, Flags:
> 100111,
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"/mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4/4d3ea6fa-6902-4037-be67-166e34575ee9","volume":{"uuid":"96f59602-8a84-4731-91a5-218d4f45ee70","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8a2181f7-86a0-3d7d-a5be-10caa89b3d27","id":1,"poolType":"NetworkFilesystem","host":"10.181.168.230","path":"/home/primary","port":2049}},"name":"ROOT-3","size":13147648,"path":"f0615600-d806-486e-bbea-d408de0225e4","volumeId":3,"vmName":"i-2-3-VM","accountId":2,"format":"QCOW2","id":3,"hypervisorType":"KVM"},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8a2181f7-86a0-3d7d-a5be-10caa89b3d27","id":1,"poolType":"NetworkFilesystem","host":"10.181.168.230","path":"/home/primary","port":2049}},"vmName":"i-2-3-VM","name":"test1_ROOT-3_20140324071233","hypervisorType":"KVM","id":1}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2/3","volume":{"uuid":"96f59602-8a84-4731-91a5-218d4f45ee70","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8a2181f7-86a0-3d7d-a5be-10caa89b3d27","id":1,"poolType":"NetworkFilesystem","host":"10.181.168.230","path":"/home/primary","port":2049}},"name":"ROOT-3","size":13147648,"path":"f0615600-d806-486e-bbea-d408de0225e4","volumeId":3,"vmName":"i-2-3-VM","accountId":2,"format":"QCOW2","id":3,"hypervisorType":"KVM"},"dataStore":{"com.cloud.agent.api.to.S3TO":{"id":2,"uuid":"c18a1785-f578-4102-bb94-0298289786e0","endPoint":"s3.cloudian.com:18080","bucketName":"cloudstack","httpsFlag":false,"created":"Mar
> 24, 2014 3:39:20
> PM","enableRRS":false,"maxSingleUploadSizeInBytes":5368709120}},"vmName":"i-2-3-VM","name":"test1_ROOT-3_20140324071233","hypervisorType":"KVM","id":1}},"cacheTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2/3","volume":{"uuid":"96f59602-8a84-4731-91a5-218d4f45ee70","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8a2181f7-86a0-3d7d-a5be-10caa89b3d27","id":1,"poolType":"NetworkFilesystem","host":"10.181.168.230","path":"/home/primary","port":2049}},"name":"ROOT-3","size":13147648,"path":"f0615600-d806-486e-bbea-d408de0225e4","volumeId":3,"vmName":"i-2-3-VM","accountId":2,"format":"QCOW2","id":3,"hypervisorType":"KVM"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.181.168.230/home/staging","_role":"ImageCache"}},"vmName":"i-2-3-VM","name":"test1_ROOT-3_20140324071233","hypervisorType":"KVM","id":1}},"executeInSequence":true,"wait":21600}}]
> }
> 2014-03-24 16:12:35,090 DEBUG [agent.transport.Request]
> (AgentManager-Handler-7:null) Seq 1-249561237: Processing: { Ans: , MgmtId:
> 143618408992, via: 1, Ver: v1, Flags: 110,
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"Failed
> to backup 4d3ea6fa-6902-4037-be67-166e34575ee9 for disk
> /mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4
> to /mnt/e5c41d36-90d2-3d5e-bdbf-8a0140dd330e/snapshots/2/3","wait":0}}] }
> 2014-03-24 16:12:35,090 DEBUG [agent.manager.AgentAttache]
> (AgentManager-Handler-7:null) Seq 1-249561237: No more commands found
> 2014-03-24 16:12:35,090 DEBUG [agent.transport.Request]
> (AgentManager-Handler-7:null) Seq 1-249561237: Processing: { Ans: , MgmtId:
> 143618408992, via: 1, Ver: v1, Flags: 110,
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"Failed
> to backup 4d3ea6fa-6902-4037-be67-166e34575ee9 for disk
> /mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4
> to /mnt/e5c41d36-90d2-3d5e-bdbf-8a0140dd330e/snapshots/2/3","wait":0}}] }
> 2014-03-24 16:12:35,090 DEBUG [agent.manager.AgentAttache]
> (AgentManager-Handler-7:null) Seq 1-249561237: No more commands found
> 2014-03-24 16:12:35,090 DEBUG [agent.transport.Request]
> (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Seq
> 1-249561237: Received: { Ans: , MgmtId: 143618408992, via: 1, Ver: v1,
> Flags: 110, { CopyCmdAnswer } }
> 2014-03-24 16:12:35,098 DEBUG [agent.transport.Request]
> (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Seq
> 3-1564409959: Sending { Cmd , MgmtId: 143618408992, via: 3, Ver: v1, Flags:
> 100011,
> [{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2/3","volume":{"uuid":"96f59602-8a84-4731-91a5-218d4f45ee70","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8a2181f7-86a0-3d7d-a5be-10caa89b3d27","id":1,"poolType":"NetworkFilesystem","host":"10.181.168.230","path":"/home/primary","port":2049}},"name":"ROOT-3","size":13147648,"path":"f0615600-d806-486e-bbea-d408de0225e4","volumeId":3,"vmName":"i-2-3-VM","accountId":2,"format":"QCOW2","id":3,"hypervisorType":"KVM"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.181.168.230/home/staging","_role":"ImageCache"}},"vmName":"i-2-3-VM","name":"test1_ROOT-3_20140324071233","hypervisorType":"KVM","id":1}},"wait":0}}]
> }
> 2014-03-24 16:12:35,260 DEBUG [agent.transport.Request]
> (AgentManager-Handler-9:null) Seq 3-1564409959: Processing: { Ans: , MgmtId:
> 143618408992, via: 3, Ver: v1, Flags: 10,
> [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }
> 2014-03-24 16:12:35,260 DEBUG [agent.transport.Request]
> (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Seq
> 3-1564409959: Received: { Ans: , MgmtId: 143618408992, via: 3, Ver: v1,
> Flags: 10, { Answer } }
> 2014-03-24 16:12:35,468 DEBUG [storage.snapshot.SnapshotManagerImpl]
> (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Failed to
> create snapshot
> com.cloud.utils.exception.CloudRuntimeException: Failed to backup
> 4d3ea6fa-6902-4037-be67-166e34575ee9 for disk
> /mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4
> to /mnt/e5c41d36-90d2-3d5e-bdbf-8a0140dd330e/snapshots/2/3
> at
> org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:280)
> at
> org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:136)
> at
> org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:279)
> at
> com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1001)
> at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at
> org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1311)
> at
> com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2779)
> at
> org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170)
> at
> com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
> at
> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> 2014-03-24 16:12:35,635 DEBUG [storage.volume.VolumeServiceImpl]
> (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Take
> snapshot: 3 failed
> com.cloud.utils.exception.CloudRuntimeException: Failed to create
> snapshot
> at
> com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1028)
> at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at
> org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1311)
> at
> com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2779)
> at
> org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170)
> at
> com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
> at
> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to
> backup 4d3ea6fa-6902-4037-be67-166e34575ee9 for disk
> /mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4
> to /mnt/e5c41d36-90d2-3d5e-bdbf-8a0140dd330e/snapshots/2/3
> at
> org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:280)
> at
> org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:136)
> at
> org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:279)
> at
> com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1001)
> ... 16 more
> 2014-03-24 16:12:35,635 DEBUG [user.snapshot.CreateSnapshotCmd]
> (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Failed to
> create snapshot
> org.apache.cloudstack.api.ServerApiException
> at
> org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:176)
> at
> com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
> at
> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> 2014-03-24 16:12:35,637 DEBUG [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Complete
> async job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ], jobStatus: 2,
> resultCode: 530, result: Error Code: 530 Error text: Failed to create
> snapshot due to an internal error creating snapshot for volume 3
>
>
> Thanks,
>
> Tom.
--
Cloudian KK - http://www.cloudian.com/get-started.html
Fancy 10TB of full featured S3 Storage?
Checkout the Cloudian® Community Edition!
