I remember this used to work on Cloudstack 4.2 when I was testing it
with devcloud but with CloudPlatform 4.2.1 and KVM I can't get snapshots
to work.
I've setup a single node using KVM and management server, db and agent.
I can start an instance using a template stored on NFS secondary
storage. If I stop the instance and goto storage, I can see it's disk
and then I can select "take snapshot". This always seems to fail for me
and I'm not sure why.
The error seems to be:
Failed to create snapshot
com.cloud.utils.exception.CloudRuntimeException: Failed to
backup 4d3ea6fa-6902-4037-be67-166e34575ee9 for
disk
/mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4
to /mnt/e5c41d36-90d2-3d5e-bdbf-8a0140dd330e/snapshots/2/3
/mnt/8a218... is mounted fine if I look at df.
/mnt/e5c41... is not mounted.
If I browse to /mnt, the e5c41... directory (used for mounting) exists.
If I manually mount it, I can see the
directory /mnt/e5c41.../snapshots/2/3 so it exists. If I leave it
mounted and try to take a snapshot again, it fails again with the same
error and again when I check the /mnt/e5c41... directory is not mounted.
As I mentioned above, the NFS server I'm using is actually the same
node. I'm using KVM but I don't think that is causing the issue? I'm
using CentOS 6.4.
If I setup S3 as secondary storage and use NFS as the staging server, I
get the same error. Any hints on what I am doing wrong is greatly
appreciated.
Here is more of the exception log:
2014-03-24 16:12:33,450 DEBUG [cloud.api.ApiServlet]
(catalina-exec-10:null) ===START=== 10.181.164.198 -- GET
command=createSnapshot&volumeid=96f59602-8a84-4731-91a5-218d4f45ee70&response=json&sessionkey=TnWYMJmMIZRly5DfGPNX%2BpZeT%2BI%3D&_=1395645153401
2014-03-24 16:12:33,819 DEBUG [cloud.async.AsyncJobManagerImpl]
(catalina-exec-10:null) submit async job-10 = [
4f6ffdef-f66d-405c-81cc-8a09cb39f5da ], details: AsyncJobVO {id:10, userId: 2,
accountId: 2, sessionKey: null, instanceType: Snapshot, instanceId: 1, cmd:
org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd,
cmdOriginator: null, cmdInfo:
{"id":"1","response":"json","sessionkey":"TnWYMJmMIZRly5DfGPNX+pZeT+I\u003d","cmdEventType":"SNAPSHOT.CREATE","ctxUserId":"2","httpmethod":"GET","_":"1395645153401","volumeid":"96f59602-8a84-4731-91a5-218d4f45ee70","ctxAccountId":"2","ctxStartEventId":"45"},
cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0,
processStatus: 0, resultCode: 0, result: null, initMsid: 143618408992,
completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2014-03-24 16:12:33,820 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Executing
org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for job-10 =
[ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]
2014-03-24 16:12:33,821 DEBUG [cloud.api.ApiServlet]
(catalina-exec-10:null) ===END=== 10.181.164.198 -- GET
command=createSnapshot&volumeid=96f59602-8a84-4731-91a5-218d4f45ee70&response=json&sessionkey=TnWYMJmMIZRly5DfGPNX%2BpZeT%2BI%3D&_=1395645153401
2014-03-24 16:12:33,847 INFO [user.snapshot.CreateSnapshotCmd]
(Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) VOLSS:
createSnapshotCmd starts:1395645153847
2014-03-24 16:12:34,179 DEBUG [agent.transport.Request]
(Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Seq
1-249561236: Sending { Cmd , MgmtId: 143618408992, via: 1, Ver: v1, Flags:
100011,
[{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"96f59602-8a84-4731-91a5-218d4f45ee70","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8a2181f7-86a0-3d7d-a5be-10caa89b3d27","id":1,"poolType":"NetworkFilesystem","host":"10.181.168.230","path":"/home/primary","port":2049}},"name":"ROOT-3","size":13147648,"path":"f0615600-d806-486e-bbea-d408de0225e4","volumeId":3,"vmName":"i-2-3-VM","accountId":2,"format":"QCOW2","id":3,"hypervisorType":"KVM"},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8a2181f7-86a0-3d7d-a5be-10caa89b3d27","id":1,"poolType":"NetworkFilesystem","host":"10.181.168.230","path":"/home/primary","port":2049}},"vmName":"i-2-3-VM","name":"test1_ROOT-3_20140324071233","hypervisorType":"KVM","id":1}},"wait":0}}]
}
2014-03-24 16:12:34,374 DEBUG [agent.transport.Request]
(AgentManager-Handler-8:null) Seq 1-249561236: Processing: { Ans: , MgmtId:
143618408992, via: 1, Ver: v1, Flags: 10,
[{"org.apache.cloudstack.storage.command.CreateObjectAnswer":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"/mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4/4d3ea6fa-6902-4037-be67-166e34575ee9","id":0}},"result":true,"wait":0}}]
}
2014-03-24 16:12:34,374 DEBUG [agent.transport.Request]
(Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Seq
1-249561236: Received: { Ans: , MgmtId: 143618408992, via: 1, Ver: v1, Flags:
10, { CreateObjectAnswer } }
2014-03-24 16:12:34,784 DEBUG
[storage.motion.AncientDataMotionStrategy] (Job-Executor-10:job-10 = [
4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) copyAsync inspecting src type SNAPSHOT
copyAsync inspecting dest type SNAPSHOT
2014-03-24 16:12:34,802 DEBUG
[storage.motion.AncientDataMotionStrategy] (Job-Executor-10:job-10 = [
4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) needCacheStorage true, dest at
snapshots/2/3 dest role
Image/mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4/4d3ea6fa-6902-4037-be67-166e34575ee9
src role Primary
2014-03-24 16:12:34,938 DEBUG [agent.transport.Request]
(Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Seq
1-249561237: Sending { Cmd , MgmtId: 143618408992, via: 1, Ver: v1, Flags:
100111,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"/mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4/4d3ea6fa-6902-4037-be67-166e34575ee9","volume":{"uuid":"96f59602-8a84-4731-91a5-218d4f45ee70","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8a2181f7-86a0-3d7d-a5be-10caa89b3d27","id":1,"poolType":"NetworkFilesystem","host":"10.181.168.230","path":"/home/primary","port":2049}},"name":"ROOT-3","size":13147648,"path":"f0615600-d806-486e-bbea-d408de0225e4","volumeId":3,"vmName":"i-2-3-VM","accountId":2,"format":"QCOW2","id":3,"hypervisorType":"KVM"},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8a2181f7-86a0-3d7d-a5be-10caa89b3d27","id":1,"poolType":"NetworkFilesystem","host":"10.181.168.230","path":"/home/primary","port":2049}},"vmName":"i-2-3-VM","name":"test1_ROOT-3_20140324071233","hypervisorType":"KVM","id":1}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2/3","volume":{"uuid":"96f59602-8a84-4731-91a5-218d4f45ee70","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8a2181f7-86a0-3d7d-a5be-10caa89b3d27","id":1,"poolType":"NetworkFilesystem","host":"10.181.168.230","path":"/home/primary","port":2049}},"name":"ROOT-3","size":13147648,"path":"f0615600-d806-486e-bbea-d408de0225e4","volumeId":3,"vmName":"i-2-3-VM","accountId":2,"format":"QCOW2","id":3,"hypervisorType":"KVM"},"dataStore":{"com.cloud.agent.api.to.S3TO":{"id":2,"uuid":"c18a1785-f578-4102-bb94-0298289786e0","endPoint":"s3.cloudian.com:18080","bucketName":"cloudstack","httpsFlag":false,"created":"Mar
24, 2014 3:39:20
PM","enableRRS":false,"maxSingleUploadSizeInBytes":5368709120}},"vmName":"i-2-3-VM","name":"test1_ROOT-3_20140324071233","hypervisorType":"KVM","id":1}},"cacheTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2/3","volume":{"uuid":"96f59602-8a84-4731-91a5-218d4f45ee70","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8a2181f7-86a0-3d7d-a5be-10caa89b3d27","id":1,"poolType":"NetworkFilesystem","host":"10.181.168.230","path":"/home/primary","port":2049}},"name":"ROOT-3","size":13147648,"path":"f0615600-d806-486e-bbea-d408de0225e4","volumeId":3,"vmName":"i-2-3-VM","accountId":2,"format":"QCOW2","id":3,"hypervisorType":"KVM"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.181.168.230/home/staging","_role":"ImageCache"}},"vmName":"i-2-3-VM","name":"test1_ROOT-3_20140324071233","hypervisorType":"KVM","id":1}},"executeInSequence":true,"wait":21600}}]
}
2014-03-24 16:12:35,090 DEBUG [agent.transport.Request]
(AgentManager-Handler-7:null) Seq 1-249561237: Processing: { Ans: , MgmtId:
143618408992, via: 1, Ver: v1, Flags: 110,
[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"Failed
to backup 4d3ea6fa-6902-4037-be67-166e34575ee9 for disk
/mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4
to /mnt/e5c41d36-90d2-3d5e-bdbf-8a0140dd330e/snapshots/2/3","wait":0}}] }
2014-03-24 16:12:35,090 DEBUG [agent.manager.AgentAttache]
(AgentManager-Handler-7:null) Seq 1-249561237: No more commands found
2014-03-24 16:12:35,090 DEBUG [agent.transport.Request]
(AgentManager-Handler-7:null) Seq 1-249561237: Processing: { Ans: , MgmtId:
143618408992, via: 1, Ver: v1, Flags: 110,
[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"Failed
to backup 4d3ea6fa-6902-4037-be67-166e34575ee9 for disk
/mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4
to /mnt/e5c41d36-90d2-3d5e-bdbf-8a0140dd330e/snapshots/2/3","wait":0}}] }
2014-03-24 16:12:35,090 DEBUG [agent.manager.AgentAttache]
(AgentManager-Handler-7:null) Seq 1-249561237: No more commands found
2014-03-24 16:12:35,090 DEBUG [agent.transport.Request]
(Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Seq
1-249561237: Received: { Ans: , MgmtId: 143618408992, via: 1, Ver: v1, Flags:
110, { CopyCmdAnswer } }
2014-03-24 16:12:35,098 DEBUG [agent.transport.Request]
(Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Seq
3-1564409959: Sending { Cmd , MgmtId: 143618408992, via: 3, Ver: v1, Flags:
100011,
[{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2/3","volume":{"uuid":"96f59602-8a84-4731-91a5-218d4f45ee70","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8a2181f7-86a0-3d7d-a5be-10caa89b3d27","id":1,"poolType":"NetworkFilesystem","host":"10.181.168.230","path":"/home/primary","port":2049}},"name":"ROOT-3","size":13147648,"path":"f0615600-d806-486e-bbea-d408de0225e4","volumeId":3,"vmName":"i-2-3-VM","accountId":2,"format":"QCOW2","id":3,"hypervisorType":"KVM"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.181.168.230/home/staging","_role":"ImageCache"}},"vmName":"i-2-3-VM","name":"test1_ROOT-3_20140324071233","hypervisorType":"KVM","id":1}},"wait":0}}]
}
2014-03-24 16:12:35,260 DEBUG [agent.transport.Request]
(AgentManager-Handler-9:null) Seq 3-1564409959: Processing: { Ans: , MgmtId:
143618408992, via: 3, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }
2014-03-24 16:12:35,260 DEBUG [agent.transport.Request]
(Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Seq
3-1564409959: Received: { Ans: , MgmtId: 143618408992, via: 3, Ver: v1, Flags:
10, { Answer } }
2014-03-24 16:12:35,468 DEBUG [storage.snapshot.SnapshotManagerImpl]
(Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Failed to
create snapshot
com.cloud.utils.exception.CloudRuntimeException: Failed to backup
4d3ea6fa-6902-4037-be67-166e34575ee9 for disk
/mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4
to /mnt/e5c41d36-90d2-3d5e-bdbf-8a0140dd330e/snapshots/2/3
at
org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:280)
at
org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:136)
at
org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:279)
at
com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1001)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1311)
at
com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2779)
at
org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at
com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
2014-03-24 16:12:35,635 DEBUG [storage.volume.VolumeServiceImpl]
(Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Take
snapshot: 3 failed
com.cloud.utils.exception.CloudRuntimeException: Failed to create
snapshot
at
com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1028)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1311)
at
com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2779)
at
org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at
com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to
backup 4d3ea6fa-6902-4037-be67-166e34575ee9 for disk
/mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4
to /mnt/e5c41d36-90d2-3d5e-bdbf-8a0140dd330e/snapshots/2/3
at
org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:280)
at
org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:136)
at
org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:279)
at
com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1001)
... 16 more
2014-03-24 16:12:35,635 DEBUG [user.snapshot.CreateSnapshotCmd]
(Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Failed to
create snapshot
org.apache.cloudstack.api.ServerApiException
at
org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:176)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at
com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
2014-03-24 16:12:35,637 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Complete
async job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ], jobStatus: 2,
resultCode: 530, result: Error Code: 530 Error text: Failed to create snapshot
due to an internal error creating snapshot for volume 3
Thanks,
Tom.
--
Cloudian KK - http://www.cloudian.com/get-started.html
Fancy 10TB of full featured S3 Storage?
Checkout the Cloudian® Community Edition!
