Re: Snapshot exceptions on 4.2.1 with NFS and S3
On 02.04.2014 08:38, Thomas O'Dowd wrote: Anyway - decided to uninstall my package and install the 355 version that you were using. After installing I rebooted. I really didn't expect anything to change but Volume Snapshots now work! YEAH! Glad it works. I'd still recommend my approach, though, otherwise you're left running an old and probably insecure KVM. HTH Lucian -- Sent from the Delta quadrant using Borg technology! Nux! www.nux.ro
Re: Snapshot exceptions on 4.2.1 with NFS and S3
Nux! Thanks for this link. I found this earlier through the list also. I initially discounted it as it seemed to be talking about VM snapshots on CentOS 6.5 and I'm having trouble with Volume snapshots on COS 6.4. However, I checked what version of RPM I had installed: Installed on my COS 6.4 box: - qemu-img-0.12.1.2-2.415.el6_5.6.x86_64 Your linked page shows this version: - qemu-img-0.12.1.2-2.355.el6_4_4.1.x86_64.rpm Hmmm. Different. My RPM looks like a newer version. I had a look at yum info qemu-img and my package comes from 6.4 Updates. Anyway - decided to uninstall my package and install the 355 version that you were using. After installing I rebooted. I really didn't expect anything to change but Volume Snapshots now work! YEAH! Snapshot also goes all the way to S3 so that part is working fine too. Happy days and thanks for your help! Tom. On Wed, 2014-04-02 at 02:08 +0100, Nux! wrote: > On 27.03.2014 02:55, Thomas O'Dowd wrote: > > I remember this used to work on Cloudstack 4.2 when I was testing it > > with devcloud but with CloudPlatform 4.2.1 and KVM I can't get > > snapshots > > to work. > > Might be this > http://www.nux.ro/archive/2014/01/Taking_KVM_volume_snapshots_with_Cloudstack_4_2_on_CentOS_6_5.html > > And update your OS. :) > -- Cloudian KK - http://www.cloudian.com/get-started.html Fancy 10TB of full featured S3 Storage? Checkout the Cloudian® Community Edition!
Re: Snapshot exceptions on 4.2.1 with NFS and S3
On 27.03.2014 02:55, Thomas O'Dowd wrote: I remember this used to work on Cloudstack 4.2 when I was testing it with devcloud but with CloudPlatform 4.2.1 and KVM I can't get snapshots to work. Might be this http://www.nux.ro/archive/2014/01/Taking_KVM_volume_snapshots_with_Cloudstack_4_2_on_CentOS_6_5.html And update your OS. :) -- Sent from the Delta quadrant using Borg technology! Nux! www.nux.ro
Re: Snapshot exceptions on 4.2.1 with NFS and S3
Thanks for the feedback Andrei, In my case the problem seems unrelated to S3 as when I configure NFS as my secondary storage I get the same problem. My problem seems to be that the system tries to write the snapshot to directory path like /mnt/e5c41.../snapshots/2/3/ but as for some reason it finds that /mnt/e5c41.../snapshots/2/3/ doesn't exist as the /mnt/e5c41... is not actually mounting secondary storage at that time. If I mount it manually, the path is there but the system is not doing it automatically for me. Tom. On Tue, 2014-04-01 at 22:41 +0100, Andrei Mikhailovsky wrote: > From what I remember, there was an issue with S3 and snapshots. > > I might be mixing version numbers, but it could be related to the size of the > image. In the past, there was no support for multi-part upload I was not able > to snapshot anything larger than 5gb. Not sure if this is the same problem > that you are experiencing. > > I would suggest trying 4.3 and letting us know how it works for you. > > I am planning to upgrade to 4.3 and test snapshotting with nfs / s3 / ceph. > Will update on the progress. > > Andrei > > > - Original Message - > From: "Thomas O'Dowd" > To: users@cloudstack.apache.org > Sent: Tuesday, 1 April, 2014 6:44:28 AM > Subject: Re: Snapshot exceptions on 4.2.1 with NFS and S3 > > Anyone any hints on this? I've installed it a few times and always have > the same issue. Wondering if it's some configuration problem or a known > bug. > > Thanks, > > Tom. > > On Thu, 2014-03-27 at 11:55 +0900, Thomas O'Dowd wrote: > > I remember this used to work on Cloudstack 4.2 when I was testing it > > with devcloud but with CloudPlatform 4.2.1 and KVM I can't get snapshots > > to work. > > > > I've setup a single node using KVM and management server, db and agent. > > > > I can start an instance using a template stored on NFS secondary > > storage. If I stop the instance and goto storage, I can see it's disk > > and then I can select "take snapshot". This always seems to fail for me > > and I'm not sure why. > > > > The error seems to be: > > Failed to create snapshot > > com.cloud.utils.exception.CloudRuntimeException: Failed to > > backup 4d3ea6fa-6902-4037-be67-166e34575ee9 for > > disk > > /mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4 > > to /mnt/e5c41d36-90d2-3d5e-bdbf-8a0140dd330e/snapshots/2/3 > > > > /mnt/8a218... is mounted fine if I look at df. > > /mnt/e5c41... is not mounted. > > > > If I browse to /mnt, the e5c41... directory (used for mounting) exists. > > If I manually mount it, I can see the > > directory /mnt/e5c41.../snapshots/2/3 so it exists. If I leave it > > mounted and try to take a snapshot again, it fails again with the same > > error and again when I check the /mnt/e5c41... directory is not mounted. > > > > As I mentioned above, the NFS server I'm using is actually the same > > node. I'm using KVM but I don't think that is causing the issue? I'm > > using CentOS 6.4. > > > > If I setup S3 as secondary storage and use NFS as the staging server, I > > get the same error. Any hints on what I am doing wrong is greatly > > appreciated. > > > > Here is more of the exception log: > > 2014-03-24 16:12:33,450 DEBUG [cloud.api.ApiServlet] > > (catalina-exec-10:null) ===START=== 10.181.164.198 -- GET > > command=createSnapshot&volumeid=96f59602-8a84-4731-91a5-218d4f45ee70&response=json&sessionkey=TnWYMJmMIZRly5DfGPNX%2BpZeT%2BI%3D&_=1395645153401 > > 2014-03-24 16:12:33,819 DEBUG [cloud.async.AsyncJobManagerImpl] > > (catalina-exec-10:null) submit async job-10 = [ > > 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ], details: AsyncJobVO {id:10, userId: > > 2, accountId: 2, sessionKey: null, instanceType: Snapshot, instanceId: 1, > > cmd: org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd, > > cmdOriginator: null, cmdInfo: > > {"id":"1","response":"json","sessionkey":"TnWYMJmMIZRly5DfGPNX+pZeT+I\u003d","cmdEventType":"SNAPSHOT.CREATE","ctxUserId":"2","httpmethod":"GET","_":"1395645153401","volumeid":"96f59602-8a84-4731-91a5-218d4f45ee70","ctxAccountId":"2","ctxStartEventId":"45"}, > > cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0, > > processStatus: 0, resultCode:
Re: Snapshot exceptions on 4.2.1 with NFS and S3
>From what I remember, there was an issue with S3 and snapshots. I might be mixing version numbers, but it could be related to the size of the image. In the past, there was no support for multi-part upload I was not able to snapshot anything larger than 5gb. Not sure if this is the same problem that you are experiencing. I would suggest trying 4.3 and letting us know how it works for you. I am planning to upgrade to 4.3 and test snapshotting with nfs / s3 / ceph. Will update on the progress. Andrei - Original Message - From: "Thomas O'Dowd" To: users@cloudstack.apache.org Sent: Tuesday, 1 April, 2014 6:44:28 AM Subject: Re: Snapshot exceptions on 4.2.1 with NFS and S3 Anyone any hints on this? I've installed it a few times and always have the same issue. Wondering if it's some configuration problem or a known bug. Thanks, Tom. On Thu, 2014-03-27 at 11:55 +0900, Thomas O'Dowd wrote: > I remember this used to work on Cloudstack 4.2 when I was testing it > with devcloud but with CloudPlatform 4.2.1 and KVM I can't get snapshots > to work. > > I've setup a single node using KVM and management server, db and agent. > > I can start an instance using a template stored on NFS secondary > storage. If I stop the instance and goto storage, I can see it's disk > and then I can select "take snapshot". This always seems to fail for me > and I'm not sure why. > > The error seems to be: > Failed to create snapshot > com.cloud.utils.exception.CloudRuntimeException: Failed to > backup 4d3ea6fa-6902-4037-be67-166e34575ee9 for > disk > /mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4 > to /mnt/e5c41d36-90d2-3d5e-bdbf-8a0140dd330e/snapshots/2/3 > > /mnt/8a218... is mounted fine if I look at df. > /mnt/e5c41... is not mounted. > > If I browse to /mnt, the e5c41... directory (used for mounting) exists. > If I manually mount it, I can see the > directory /mnt/e5c41.../snapshots/2/3 so it exists. If I leave it > mounted and try to take a snapshot again, it fails again with the same > error and again when I check the /mnt/e5c41... directory is not mounted. > > As I mentioned above, the NFS server I'm using is actually the same > node. I'm using KVM but I don't think that is causing the issue? I'm > using CentOS 6.4. > > If I setup S3 as secondary storage and use NFS as the staging server, I > get the same error. Any hints on what I am doing wrong is greatly > appreciated. > > Here is more of the exception log: > 2014-03-24 16:12:33,450 DEBUG [cloud.api.ApiServlet] > (catalina-exec-10:null) ===START=== 10.181.164.198 -- GET > command=createSnapshot&volumeid=96f59602-8a84-4731-91a5-218d4f45ee70&response=json&sessionkey=TnWYMJmMIZRly5DfGPNX%2BpZeT%2BI%3D&_=1395645153401 > 2014-03-24 16:12:33,819 DEBUG [cloud.async.AsyncJobManagerImpl] > (catalina-exec-10:null) submit async job-10 = [ > 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ], details: AsyncJobVO {id:10, userId: > 2, accountId: 2, sessionKey: null, instanceType: Snapshot, instanceId: 1, > cmd: org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd, > cmdOriginator: null, cmdInfo: > {"id":"1","response":"json","sessionkey":"TnWYMJmMIZRly5DfGPNX+pZeT+I\u003d","cmdEventType":"SNAPSHOT.CREATE","ctxUserId":"2","httpmethod":"GET","_":"1395645153401","volumeid":"96f59602-8a84-4731-91a5-218d4f45ee70","ctxAccountId":"2","ctxStartEventId":"45"}, > cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0, > processStatus: 0, resultCode: 0, result: null, initMsid: 143618408992, > completeMsid: null, lastUpdated: null, lastPolled: null, created: null} > 2014-03-24 16:12:33,820 DEBUG [cloud.async.AsyncJobManagerImpl] > (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Executing > org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for job-10 > = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ] > 2014-03-24 16:12:33,821 DEBUG [cloud.api.ApiServlet] > (catalina-exec-10:null) ===END=== 10.181.164.198 -- GET > command=createSnapshot&volumeid=96f59602-8a84-4731-91a5-218d4f45ee70&response=json&sessionkey=TnWYMJmMIZRly5DfGPNX%2BpZeT%2BI%3D&_=1395645153401 > 2014-03-24 16:12:33,847 INFO [user.snapshot.CreateSnapshotCmd] > (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) VOLSS: > createSnapshotCmd starts:1395645153847 > 2014-03-24 16:12:34,179 DEBUG [agent.transport.Request]
Re: Snapshot exceptions on 4.2.1 with NFS and S3
Anyone any hints on this? I've installed it a few times and always have the same issue. Wondering if it's some configuration problem or a known bug. Thanks, Tom. On Thu, 2014-03-27 at 11:55 +0900, Thomas O'Dowd wrote: > I remember this used to work on Cloudstack 4.2 when I was testing it > with devcloud but with CloudPlatform 4.2.1 and KVM I can't get snapshots > to work. > > I've setup a single node using KVM and management server, db and agent. > > I can start an instance using a template stored on NFS secondary > storage. If I stop the instance and goto storage, I can see it's disk > and then I can select "take snapshot". This always seems to fail for me > and I'm not sure why. > > The error seems to be: > Failed to create snapshot > com.cloud.utils.exception.CloudRuntimeException: Failed to > backup 4d3ea6fa-6902-4037-be67-166e34575ee9 for > disk > /mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4 > to /mnt/e5c41d36-90d2-3d5e-bdbf-8a0140dd330e/snapshots/2/3 > > /mnt/8a218... is mounted fine if I look at df. > /mnt/e5c41... is not mounted. > > If I browse to /mnt, the e5c41... directory (used for mounting) exists. > If I manually mount it, I can see the > directory /mnt/e5c41.../snapshots/2/3 so it exists. If I leave it > mounted and try to take a snapshot again, it fails again with the same > error and again when I check the /mnt/e5c41... directory is not mounted. > > As I mentioned above, the NFS server I'm using is actually the same > node. I'm using KVM but I don't think that is causing the issue? I'm > using CentOS 6.4. > > If I setup S3 as secondary storage and use NFS as the staging server, I > get the same error. Any hints on what I am doing wrong is greatly > appreciated. > > Here is more of the exception log: > 2014-03-24 16:12:33,450 DEBUG [cloud.api.ApiServlet] > (catalina-exec-10:null) ===START=== 10.181.164.198 -- GET > command=createSnapshot&volumeid=96f59602-8a84-4731-91a5-218d4f45ee70&response=json&sessionkey=TnWYMJmMIZRly5DfGPNX%2BpZeT%2BI%3D&_=1395645153401 > 2014-03-24 16:12:33,819 DEBUG [cloud.async.AsyncJobManagerImpl] > (catalina-exec-10:null) submit async job-10 = [ > 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ], details: AsyncJobVO {id:10, userId: > 2, accountId: 2, sessionKey: null, instanceType: Snapshot, instanceId: 1, > cmd: org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd, > cmdOriginator: null, cmdInfo: > {"id":"1","response":"json","sessionkey":"TnWYMJmMIZRly5DfGPNX+pZeT+I\u003d","cmdEventType":"SNAPSHOT.CREATE","ctxUserId":"2","httpmethod":"GET","_":"1395645153401","volumeid":"96f59602-8a84-4731-91a5-218d4f45ee70","ctxAccountId":"2","ctxStartEventId":"45"}, > cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0, > processStatus: 0, resultCode: 0, result: null, initMsid: 143618408992, > completeMsid: null, lastUpdated: null, lastPolled: null, created: null} > 2014-03-24 16:12:33,820 DEBUG [cloud.async.AsyncJobManagerImpl] > (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Executing > org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for job-10 > = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ] > 2014-03-24 16:12:33,821 DEBUG [cloud.api.ApiServlet] > (catalina-exec-10:null) ===END=== 10.181.164.198 -- GET > command=createSnapshot&volumeid=96f59602-8a84-4731-91a5-218d4f45ee70&response=json&sessionkey=TnWYMJmMIZRly5DfGPNX%2BpZeT%2BI%3D&_=1395645153401 > 2014-03-24 16:12:33,847 INFO [user.snapshot.CreateSnapshotCmd] > (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) VOLSS: > createSnapshotCmd starts:1395645153847 > 2014-03-24 16:12:34,179 DEBUG [agent.transport.Request] > (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Seq > 1-249561236: Sending { Cmd , MgmtId: 143618408992, via: 1, Ver: v1, Flags: > 100011, > [{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"96f59602-8a84-4731-91a5-218d4f45ee70","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8a2181f7-86a0-3d7d-a5be-10caa89b3d27","id":1,"poolType":"NetworkFilesystem","host":"10.181.168.230","path":"/home/primary","port":2049}},"name":"ROOT-3","size":13147648,"path":"f0615600-d806-486e-bbea-d408de0225e4","volumeId":3,"vmName":"i-2-3-VM","accountId":2,"format":"QCOW2","id":3,"hypervisorType":"KVM"},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8a2181f7-86a0-3d7d-a5be-10caa89b3d27","id":1,"poolType":"NetworkFilesystem","host":"10.181.168.230","path":"/home/primary","port":2049}},"vmName":"i-2-3-VM","name":"test1_ROOT-3_20140324071233","hypervisorType":"KVM","id":1}},"wait":0}}] > } > 2014-03-24 16:12:34,374 DEBUG [agent.transport.Request] > (AgentManager-Handler-8:null) Seq 1-249561236: Processing: { Ans: , MgmtI
Snapshot exceptions on 4.2.1 with NFS and S3
I remember this used to work on Cloudstack 4.2 when I was testing it with devcloud but with CloudPlatform 4.2.1 and KVM I can't get snapshots to work. I've setup a single node using KVM and management server, db and agent. I can start an instance using a template stored on NFS secondary storage. If I stop the instance and goto storage, I can see it's disk and then I can select "take snapshot". This always seems to fail for me and I'm not sure why. The error seems to be: Failed to create snapshot com.cloud.utils.exception.CloudRuntimeException: Failed to backup 4d3ea6fa-6902-4037-be67-166e34575ee9 for disk /mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4 to /mnt/e5c41d36-90d2-3d5e-bdbf-8a0140dd330e/snapshots/2/3 /mnt/8a218... is mounted fine if I look at df. /mnt/e5c41... is not mounted. If I browse to /mnt, the e5c41... directory (used for mounting) exists. If I manually mount it, I can see the directory /mnt/e5c41.../snapshots/2/3 so it exists. If I leave it mounted and try to take a snapshot again, it fails again with the same error and again when I check the /mnt/e5c41... directory is not mounted. As I mentioned above, the NFS server I'm using is actually the same node. I'm using KVM but I don't think that is causing the issue? I'm using CentOS 6.4. If I setup S3 as secondary storage and use NFS as the staging server, I get the same error. Any hints on what I am doing wrong is greatly appreciated. Here is more of the exception log: 2014-03-24 16:12:33,450 DEBUG [cloud.api.ApiServlet] (catalina-exec-10:null) ===START=== 10.181.164.198 -- GET command=createSnapshot&volumeid=96f59602-8a84-4731-91a5-218d4f45ee70&response=json&sessionkey=TnWYMJmMIZRly5DfGPNX%2BpZeT%2BI%3D&_=1395645153401 2014-03-24 16:12:33,819 DEBUG [cloud.async.AsyncJobManagerImpl] (catalina-exec-10:null) submit async job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ], details: AsyncJobVO {id:10, userId: 2, accountId: 2, sessionKey: null, instanceType: Snapshot, instanceId: 1, cmd: org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd, cmdOriginator: null, cmdInfo: {"id":"1","response":"json","sessionkey":"TnWYMJmMIZRly5DfGPNX+pZeT+I\u003d","cmdEventType":"SNAPSHOT.CREATE","ctxUserId":"2","httpmethod":"GET","_":"1395645153401","volumeid":"96f59602-8a84-4731-91a5-218d4f45ee70","ctxAccountId":"2","ctxStartEventId":"45"}, cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0, processStatus: 0, resultCode: 0, result: null, initMsid: 143618408992, completeMsid: null, lastUpdated: null, lastPolled: null, created: null} 2014-03-24 16:12:33,820 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Executing org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ] 2014-03-24 16:12:33,821 DEBUG [cloud.api.ApiServlet] (catalina-exec-10:null) ===END=== 10.181.164.198 -- GET command=createSnapshot&volumeid=96f59602-8a84-4731-91a5-218d4f45ee70&response=json&sessionkey=TnWYMJmMIZRly5DfGPNX%2BpZeT%2BI%3D&_=1395645153401 2014-03-24 16:12:33,847 INFO [user.snapshot.CreateSnapshotCmd] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) VOLSS: createSnapshotCmd starts:1395645153847 2014-03-24 16:12:34,179 DEBUG [agent.transport.Request] (Job-Executor-10:job-10 = [ 4f6ffdef-f66d-405c-81cc-8a09cb39f5da ]) Seq 1-249561236: Sending { Cmd , MgmtId: 143618408992, via: 1, Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"96f59602-8a84-4731-91a5-218d4f45ee70","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8a2181f7-86a0-3d7d-a5be-10caa89b3d27","id":1,"poolType":"NetworkFilesystem","host":"10.181.168.230","path":"/home/primary","port":2049}},"name":"ROOT-3","size":13147648,"path":"f0615600-d806-486e-bbea-d408de0225e4","volumeId":3,"vmName":"i-2-3-VM","accountId":2,"format":"QCOW2","id":3,"hypervisorType":"KVM"},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"8a2181f7-86a0-3d7d-a5be-10caa89b3d27","id":1,"poolType":"NetworkFilesystem","host":"10.181.168.230","path":"/home/primary","port":2049}},"vmName":"i-2-3-VM","name":"test1_ROOT-3_20140324071233","hypervisorType":"KVM","id":1}},"wait":0}}] } 2014-03-24 16:12:34,374 DEBUG [agent.transport.Request] (AgentManager-Handler-8:null) Seq 1-249561236: Processing: { Ans: , MgmtId: 143618408992, via: 1, Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CreateObjectAnswer":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"/mnt/8a2181f7-86a0-3d7d-a5be-10caa89b3d27/f0615600-d806-486e-bbea-d408de0225e4/4d3ea6fa-6902-4037-be67-166e34575ee9","id":0}},"result":true,"wait":0}}] } 2014-03-24 16:12