宿主机是Centos吗? virsh version 把版本信息贴一下 2014-11-07
aix.niuy 发件人:刘宇超 <tanthalas...@hotmail.com> 发送时间:2014-11-07 16:50 主题:Re: 回复:cloudstack4.4.1+KVM高级网络 给虚拟机root磁盘卷创建快照总是出错 收件人:"赵涛"<zhaotao...@qq.com>,"users-cn"<users-cn@cloudstack.apache.org> 抄送: 请问那位大神遇到过? ------------------ 原始邮件 ------------------ 发件人: "赵涛";<zhaotao...@qq.com>; 发送时间: 2014年11月5日(星期三) 中午11:05 收件人: "users-cn"<users-cn@cloudstack.apache.org>; 主题: cloudstack4.4.1+KVM高级网络 给虚拟机root磁盘卷创建快照总是出错 测试环境cloudstack4.4.1+KVM高级网络:管理节点、kvm host都安装CentOS6.3-64位在全局参数里设置了 kvm.snapshot.enabled 值为true,重启了管理节点。但是关闭虚拟机后给虚拟机root磁盘卷创建快照总是出错,cs 通知栏里提示创建成功,查看快照后发现快照显示列表里是Err。请问该如何解决?错误日志如下: 2014-11-05 09:24:21,505 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-bc8070de) Execute sync-queue item: SyncQueueItemVO {id:13, queueId: 11, contentType: AsyncJob, contentId: 73, lastProcessMsid: null, lastprocessNumber: null, lastProcessTime: null, created: Wed Nov 05 09:24:19 CST 2014} 2014-11-05 09:24:21,506 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-bc8070de) Schedule queued job-73 2014-11-05 09:24:21,510 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-8:ctx-8fc5eb5a job-72/job-73) Add job-73 into job monitoring 2014-11-05 09:24:21,510 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-8:ctx-8fc5eb5a job-72/job-73) Executing AsyncJobVO {id:73, userId: 2, accountId: 2, instanceType: null, instanceId: null, cmd: com.cloud.storage.VmWorkTakeVolumeSnapshot, cmdInfo: rO0ABXNyACpjb20uY2xvdWQuc3RvcmFnZS5WbVdvcmtUYWtlVm9sdW1lU25hcHNob3QEvmAbguLVzwIABFoACXF1aWVzY2VWbUwACHBvbGljeUlkdAAQTGphdmEvbGFuZy9Mb25nO0wACnNuYXBzaG90SWRxAH4AAUwACHZvbHVtZUlkcQB-AAF4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1ldAASTGphdmEvbGFuZy9TdHJpbmc7eHAAAAAAAAAAAgAAAAAAAAACAAAAAAAAAAd0ABRWb2x1bWVBcGlTZXJ2aWNlSW1wbABzcgAOamF2YS5sYW5nLkxvbmc7i-SQzI8j3wIAAUoABXZhbHVleHIAEGphdmEubGFuZy5OdW1iZXKGrJUdC5TgiwIAAHhwAAAAAAAAAABzcQB-AAYAAAAAAAAABnEAfgAJ, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 52238016467, completeMsid: null, lastUpdated: null, lastPolled: null, created: Wed Nov 05 09:24:19 CST 2014} 2014-11-05 09:24:21,511 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-8:ctx-8fc5eb5a job-72/job-73) Run VM work job: com.cloud.storage.VmWorkTakeVolumeSnapshot for VM 7, job origin: 72 2014-11-05 09:24:21,512 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-8:ctx-8fc5eb5a job-72/job-73 ctx-b877aaf9) Execute VM work job: com.cloud.storage.VmWorkTakeVolumeSnapshot{"volumeId":6,"policyId":0,"snapshotId":6,"quiesceVm":false,"userId":2,"accountId":2,"vmId":7,"handlerName":"VolumeApiServiceImpl"} 2014-11-05 09:24:21,548 DEBUG [c.c.a.t.Request] (Work-Job-Executor-8:ctx-8fc5eb5a job-72/job-73 ctx-b877aaf9) Seq 1-5995979953890398722: Sending { Cmd , MgmtId: 52238016467, via: 1(node1-kvm), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"e2671633-076c-4d3e-a139-ed3807159312","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"e7a222aa-a542-300f-a7bb-7fb8b24c4bde","id":1,"poolType":"NetworkFilesystem","host":"10.6.26.21","path":"/export/primary","port":2049,"url":"NetworkFilesystem://10.6.26.21/export/primary/?ROLE=Primary&STOREUUID=e7a222aa-a542-300f-a7bb-7fb8b24c4bde"}},"name":"ROOT-7","size":53687091200,"path":"e2671633-076c-4d3e-a139-ed3807159312","volumeId":6,"vmName":"i-2-7-VM","accountId":2,"format":"QCOW2","id":6,"deviceId":0,"hypervisorType":"KVM"},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"e7a222aa-a542-300f-a7bb-7fb8b24c4bde","id":1,"poolType":"NetworkFilesystem","host":"10.6.26.21","path":"/export/primary","port":2049,"url":"NetworkFilesystem://10.6.26.21/export/primary/?ROLE=Primary&STOREUUID=e7a222aa-a542-300f-a7bb-7fb8b24c4bde"}},"vmName":"i-2-7-VM","name":"test03_ROOT-7_20141105012419","hypervisorType":"KVM","id":6,"quiescevm":false,"physicalSize":0}},"wait":0}}] } 2014-11-05 09:24:22,858 DEBUG [c.c.a.ApiServlet] (catalina-exec-23:ctx-2f001f7c) ===START=== 192.168.9.76 -- GET command=queryAsyncJobResult&jobId=b2479cee-d5b5-464a-bae4-6974027f0a67&response=json&sessionkey=tMEAJPWMmcthAjo86aaTwBSDozQ%3D&_=1415150657767 2014-11-05 09:24:22,872 DEBUG [c.c.a.ApiServlet] (catalina-exec-23:ctx-2f001f7c ctx-9fe6010e) ===END=== 192.168.9.76 -- GET command=queryAsyncJobResult&jobId=b2479cee-d5b5-464a-bae4-6974027f0a67&response=json&sessionkey=tMEAJPWMmcthAjo86aaTwBSDozQ%3D&_=1415150657767 2014-11-05 09:24:23,750 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-10:null) SeqA 2-19036: Processing Seq 2-19036: { Cmd , MgmtId: -1, via: 2, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":2,"_loadInfo":"{\n \"connections\": [\n {\n \"id\": 8,\n \"clientInfo\": \"\",\n \"host\": \"10.6.31.4\",\n \"port\": 5905,\n \"tag\": \"a53e9399-e0e5-4df0-b361-1da2380bceef\",\n \"createTime\": 1415150339521,\n \"lastUsedTime\": 1415150583469\n },\n {\n \"id\": 7,\n \"clientInfo\": \"\",\n \"host\": \"10.6.31.4\",\n \"port\": 5904,\n \"tag\": \"93e383ff-3143-4830-b3e5-19d6bd248eaa\",\n \"createTime\": 1415150282559,\n \"lastUsedTime\": 1415150580763\n }\n ]\n}","wait":0}}] } 2014-11-05 09:24:23,754 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-10:null) SeqA 2-19036: Sending Seq 2-19036: { Ans: , MgmtId: 52238016467, via: 2, Ver: v1, Flags: 100010, [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] } 2014-11-05 09:24:25,851 DEBUG [c.c.a.ApiServlet] (catalina-exec-13:ctx-8d3d7287) ===START=== 192.168.9.76 -- GET command=queryAsyncJobResult&jobId=b2479cee-d5b5-464a-bae4-6974027f0a67&response=json&sessionkey=tMEAJPWMmcthAjo86aaTwBSDozQ%3D&_=1415150660759 2014-11-05 09:24:25,869 DEBUG [c.c.a.ApiServlet] (catalina-exec-13:ctx-8d3d7287 ctx-f46c1c3d) ===END=== 192.168.9.76 -- GET command=queryAsyncJobResult&jobId=b2479cee-d5b5-464a-bae4-6974027f0a67&response=json&sessionkey=tMEAJPWMmcthAjo86aaTwBSDozQ%3D&_=1415150660759 2014-11-05 09:24:27,606 DEBUG [c.c.a.t.Request] (AgentManager-Handler-7:null) Seq 1-5995979953890398722: Processing: { Ans: , MgmtId: 52238016467, via: 1, Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CreateObjectAnswer":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"/mnt/e7a222aa-a542-300f-a7bb-7fb8b24c4bde/e2671633-076c-4d3e-a139-ed3807159312/00eb9935-f0a3-4af1-aa92-a39bcb8c060f","id":0,"quiescevm":false,"physicalSize":0}},"result":true,"wait":0}}] } 2014-11-05 09:24:27,607 DEBUG [c.c.a.t.Request] (Work-Job-Executor-8:ctx-8fc5eb5a job-72/job-73 ctx-b877aaf9) Seq 1-5995979953890398722: Received: { Ans: , MgmtId: 52238016467, via: 1, Ver: v1, Flags: 10, { CreateObjectAnswer } } 2014-11-05 09:24:27,640 DEBUG [o.a.c.s.m.AncientDataMotionStrategy] (Work-Job-Executor-8:ctx-8fc5eb5a job-72/job-73 ctx-b877aaf9) copyAsync inspecting src type SNAPSHOT copyAsync inspecting dest type SNAPSHOT 2014-11-05 09:24:27,675 DEBUG [c.c.a.t.Request] (Work-Job-Executor-8:ctx-8fc5eb5a job-72/job-73 ctx-b877aaf9) Seq 1-5995979953890398723: Sending { Cmd , MgmtId: 52238016467, via: 1(node1-kvm), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"/mnt/e7a222aa-a542-300f-a7bb-7fb8b24c4bde/e2671633-076c-4d3e-a139-ed3807159312/00eb9935-f0a3-4af1-aa92-a39bcb8c060f","volume":{"uuid":"e2671633-076c-4d3e-a139-ed3807159312","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"e7a222aa-a542-300f-a7bb-7fb8b24c4bde","id":1,"poolType":"NetworkFilesystem","host":"10.6.26.21","path":"/export/primary","port":2049,"url":"NetworkFilesystem://10.6.26.21/export/primary/?ROLE=Primary&STOREUUID=e7a222aa-a542-300f-a7bb-7fb8b24c4bde"}},"name":"ROOT-7","size":53687091200,"path":"e2671633-076c-4d3e-a139-ed3807159312","volumeId":6,"vmName":"i-2-7-VM","accountId":2,"format":"QCOW2","id":6,"deviceId":0,"hypervisorType":"KVM"},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"e7a222aa-a542-300f-a7bb-7fb8b24c4bde","id":1,"poolType":"NetworkFilesystem","host":"10.6.26.21","path":"/export/primary","port":2049,"url":"NetworkFilesystem://10.6.26.21/export/primary/?ROLE=Primary&STOREUUID=e7a222aa-a542-300f-a7bb-7fb8b24c4bde"}},"vmName":"i-2-7-VM","name":"test03_ROOT-7_20141105012419","hypervisorType":"KVM","id":6,"quiescevm":false,"physicalSize":0}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2/6","volume":{"uuid":"e2671633-076c-4d3e-a139-ed3807159312","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"e7a222aa-a542-300f-a7bb-7fb8b24c4bde","id":1,"poolType":"NetworkFilesystem","host":"10.6.26.21","path":"/export/primary","port":2049,"url":"NetworkFilesystem://10.6.26.21/export/primary/?ROLE=Primary&STOREUUID=e7a222aa-a542-300f-a7bb-7fb8b24c4bde"}},"name":"ROOT-7","size":53687091200,"path":"e2671633-076c-4d3e-a139-ed3807159312","volumeId":6,"vmName":"i-2-7-VM","accountId":2,"format":"QCOW2","id":6,"deviceId":0,"hypervisorType":"KVM"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.6.26.21/export/secondary","_role":"Image"}},"vmName":"i-2-7-VM","name":"test03_ROOT-7_20141105012419","hypervisorType":"KVM","id":6,"quiescevm":false,"physicalSize":0}},"executeInSequence":false,"options":{"fullSnapshot":"true"},"wait":21600}}] } 2014-11-05 09:24:28,034 DEBUG [c.c.a.t.Request] (AgentManager-Handler-9:null) Seq 1-5995979953890398723: Processing: { Ans: , MgmtId: 52238016467, via: 1, Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"Failed to backup 00eb9935-f0a3-4af1-aa92-a39bcb8c060f for disk /mnt/e7a222aa-a542-300f-a7bb-7fb8b24c4bde/e2671633-076c-4d3e-a139-ed3807159312 to /mnt/cb13ca02-b0f4-371a-a676-768194da25f9/snapshots/2/6","wait":0}}] } 2014-11-05 09:24:28,034 DEBUG [c.c.a.t.Request] (Work-Job-Executor-8:ctx-8fc5eb5a job-72/job-73 ctx-b877aaf9) Seq 1-5995979953890398723: Received: { Ans: , MgmtId: 52238016467, via: 1, Ver: v1, Flags: 10, { CopyCmdAnswer } } 2014-11-05 09:24:28,049 DEBUG [c.c.s.s.SnapshotManagerImpl] (Work-Job-Executor-8:ctx-8fc5eb5a job-72/job-73 ctx-b877aaf9) Failed to create snapshot com.cloud.utils.exception.CloudRuntimeException: Failed to backup 00eb9935-f0a3-4af1-aa92-a39bcb8c060f for disk /mnt/e7a222aa-a542-300f-a7bb-7fb8b24c4bde/e2671633-076c-4d3e-a139-ed3807159312 to /mnt/cb13ca02-b0f4-371a-a676-768194da25f9/snapshots/2/6 at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:282) at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:137) at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:300) at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:966) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) at com.sun.proxy.$Proxy179.takeSnapshot(Unknown Source) at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1503) at com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(VolumeApiServiceImpl.java:1766) at com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(VolumeApiServiceImpl.java:2503) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) at com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:2511) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) at com.sun.proxy.$Proxy183.handleVmWorkJob(Unknown Source) at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2014-11-05 09:24:28,058 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-8:ctx-8fc5eb5a job-72/job-73 ctx-b877aaf9) Take snapshot: 6 failed com.cloud.utils.exception.CloudRuntimeException: Failed to create snapshot at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:983) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) at com.sun.proxy.$Proxy179.takeSnapshot(Unknown Source) at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1503) at com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(VolumeApiServiceImpl.java:1766) at com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(VolumeApiServiceImpl.java:2503) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) at com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:2511) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) at com.sun.proxy.$Proxy183.handleVmWorkJob(Unknown Source) at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to backup 00eb9935-f0a3-4af1-aa92-a39bcb8c060f for disk /mnt/e7a222aa-a542-300f-a7bb-7fb8b24c4bde/e2671633-076c-4d3e-a139-ed3807159312 to /mnt/cb13ca02-b0f4-371a-a676-768194da25f9/snapshots/2/6 at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:282) at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:137) at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:300) at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:966) ... 44 more 2014-11-05 09:24:28,058 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-8:ctx-8fc5eb5a job-72/job-73 ctx-b877aaf9) Done executing VM work job: com.cloud.storage.VmWorkTakeVolumeSnapshot{"volumeId":6,"policyId":0,"snapshotId":6,"quiesceVm":false,"userId":2,"accountId":2,"vmId":7,"handlerName":"VolumeApiServiceImpl"} 2014-11-05 09:24:28,058 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-8:ctx-8fc5eb5a job-72/job-73 ctx-b877aaf9) Complete async job-73, jobStatus: SUCCEEDED, resultCode: 0, result: rO0ABXNyAA5qYXZhLmxhbmcuTG9uZzuL5JDMjyPfAgABSgAFdmFsdWV4cgAQamF2YS5sYW5nLk51bWJlcoaslR0LlOCLAgAAeHAAAAAAAAAABg 2014-11-05 09:24:28,063 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-8:ctx-8fc5eb5a job-72/job-73) Done with run of VM work job: com.cloud.storage.VmWorkTakeVolumeSnapshot for VM 7, job origin: 72 2014-11-05 09:24:28,063 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-8:ctx-8fc5eb5a job-72/job-73) Done executing com.cloud.storage.VmWorkTakeVolumeSnapshot for job-73 2014-11-05 09:24:28,070 DEBUG [o.a.c.f.j.i.SyncQueueManagerImpl] (Work-Job-Executor-8:ctx-8fc5eb5a job-72/job-73) Sync queue (11) is currently empty 2014-11-05 09:24:28,071 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-8:ctx-8fc5eb5a job-72/job-73) Remove job-73 from job monitoring