[ https://issues.apache.org/jira/browse/CLOUDSTACK-5469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
edison su resolved CLOUDSTACK-5469. ----------------------------------- Resolution: Fixed It's because, mgt server sends snapshot related commands to wrong kvm host. > Snapshot creation fails with following exception - "Failed to backup > snapshot: qemu-img: Could not delete snapshot > '89eced14-9121-44a7-bb97-26b567795726': -2 (No such file or directory)" > ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > > Key: CLOUDSTACK-5469 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5469 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Management Server > Affects Versions: 4.3.0 > Environment: Build from 4.3 > Reporter: Sangeetha Hariharan > Assignee: edison su > Priority: Critical > Fix For: 4.4.0, 4.5.0 > > Attachments: deletesnapshot.rar > > > Set up: > Advanced Zone with 2 KVM (RHEL 6.3) hosts. > 2 NFS secondary stores set up. > Steps to reproduce the problem: > 1. Deploy 5 Vms in each of the hosts with 10 GB ROOT volume size , so we > start with 10 Vms. > 2. Start concurrent snapshots for ROOT volumes of all the Vms. > 1 of the secondary store -ss1- had the nfs server down for 1 and 1/2 hours. > The other secondary store -ss2 - was always reachable. > Snapshot tasks that went to the ss1 , succeeded after the nfs server was > brought up (It temporarily halted when the the nfs server was down and > resumed when the nsf server was made available). > First set of snapshot tasks that went to the ss2 all succeeded. > But the next hourly snapshot tasks, few of them failed with following > exception: 2013-12-11 16:33:22,427 DEBUG [c.c.s.s.SnapshotManagerImpl] > (Job-Executor-64:ctx-9c70ad77 ctx-3d959fa6) Failed t o create snapshot > com.cloud.utils.exception.CloudRuntimeException: Failed to backup snapshot: > qemu-img: Could not delete snapshot '89eced14-9121-44a7-bb97-26b567795726': > -2 (No such file or directory)Failed to delete snapshot 89eced14-9121-44 > a7-bb97-26b567795726 for path > /mnt/c20ea198-e8ca-33c3-9f11-e361ec9b5532/71a5dce2-da7c-4692-8f25-ba37e5296886 > at > org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:27 > 5) at > org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStra > tegy.java:135) at > org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrate > gy.java:294) at > com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:951) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) at > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocati > on.java:183) at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java: > 150) at > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.ja > va:91) at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java: > 172) at > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) > at $Proxy161.takeSnapshot(Unknown Source) at > org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1341) > at > com.cloud.storage.VolumeApiServiceImpl.takeSnapshot(VolumeApiServiceImpl.java:1461) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) at > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) > Copy to the secondary has succeed. Failure happens after this. > [root@Rack3Host5 118]# ls -ltr > total 10002852 > -rw-r--r--. 1 root root 3637903360 Dec 11 20:33 > 89eced14-9121-44a7-bb97-26b567795726 > -rw-r--r--. 1 root root 3638755328 Dec 11 21:37 > b38d93db-4c14-45a7-9274-639ad95a3f29 > -rw-r--r--. 1 root root 2956619776 Dec 11 22:24 > 452c8841-2025-41da-b6ec-49cea2a49da8 > [root@Rack3Host5 118]# > Following are the volumes which are in "CreatedOnPrimary" state for which the > failure occured. > | 113 | BackedUp | 2013-12-11 20:57:03 | > | 112 | BackedUp | 2013-12-11 20:57:03 | > | 110 | BackedUp | 2013-12-11 20:57:03 | > | 121 | CreatedOnPrimary | 2013-12-11 20:57:04 | > | 118 | CreatedOnPrimary | 2013-12-11 20:57:04 | > | 117 | BackedUp | 2013-12-11 20:57:04 | > | 116 | CreatedOnPrimary | 2013-12-11 20:57:04 | -- This message was sent by Atlassian JIRA (v6.3.4#6332)