Sangeetha Hariharan created CLOUDSTACK-5469:
-----------------------------------------------
Summary: Snapshot creation fails with following exception -
"Failed to backup snapshot: qemu-img: Could not delete snapshot
'89eced14-9121-44a7-bb97-26b567795726': -2 (No such file or directory)"
Key: CLOUDSTACK-5469
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5469
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Management Server
Affects Versions: 4.3.0
Environment: Build from 4.3
Reporter: Sangeetha Hariharan
Priority: Critical
Fix For: 4.3.0
Set up:
Advanced Zone with 2 KVM (RHEL 6.3) hosts.
2 NFS secondary stores set up.
Steps to reproduce the problem:
1. Deploy 5 Vms in each of the hosts with 10 GB ROOT volume size , so we start
with 10 Vms.
2. Start concurrent snapshots for ROOT volumes of all the Vms.
1 of the secondary store -ss1- had the nfs server down for 1 and 1/2 hours.
The other secondary store -ss2 - was always reachable.
Snapshot tasks that went to the ss1 , succeeded after the nfs server was
brought up (It temporarily halted when the the nfs server was down and resumed
when the nsf server was made available).
First set of snapshot tasks that went to the ss2 all succeeded.
But the next hourly snapshot tasks, few of them failed with following
exception: 2013-12-11 16:33:22,427 DEBUG [c.c.s.s.SnapshotManagerImpl]
(Job-Executor-64:ctx-9c70ad77 ctx-3d959fa6) Failed t o create snapshot
com.cloud.utils.exception.CloudRuntimeException: Failed to backup snapshot:
qemu-img: Could not delete snapshot '89eced14-9121-44a7-bb97-26b567795726': -2
(No such file or directory)Failed to delete snapshot 89eced14-9121-44
a7-bb97-26b567795726 for path
/mnt/c20ea198-e8ca-33c3-9f11-e361ec9b5532/71a5dce2-da7c-4692-8f25-ba37e5296886
at
org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:27
5) at
org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStra
tegy.java:135) at
org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrate
gy.java:294) at
com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:951)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601) at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocati
on.java:183) at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:
150) at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.ja
va:91) at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:
172) at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy161.takeSnapshot(Unknown Source) at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1341)
at
com.cloud.storage.VolumeApiServiceImpl.takeSnapshot(VolumeApiServiceImpl.java:1461)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601) at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
Copy to the secondary has succeed. Failure happens after this.
[root@Rack3Host5 118]# ls -ltr
total 10002852
-rw-r--r--. 1 root root 3637903360 Dec 11 20:33
89eced14-9121-44a7-bb97-26b567795726
-rw-r--r--. 1 root root 3638755328 Dec 11 21:37
b38d93db-4c14-45a7-9274-639ad95a3f29
-rw-r--r--. 1 root root 2956619776 Dec 11 22:24
452c8841-2025-41da-b6ec-49cea2a49da8
[root@Rack3Host5 118]#
Following are the volumes which are in "CreatedOnPrimary" state for which the
failure occured.
| 113 | BackedUp | 2013-12-11 20:57:03 |
| 112 | BackedUp | 2013-12-11 20:57:03 |
| 110 | BackedUp | 2013-12-11 20:57:03 |
| 121 | CreatedOnPrimary | 2013-12-11 20:57:04 |
| 118 | CreatedOnPrimary | 2013-12-11 20:57:04 |
| 117 | BackedUp | 2013-12-11 20:57:04 |
| 116 | CreatedOnPrimary | 2013-12-11 20:57:04 |
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
