[ https://issues.apache.org/jira/browse/CLOUDSTACK-5395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sangeetha Hariharan updated CLOUDSTACK-5395: -------------------------------------------- Priority: Major (was: Critical) > When backup snapshot fails becasue of backup.snapshot.wait time exceeding , > the vhd entries form the primary store is not getting cleared. > ------------------------------------------------------------------------------------------------------------------------------------------ > > Key: CLOUDSTACK-5395 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5395 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Management Server > Affects Versions: 4.3.0 > Environment: Build from 4.3 > Reporter: Sangeetha Hariharan > Assignee: edison su > Fix For: 4.3.0 > > Attachments: 1205logs.rar > > > Steps to reproduce the problem: > Deploy 5 Vms in each of the hosts with 10 GB , so we start with 10 Vms. > We will be constantly writing to the ROOT volume. > Change the backup.snapshot.wait to 10 mts and restart management server. > Start concurrent snapshots for ROOT volumes of all the Vms. > After 10 mts , the snapshots fail. They are present in the database in > "CreatedOnPrimary" state. > Vhd entries from the primary store fails to be cleaned up once the > backsnapshpot job fails. > In this case ,API fails with - "Failed to create snapshot due to an internal > error creating snapshot for volume 81" . We should be able to present a more > meaningful error message in this case. > Expected Behavior: > We should be able to clean up the Vhd entries from the primary store when > the backsnapshpot job fails. > In such cases , instead of > select * from snapshot_store_ref; > 702 | 1 | 355 | 2013-12-06 01:25:43 | NULL | NULL | > Primary | 0 | 0 | 0 | > 2eedb23e-6c3f-4cae-832b-8ddb67c1fc60 | Ready | > 2 | 0 | 2013-12-06 01:25:44 | 81 | > | 703 | 1 | 356 | 2013-12-06 01:25:43 | NULL | NULL > | Primary | 0 | 0 | 0 | > 9d88bc01-9406-41ad-a134-e74dc1457954 | Ready | > 2 | 0 | 2013-12-06 01:26:12 | 80 | > | 704 | 1 | 357 | 2013-12-06 01:25:43 | NULL | NULL > | Primary | 0 | 0 | 0 | > 2667f2bc-6086-4ec3-a88d-20811eabde91 | Ready | > 2 | 0 | 2013-12-06 01:26:08 | 79 | > | 705 | 1 | 358 | 2013-12-06 01:25:44 | NULL | NULL > | Primary | 0 | 0 | 0 | > 522b2296-6960-46f2-af7d-10ddfbede1da | Ready | > 2 | 0 | 2013-12-06 01:26:45 | 78 | > | 706 | 1 | 359 | 2013-12-06 01:25:44 | NULL | NULL > | Primary | 0 | 0 | 0 | > 3b94fa9d-a5a5-4441-8f9f-275dcef90368 | Ready | > 2 | 0 | 2013-12-06 01:26:04 | 77 | > | 707 | 1 | 360 | 2013-12-06 01:25:44 | NULL | NULL > | Primary | 0 | 0 | 0 | > 1ec1d5ef-177f-4da4-8464-f0c6d71a4e84 | Ready | > 2 | 0 | 2013-12-06 01:25:59 | 76 | > | 708 | 1 | 361 | 2013-12-06 01:25:44 | NULL | NULL > | Primary | 0 | 0 | 0 | > 324e7552-b42a-4660-90d6-62015a7a478e | Ready | > 2 | 0 | 2013-12-06 01:26:21 | 75 | > | 709 | 1 | 362 | 2013-12-06 01:25:44 | NULL | NULL > | Primary | 0 | 0 | 0 | > 65bd522c-c2c8-471a-be37-095558d058f2 | Ready | > 2 | 0 | 2013-12-06 01:26:16 | 74 | > | 710 | 1 | 363 | 2013-12-06 01:25:44 | NULL | NULL > | Primary | 0 | 0 | 0 | > d45ca6c7-7284-4150-907c-9499e9737c47 | Ready | > 2 | 0 | 2013-12-06 01:25:46 | 73 | > | 711 | 1 | 364 | 2013-12-06 01:25:44 | NULL | NULL > | Primary | 0 | 0 | 0 | > 4422f362-0be5-4a10-b172-45678d56f807 | Ready | > 2 | 0 | 2013-12-06 01:25:55 | 72 | > | 712 | 1 | 365 | 2013-12-06 01:25:44 | NULL | NULL > | Primary | 0 | 0 | 0 | > 89ffd430-3c03-45d2-9c48-9384636b9cd8 | Ready | > 2 | 0 | 2013-12-06 01:26:01 | 71 | > | 714 | 1 | 366 | 2013-12-06 01:25:45 | NULL | NULL > | Primary | 0 | 0 | 0 | > fca5545c-9b83-4bc1-abd2-dd1bc82b23bd | Ready | > 2 | 0 | 2013-12-06 01:25:53 | 70 | > | 715 | 1 | 367 | 2013-12-06 01:25:45 | NULL | NULL > | Primary | 0 | 0 | 0 | > 033d8f55-8895-40b8-a120-11b28fa1f96e | Ready | > 2 | 0 | 2013-12-06 01:25:50 | 69 | > | 716 | 1 | 368 | 2013-12-06 01:25:45 | NULL | NULL > | Primary | 0 | 0 | 0 | > e4d02558-28c2-474e-a379-970b22f33f55 | Ready | > 2 | 0 | 2013-12-06 01:26:23 | 68 | > | 717 | 1 | 369 | 2013-12-06 01:25:45 | NULL | NULL > | Primary | 0 | 0 | 0 | > 6f7c1ca0-9877-44af-9f77-4db7b8efc934 | Ready | > 2 | 0 | 2013-12-06 01:26:37 | 67 | > | 718 | 1 | 370 | 2013-12-06 01:25:45 | NULL | NULL > | Primary | 0 | 0 | 0 | > e3860504-7e37-416a-8216-c4a021f5e8bf | Ready | > 2 | 0 | 2013-12-06 01:25:47 | 66 | > | 719 | 1 | 371 | 2013-12-06 01:25:45 | NULL | NULL > | Primary | 0 | 0 | 0 | > 48c2a9bd-e7ce-429f-be73-ef9c19964ad3 | Ready | > 2 | 0 | 2013-12-06 01:26:17 | 65 | > | 720 | 1 | 372 | 2013-12-06 01:25:45 | NULL | NULL > | Primary | 0 | 0 | 0 | > 3eabb081-d0d9-41d6-9c11-00e002031461 | Ready | > 2 | 0 | 2013-12-06 01:25:51 | 64 | > | 721 | 1 | 373 | 2013-12-06 01:25:46 | NULL | NULL > | Primary | 0 | 0 | 0 | > c9968fbb-0876-4846-94aa-e44ea7e3143d | Ready | > 2 | 0 | 2013-12-06 01:26:06 | 63 | > | 723 | 1 | 374 | 2013-12-06 01:25:46 | NULL | NULL > | Primary | 0 | 0 | 0 | > 72fdec4a-c2f1-48e1-80bf-806b6db46820 | Ready | > 2 | 0 | 2013-12-06 01:26:02 | 62 | > | 742 | 1 | 375 | 2013-12-06 01:40:43 | NULL | NULL > | Primary | 0 | 0 | 0 | > 8a692b5a-569e-473c-83c4-dc32c09c8491 | Ready | > 2 | 0 | 2013-12-06 01:40:47 | 60 | > +-----+----------+-------------+---------------------+--------------+--------+------------+------+---------------+--------------------+-----------------------------------------------------+-----------+--------------+---------+---------------------+-----------+ > 2013-12-05 20:51:41,212 DEBUG [c.c.s.s.SnapshotManagerImpl] > (Job-Executor-42:ctx-c20be5fc ctx-5f8cfbf3) Failed to create snapshot > com.cloud.utils.exception.CloudRuntimeException: BackupSnapshot Failed due to > Async 600 seconds timeout for task com.xensource.xenapi.Task@8b2ca24f > at > org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:275) > at > org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:135) > at > org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:294) > at > com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:951) > at sun.reflect.GeneratedMethodAccessor405.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) > at > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) > at > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) > at $Proxy161.takeSnapshot(Unknown Source) > at > org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1341) > at > com.cloud.storage.VolumeApiServiceImpl.takeSnapshot(VolumeApiServiceImpl.java:1408) > at sun.reflect.GeneratedMethodAccessor404.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) > at > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) > at > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) > at $Proxy233.takeSnapshot(Unknown Source) > at > org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:181) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161) > at > com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109) > at > com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:520) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:722) > 2013-12-05 20:51:41,217 DEBUG [o.a.c.s.v.VolumeServiceImpl] > (Job-Executor-42:ctx-c20be5fc ctx-5f8cfbf3) Take snapshot: 60 failed > com.cloud.utils.exception.CloudRuntimeException: Failed to create snapshot > at > com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:972) > at sun.reflect.GeneratedMethodAccessor405.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) > at > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) > at > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) > at $Proxy161.takeSnapshot(Unknown Source) > at > org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1341) > at > com.cloud.storage.VolumeApiServiceImpl.takeSnapshot(VolumeApiServiceImpl.java:1408) > at sun.reflect.GeneratedMethodAccessor404.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) > at > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) > at > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) > at $Proxy233.takeSnapshot(Unknown Source) > at > org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:181) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161) > at > com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109) > at > com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:520) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:722) > Caused by: com.cloud.utils.exception.CloudRuntimeException: BackupSnapshot > Failed due to Async 600 seconds timeout for task > com.xensource.xenapi.Task@8b2ca24f > at > org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:275) > at > org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:135) > at > org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:294) > at > com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:951) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) > at > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) > at > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) > at $Proxy161.takeSnapshot(Unknown Source) > at > org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1341) > at > com.cloud.storage.VolumeApiServiceImpl.takeSnapshot(VolumeApiServiceImpl.java:1408) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) > at > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) > at > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) > at $Proxy233.takeSnapshot(Unknown Source) > at > org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:181) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161) > at > com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109) > at > com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:520) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:722) > 2013-12-05 20:35:50,550 DEBUG [o.a.c.s.v.VolumeServiceImpl] > (Job-Executor-22:ctx-ae0b13d4 ctx-1bc38e81) Take snapshot: 81 failed > com.cloud.utils.exception.CloudRuntimeException: Failed to create snapshot > at > com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:972) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) > at > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) > at > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) > at $Proxy161.takeSnapshot(Unknown Source) > at > org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1341) > at > com.cloud.storage.VolumeApiServiceImpl.takeSnapshot(VolumeApiServiceImpl.java:1408) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) > at > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) > at > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) > at $Proxy233.takeSnapshot(Unknown Source) > at > org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:181) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161) > at > com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109) > at > com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:520) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:722) > Caused by: com.cloud.utils.exception.CloudRuntimeException: BackupSnapshot > Failed due to Async 600 seconds timeout for task > com.xensource.xenapi.Task@db9844a9 > at > org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:275) > at > org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:135) > at > org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:294) > at > com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:951) > ... 44 more > 2013-12-05 20:35:50,552 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (Job-Executor-22:ctx-ae0b13d4) Complete async job-697, jobStatus: FAILED, > resultCode: 530, result: > org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed > to create snapshot due to an internal error creating snapshot for volume 81"} > 2013-12-05 20:35:50,557 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (Job-Executor-22:ctx-ae0b13d4) Done executing > org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for job-697 -- This message was sent by Atlassian JIRA (v6.1.5#6160)