Sangeetha Hariharan created CLOUDSTACK-5357: -----------------------------------------------
Summary: Xenserver - Failed to create snapshot due to "unable to destroy task(com.xe nsource.xenapi.Task@67d312d6) on host(23af93a0-93ff-40cb-ba11-a11d1b884d37)" when secondary store was unavaiable for 1 and 1/2 hours and then brought up.. Key: CLOUDSTACK-5357 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5357 Project: CloudStack Issue Type: Bug Security Level: Public (Anyone can view this level - this is the default.) Components: Management Server Affects Versions: 4.3.0 Environment: Build from 4.3 Reporter: Sangeetha Hariharan Priority: Critical Fix For: 4.3.0 Xenserver - Failed to create snapshot due to "unable to destroy task(com.xe nsource.xenapi.Task@67d312d6) on host(23af93a0-93ff-40cb-ba11-a11d1b884d37)" when secondary store was unavaiable for 1 and 1/2 hours and then brought up. Set up: Advanced Zone with 2 Xenserver 6.2 hosts: Steps to reproduce the problem: 1. Deploy 5 Vms in each of the hosts with 10 GB ROOT volume size , so we start with 10 Vms. 2. Start concurrent hourly snapshots for ROOT volumes of all the Vms. 3. Shutdown the Secondary storage server when the snapshots are in the progress. 4. Bring the Secondary storage server up after 1and 1/2 hour. After the secondary store was brought up , we see the snapshot process continue to be active ( I see the copy command that was halted when the secondary store was down , starts progressing when server is drought up online). I also see that the snapshot process has been completed on the secondary store. But management server throws the following exception and leaves behind the snapshot in "CreatedOnPrimary" state in DB. 2013-12-03 19:24:36,296 WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-252:ctx-8ca50e48) unable to destroy task(com.xe nsource.xenapi.Task@67d312d6) on host(23af93a0-93ff-40cb-ba11-a11d1b884d37) due to You gave an invalid object reference. The object may have recently been deleted. The class parameter gives the type of reference given, and the handle parameter echoes the bad value given. at com.xensource.xenapi.Types.checkResponse(Types.java:209) at com.xensource.xenapi.Connection.dispatch(Connection.java:368) at com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool .java:909) at com.xensource.xenapi.Task.destroy(Task.java:616) at com.cloud.hypervisor.xen.resource.CitrixResourceBase.cloudVDIcopy(CitrixResourceBase.java:3920) at com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.backupSnapshot(XenServerStorageProcessor.java:1268 ) at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java: 90) at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHan dlerBase.java:50) at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:613) at com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:59) at com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:106) at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:10 3) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.ja va:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) 2013-12-03 19:24:36,297 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-252:ctx-8ca50e48) Host 10.223.59.67 OpaqueRef:50d81c75-f239-1746-ae06-c6228e788104: Removing SR 2013-12-03 19:24:36,305 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-252:ctx-8ca50e48) Host 10.223.59.67 OpaqueRef:e2bcc8f5-a04a-3f5a-c234-85beb190aa02: Unplugging pbd [root@Rack3Host5 6]# ls -lth total 1.3G -rw-r--r--. 1 root root 6.7G Dec 3 23:24 1f4ec466-688f-4a53-bfcb-e5f2da3bccae.vhd [root@Rack3Host5 6]# -- This message was sent by Atlassian JIRA (v6.1#6144)