Sangeetha Hariharan created CLOUDSTACK-5372: -----------------------------------------------
Summary: Xenserver - SR not being recreated when the Primary storage is brought down and brought back up again resulting in not being able to start the Vms that have their volumes in this primary store. Key: CLOUDSTACK-5372 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5372 Project: CloudStack Issue Type: Bug Security Level: Public (Anyone can view this level - this is the default.) Components: Management Server Affects Versions: 4.3.0 Environment: Build from 4.3 Reporter: Sangeetha Hariharan Priority: Critical Fix For: 4.3.0 Xenserver - SR not being recreated when the Primary storage is brought down and brought back up again resulting in not being able to start the Vms that have their volumes in this primary store. Set up: 1 cluster with 2 hosts and 2 Primary storages (PS1 and PS2). Start snapshot for couple of VMs which have primary store in PS1. Reboot PS2. ( After this nfs server was still down) I see that host1 and host2 reboot. But HA is not triggered , since host status remains as “UP”. After about 10 mts – Vmsync kicks in and stops all the Vms. After about 12 mts (from the time snaoshots were created) , I see the snapshot job failing (read timeouts). I start nfs server now. Issues: I was not able to take another snapshot for the ROOT volume of the Vms that reside in PS2. I try to start the VM , Vm also fails to start. I see the SR for PS2 is still in not in Connected state in Xenserver side. Following exception seen when attempting to take a snapshot: 2013-12-04 15:48:19,502 WARN [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-311:ctx-59768803) create snapshot operation Failed for snapshotId: 251, reason: The SR has no attached PBDs The SR has no attached PBDs at com.xensource.xenapi.Types.checkResponse(Types.java:510) at com.xensource.xenapi.Connection.dispatch(Connection.java:368) at com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:909) at com.xensource.xenapi.VDI.miamiSnapshot(VDI.java:1217) at com.xensource.xenapi.VDI.snapshot(VDI.java:1192) at com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.createSnapshot(XenServerStorageProcessor.java:426) at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:107) at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:52) at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:613) at com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:59) at com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:106) at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) 2013-12-04 15:48:19,503 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-311:ctx-59768803) Seq 2-1599671754: Response Received: 2013-12-04 15:48:19,503 DEBUG [c.c.a.t.Request] (DirectAgent-311:ctx-59768803) Seq 2-1599671754: Processing: { Ans: , MgmtId: 112516401760401, via: 2, Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CreateObjectAnswer":{"result":false,"details":"create snapshot operation Failed for snapshotId: 251, reason: The SR has no attached PBDs","wait":0}}] Following exception seen when attempting to start the VM: 2013-12-04 15:52:54,833 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-75:ctx-460a2438) Seq 2-1599671770: Exec uting request 2013-12-04 15:52:54,909 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-75:ctx-460a2438) 1. The VM i-2-19-VM is in Starting state. 2013-12-04 15:52:54,930 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-75:ctx-460a2438) Created VM 9d2efad8- 8bae-fe4a-880a-22188dd5c624 for i-2-19-VM 2013-12-04 15:52:54,948 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-75:ctx-460a2438) VBD 7ff7a8bb-16b7-ca 37-0cf6-0bb60eb37f5b created for com.cloud.agent.api.to.DiskTO@6b27969a 2013-12-04 15:52:54,961 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-75:ctx-460a2438) VBD 0b0da9fa-b78b-1d 23-2f0f-ebf33bfd3e40 created for com.cloud.agent.api.to.DiskTO@74108f6b 2013-12-04 15:52:54,961 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-75:ctx-460a2438) Creating VIF for i-2 -19-VM on nic [Nic:Guest-10.1.1.155-vlan://2642] 2013-12-04 15:52:54,975 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-75:ctx-460a2438) Found a network call ed VLAN-4ed5d099-99cb-5c18-8d49-c8e619afaeac-2642 on host=10.223.59.67; Network=a5d975ae-4d80-4d12-2229-473c8e 9386a3; pif=cbe84007-3a45-2f3c-1834-4e73b3a1ce4e 2013-12-04 15:52:54,981 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-75:ctx-460a2438) Created a vif 3aea57 fa-3a12-98c9-8914-97826fd293c3 on 0 2013-12-04 15:52:55,996 WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-75:ctx-460a2438) Task failed! Task re cord: uuid: 781530e0-9fee-ea19-e9c4-850f28bda11b nameLabel: Async.VM.start_on nameDescription: allowedOperations: [] currentOperations: {} created: Wed Dec 04 15:53:03 EST 2013 finished: Wed Dec 04 15:53:03 EST 2013 status: failure residentOn: com.xensource.xenapi.Host@72f2fe96 progress: 1.0 type: <none/> result: errorInfo: [VM_REQUIRES_SR, OpaqueRef:c2c0b272-2439-4070-fe4f-a4759129e02a, OpaqueRef:fa8c0cf7-6d7b- 73e2-6207-c593d1ed4ff1] otherConfig: {} subtaskOf: com.xensource.xenapi.Task@aaf13f6f subtasks: [] 2013-12-04 15:52:55,998 WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-75:ctx-460a2438) Unable to start VM(i -2-19-VM) on host(23af93a0-93ff-40cb-ba11-a11d1b884d37) due to Task failed! Task record: uuid: 781530e0-9fee-ea19-e9c4-850f28bda11b -- This message was sent by Atlassian JIRA (v6.1#6144)