Sangeetha Hariharan created CLOUDSTACK-5372:
-----------------------------------------------
Summary: Xenserver - SR not being recreated when the Primary
storage is brought down and brought back up again resulting in not being able
to start the Vms that have their volumes in this primary store.
Key: CLOUDSTACK-5372
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5372
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Management Server
Affects Versions: 4.3.0
Environment: Build from 4.3
Reporter: Sangeetha Hariharan
Priority: Critical
Fix For: 4.3.0
Xenserver - SR not being recreated when the Primary storage is brought down and
brought back up again resulting in not being able to start the Vms that have
their volumes in this primary store.
Set up:
1 cluster with 2 hosts and 2 Primary storages (PS1 and PS2).
Start snapshot for couple of VMs which have primary store in PS1.
Reboot PS2. ( After this nfs server was still down)
I see that host1 and host2 reboot. But HA is not triggered , since host status
remains as “UP”.
After about 10 mts – Vmsync kicks in and stops all the Vms.
After about 12 mts (from the time snaoshots were created) , I see the snapshot
job failing (read timeouts).
I start nfs server now.
Issues:
I was not able to take another snapshot for the ROOT volume of the Vms that
reside in PS2.
I try to start the VM , Vm also fails to start.
I see the SR for PS2 is still in not in Connected state in Xenserver side.
Following exception seen when attempting to take a snapshot:
2013-12-04 15:48:19,502 WARN [c.c.h.x.r.XenServerStorageProcessor]
(DirectAgent-311:ctx-59768803) create snapshot operation Failed for snapshotId:
251, reason: The SR has no attached PBDs
The SR has no attached PBDs
at com.xensource.xenapi.Types.checkResponse(Types.java:510)
at com.xensource.xenapi.Connection.dispatch(Connection.java:368)
at
com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:909)
at com.xensource.xenapi.VDI.miamiSnapshot(VDI.java:1217)
at com.xensource.xenapi.VDI.snapshot(VDI.java:1192)
at
com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.createSnapshot(XenServerStorageProcessor.java:426)
at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:107)
at
com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:52)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:613)
at
com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:59)
at
com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:106)
at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
2013-12-04 15:48:19,503 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-311:ctx-59768803) Seq 2-1599671754: Response Received:
2013-12-04 15:48:19,503 DEBUG [c.c.a.t.Request] (DirectAgent-311:ctx-59768803)
Seq 2-1599671754: Processing: { Ans: , MgmtId: 112516401760401, via: 2, Ver:
v1, Flags: 10,
[{"org.apache.cloudstack.storage.command.CreateObjectAnswer":{"result":false,"details":"create
snapshot operation Failed for snapshotId: 251, reason: The SR has no attached
PBDs","wait":0}}]
Following exception seen when attempting to start the VM:
2013-12-04 15:52:54,833 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-75:ctx-460a2438) Seq 2-1599671770: Exec
uting request
2013-12-04 15:52:54,909 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-75:ctx-460a2438) 1. The VM i-2-19-VM
is in Starting state.
2013-12-04 15:52:54,930 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-75:ctx-460a2438) Created VM 9d2efad8-
8bae-fe4a-880a-22188dd5c624 for i-2-19-VM
2013-12-04 15:52:54,948 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-75:ctx-460a2438) VBD 7ff7a8bb-16b7-ca
37-0cf6-0bb60eb37f5b created for com.cloud.agent.api.to.DiskTO@6b27969a
2013-12-04 15:52:54,961 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-75:ctx-460a2438) VBD 0b0da9fa-b78b-1d
23-2f0f-ebf33bfd3e40 created for com.cloud.agent.api.to.DiskTO@74108f6b
2013-12-04 15:52:54,961 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-75:ctx-460a2438) Creating VIF for i-2
-19-VM on nic [Nic:Guest-10.1.1.155-vlan://2642]
2013-12-04 15:52:54,975 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-75:ctx-460a2438) Found a network call
ed VLAN-4ed5d099-99cb-5c18-8d49-c8e619afaeac-2642 on host=10.223.59.67;
Network=a5d975ae-4d80-4d12-2229-473c8e
9386a3; pif=cbe84007-3a45-2f3c-1834-4e73b3a1ce4e
2013-12-04 15:52:54,981 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-75:ctx-460a2438) Created a vif 3aea57
fa-3a12-98c9-8914-97826fd293c3 on 0
2013-12-04 15:52:55,996 WARN [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-75:ctx-460a2438) Task failed! Task re
cord: uuid: 781530e0-9fee-ea19-e9c4-850f28bda11b
nameLabel: Async.VM.start_on
nameDescription:
allowedOperations: []
currentOperations: {}
created: Wed Dec 04 15:53:03 EST 2013
finished: Wed Dec 04 15:53:03 EST 2013
status: failure
residentOn: com.xensource.xenapi.Host@72f2fe96
progress: 1.0
type: <none/>
result:
errorInfo: [VM_REQUIRES_SR,
OpaqueRef:c2c0b272-2439-4070-fe4f-a4759129e02a, OpaqueRef:fa8c0cf7-6d7b-
73e2-6207-c593d1ed4ff1]
otherConfig: {}
subtaskOf: com.xensource.xenapi.Task@aaf13f6f
subtasks: []
2013-12-04 15:52:55,998 WARN [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-75:ctx-460a2438) Unable to start VM(i
-2-19-VM) on host(23af93a0-93ff-40cb-ba11-a11d1b884d37) due to Task failed!
Task record: uuid:
781530e0-9fee-ea19-e9c4-850f28bda11b
--
This message was sent by Atlassian JIRA
(v6.1#6144)
