Re: Help! After network outage, can't start System VMs; focused debug info attached

Thank you, Kirk, that was most helpful. I currently have the system running with SystemVMs using local storage, with the loss of all our (supposedly) persistent volumes in shared storage. After we get past the emergency I will try Kirk's suggestion of deleting the stale line from the template_spo

Re: Help! After network outage, can't start System VMs; focused debug info attached

Hi, secondary storage is only mounted on an as-needed basis. When a KVM or XenServer host needs to do something on secondary storage, it will mount the full path it needs (e.g. nfshost:/share/template/tmpl/2/123), do what it needs to do, and unmount it. The error seems to be that CloudStack is lo

Re: Help! After network outage, can't start System VMs; focused debug info attached

I went through your logs, there seems to be an issue with secondary storage, have you tried to create another secondary storage? and maybe prepare new system templates link

Re: Help! After network outage, can't start System VMs; focused debug info attached

Hi, I've now heard that this problem, of Cloudstack being messed up after interruption of the NFS shared storage access, is well known. Does anyone have a fix or work-around? Kirk, thanks for your help so far. Both the master and the host servers can mount both primary and secondary stores, and r

Re: Help! After network outage, can't start System VMs; focused debug info attached

Does Reconfiguring CloudStack to use local storage for system VMs retain all the VMs that are already in Cloudstack. There are about 200 vms running in our cloudstack. How about the VM templates, snapshots and all other stuff that were already in Cloudstack. I mean would it still restore the Cloud

Re: Help! After network outage, can't start System VMs; focused debug info attached

Okay, so system VMs are using NFS primary storage (I mis-read the OP, sorry). Make sure the KVM hosts can mount and write to: 10.42.1.101:/srv/nfs/eng/cs-primary Also check libvirtd.log for errors. If you're not making progress and want to get up and running ASAP, try reconfiguring CloudStack t

Re: Help! After network outage, can't start System VMs; focused debug info attached

Hi Kirk, Thanks for your reply. This is a blocker for us and currently affected all of our work. It is very helpful to debug more into the issue. I have a question. 1. What should the directory be when mounting [2] systemVM template location on the nfs drive. Error from agent.log on the host. C

Re: Help! After network outage, can't start System VMs; focused debug info attached

Hi, here is the error: 2013-09-16 15:08:17,168 DEBUG [agent.transport.Request] (AgentManager-Handler-5:null) Seq 13-931004532: Processing: { Ans: , MgmtId: 161340856362, via: 13, Ver: v1, Flags: 110, [{"storage.CreateAnswer":{"requestTemplateReload":false,"result":false,"details":"Exception: com.

Re: Help! After network outage, can't start System VMs; focused debug info attached

Replying on behalf of Matt. We are able to write data to the Nfs drives. That's not an issue. Thanks Sriharsha Sent from my iPhone > On Sep 16, 2013, at 19:30, Ahmad Emneina wrote: > > Try to mount your primary storage to a compute host and try to write to it. > Your NFS server might not have c

Re: Help! After network outage, can't start System VMs; focused debug info attached

Try to mount your primary storage to a compute host and try to write to it. Your NFS server might not have come back up properly (settings-wise or all the relevant services). On Sep 16, 2013 6:08 PM, "Matt Foley" wrote: > Thank you Chiradeep. Log snippet now available as http://apaste.info/qBIB

Re: Help! After network outage, can't start System VMs; focused debug info attached

Thank you Chiradeep. Log snippet now available as http://apaste.info/qBIB --Matt On Mon, Sep 16, 2013 at 5:19 PM, Chiradeep Vittal < chiradeep.vit...@citrix.com> wrote: > Attachments are stripped. Can you paste (say at http://apaste.info/) > > From: Matt Foley > Date: Monday, September 16, 2013

Re: Help! After network outage, can't start System VMs; focused debug info attached

16, 2013 4:58 PM To: "users@cloudstack.apache.org<mailto:users@cloudstack.apache.org>" mailto:users@cloudstack.apache.org>> Subject: Help! After network outage, can't start System VMs; focused debug info attached We had a planned network outage this weekend, which inadvertently resu

Help! After network outage, can't start System VMs; focused debug info attached

We had a planned network outage this weekend, which inadvertently resulted in making the NFS Shared Primary Storage (used by System VMs) unavailable for a day and a half. (Guest VMs use local storage only, but System VMs use shared storage only.) Cloudstack was not brought down prior to the outag