Richard, From the logs, there seems to be an issue with your storage configuration. It _looks_ like you have enabled 'local storage' and that the system vms are using that. Did you explicitly add your NFS from each of the servers in the CloudStack GUI or API? The built-in service offerings for VMs don't include ones for local storage, only shared storage. As CloudStack can't find any shared storage, and the system vms appear to be running on local storage, it suggests to me that you only have local storage attached.
Regards Paul. paul.an...@shapeblue.com www.shapeblue.com Amadeus House, Floral Street, London WC2E 9DPUK @shapeblue -----Original Message----- From: Richard Persaud <richard.pers...@macys.com> Sent: 10 May 2019 22:45 To: users@cloudstack.apache.org Cc: /usr/local/cloud/systemvm/ssvm-check...@gmail.com Subject: RE: issue with system vm template not downloading Hi Andrija, Advanced zone. I ran the ssvm-check.sh and I did correct some NFS errors. Now, ssvm-check.sh runs without any issues. The situation seems to have worsened – now I am unable to deploy any VM: http://www.filedropper.com/cs-mgmt-log2 I am getting an insufficient capacity error, but this is a new environment, so no guest VMs or guest Networks are created. Regards, Richard Persaud From: Andrija Panic <andrija.pa...@gmail.com> Sent: Friday, May 10, 2019 2:00 PM To: users <users@cloudstack.apache.org> Cc: /usr/local/cloud/systemvm/ssvm-check...@gmail.com Subject: Re: issue with system vm template not downloading ⚠ EXT MSG: Hi Richard, Can you also, beside logs, try/confirm the following: - are you running a Basic zone ? - since systemVM tempate should be pre-seeded with a script, perhaps you are having issues with the SSVM itself and/or downloading default CentOS template? If so please execute from inside SSVM the following script and provide results : /usr/local/cloud/systemvm/ssvm-check.sh<https://isolate.menlosecurity.com/0/eJyrViotylGyUsooKSmw0tcvLi7L1U3OSE3O1ivOUNJRKspXsjLUUSrJTAGqMTQwVqoFAI6zDuI> and send us the output. Regards, Andrija On Fri, May 10, 2019, 17:52 Paul Angus <paul.an...@shapeblue.com<mailto:paul.an...@shapeblue.com>> wrote: > Richard, > > Could you post (using > pastebin.com<https://isolate.menlosecurity.com/0/eJyrViotylGyUsooKSmw0tcvSCwuSU3KzNNLzs9V0lEqyleyMtRRKslMASoxNDBWqgUAgOAOqA> > or similar) a large chunk of your management server logs around the time of > this error please. > I would think that we need some context to the message. 'Storage host' > usually refers to the SSVM itself, a timeout waiting for a response > _suggests_ that the SSVM has been deployed, but isn't 'checking-in'. > Much more of the log information would probably help, whatever the problem. > > Also... > I can kinda see what you're trying to achieve with NFS on all of the > hosts > - but I agree with Ivan, I think that you'll soon regret it. > > > Regards > > Paul Angus > > paul.an...@shapeblue.com<mailto:paul.an...@shapeblue.com> > www.shapeblue.com<https://isolate.menlosecurity.com/0/eJyrViotylGyUsoo > KSmw0tcvLy_XK85ILEhNyilN1UvOz1XSUSrKV7Iy1FEqyUwBqjM0MFaqBQDf4BCe> > Amadeus House, Floral Street, London WC2E 9DPUK @shapeblue > > > > > -----Original Message----- > From: Ivan Kudryavtsev > <kudryavtsev...@bw-sw.com<mailto:kudryavtsev...@bw-sw.com>> > Sent: 10 May 2019 16:28 > To: users > <users@cloudstack.apache.org<mailto:users@cloudstack.apache.org>> > Subject: Re: issue with system vm template not downloading > > Richard, > > 1. About bridges, just check traffic goes correctly between HV hosts > and SS. All your hosts should be able to mount SS. > 2. About your HV/Storage topology. > > 1st. Cloudstack doesn't balance the storages. While the first chosen > storage is capable to deploy the image, it will be used. You will not > able to balance volumes between them. > 2nd. Every HV will mount every storage. If HV fails (which is probably > happen more frequently than storage) it will cause __all the__ hosts > will meet the problem with the NFS share and kick reboot, so, all your > cloud will reboot. > Frankly, it's the worst topology possible. What I recommend is to > switch either to Ceph or Gluster if you want shared storage, split all > the hosts to separate clusters or use LOCAL STORAGE instead of NFS > storage, so your VMS will use local storage. Later if you wish to move > VMs between hosts, you can do it manually. > > Best wishes > > > > > > пт, 10 мая 2019 г. в 20:14, Richard Persaud > <richard.pers...@macys.com<mailto:richard.pers...@macys.com>>: > > > Hi Ivan, > > > > Thanks for the info. > > > > Will you clarify what I should be looking for in my bridge set up? > > It's fairly standard other then setting the MTU to 9000. > > > > The host/storage devices are using hardware RAID5. And all > > hypervisors are capable of mounting any of the NFS shares. > > > > Will you give me some detail on what you mean when you say using > > native RAID is a bad idea? Why is that and what is the recommended > > way > to set up? > > > > Thanks in advance > > > > > > Regards, > > Richard Persaud > > Sys Spec, Info Security Del | Macy's, Inc. > > 5985 State Bridge Rd. | Johns Creek, GA 30097 > > Office: 678-474-2357 > > https://macyspartners.com/PublishingImages/MakeLifeShineBrighter.png > > > > ________________________________ > > From: Ivan Kudryavtsev > > <kudryavtsev...@bw-sw.com<mailto:kudryavtsev...@bw-sw.com>> > > Sent: Thursday, May 9, 2019 21:57 > > To: users > > Subject: Re: issue with system vm template not downloading > > > > ⚠ EXT MSG: > > > > Richard, the most probable problem is with bridge devices. > > Management server doesn't care about systemvm. The only unit which > > cares - ssvm and hypervisor. Also, if you are using naive RAID/NFS > > within one cluster when any HV can mount any storage (mesh) it's > > extremely bad idea. You will get s lot of reboots if any of node > > meets outage. If you have DRBD or Gluster, then, it's fine. > > > > пт, 10 мая 2019 г., 6:32 Richard Persaud > > <richard.pers...@macys.com<mailto:richard.pers...@macys.com> > > <mailto:richard.pers...@macys.com<mailto:richard.pers...@macys.com>>>: > > > > > Hello, > > > > > > Our setup: > > > 4.11 on Ubuntu 16.04 LTS. One management server, eight > > > compute/storage hosts (dual function). > > > NFS for storage. > > > No firewall in between the mgmt server and the hosts. > > > Management and storage traffic run over the same VLAN (same network). > > > > > > We are having an issue with the system vm template not downloading. > > > We have seen this issue on multiple occasions > > > > > > "Timeout waiting for response from storage host" > > > > > > It does not give any further information. > > > > > > The management server can successfully contact and mount the NFS > > > shares from all the compute/storage hosts. > > > > > > How can I determine which storage host is timing out? Why is it > > > timing > > out? > > > > > > Regards, > > > > > > Richard Persaud > > > > > > > * This is an EXTERNAL EMAIL. Stop and think before clicking a link > > or opening attachments. > > > > > -- > With best regards, Ivan Kudryavtsev > Bitworks LLC > Cell RU: +7-923-414-1515 > Cell USA: +1-201-257-1512 > WWW: > http://bitworks.software/<https://isolate.menlosecurity.com/0/eJyrViot > ylGyUsooKSmw0tdPyiwpzy_KLtYrzk8rKU8sStVX0lEqyleyMtRRKslMASo0NDBWqgUA-M > 8RIg> > <http://bw-sw.com/<https://isolate.menlosecurity.com/0/eJyrViotylGyUso > oKSmw0tdPKtctLtdLzs_VV9JRKspXsjLUUSrJTAEqMDQwVqoFAFJWDXE>> > * This is an EXTERNAL EMAIL. Stop and think before clicking a link or opening attachments.
