Hi Simon, Yes existing VMs are successfully running. Yesterday I was able to create a VM in Dev Environment but I can't create VMs in Prod environment.
Thanks, Nezar. On 18 March 2016 at 09:12, Simon Weller <swel...@ena.com> wrote: > I'm assuming you have other VMs successfully running, correct? So you can > write to your primary storage. > > > > ________________________________________ > From: Nezar Madbouh <nezar.madb...@ipscape.com.au> > Sent: Thursday, March 17, 2016 4:43 PM > To: users@cloudstack.apache.org > Subject: Re: Unable to create a New VM - Error message: Unable to start > instance due to Unable to get answer that is of class > com.cloud.agent.api.StartAnswer > > Hi Simon and Shweta, > > Please see the result below: > > root@s-11-VM:~# /usr/local/cloud/systemvm/ssvm-check.sh > ================================================ > First DNS server is 8.8.8.8 > PING 8.8.8.8 (8.8.8.8): 48 data bytes > 56 bytes from 8.8.8.8: icmp_seq=0 ttl=57 time=5.209 ms > 56 bytes from 8.8.8.8: icmp_seq=1 ttl=57 time=0.936 ms > --- 8.8.8.8 ping statistics --- > 2 packets transmitted, 2 packets received, 0% packet loss > round-trip min/avg/max/stddev = 0.936/3.072/5.209/2.137 ms > Good: Can ping DNS server > ================================================ > Good: DNS resolves download.cloud.com > ================================================ > nfs is currently mounted > Mount point is /mnt/SecStorage/d11d5c92-6950-30e8-a0b5-3cd46a17ec14 > Good: Can write to mount point > ================================================ > Management server is 10.61.153.20. Checking connectivity. > Good: Can connect to management server port 8250 > ================================================ > Good: Java process is running > ================================================ > Tests Complete. Look for ERROR or WARNING above. > root@s-11-VM:~# > > Thanks & Regards, > Nezar. > > On 17 March 2016 at 09:42, Simon Weller <swel...@ena.com> wrote: > > > Nezar, > > > > Log into the Cloudstack UI. > > > > Click on "Infrastructure" on the left menu. Click on View All below > > "System VMs". find the item marked "Secondary Storage VM" and then click > > on the Name of that VM on the left (it will be s-XXXX-VM). You'll see a > > bunch of information about the VM. Find the Host line and also the Link > > Local IP Address line and then follow the original instructions in my > > previous email. > > > > - Si > > > > > > > > > > > > ________________________________________ > > From: Shweta Agarwal <shweta.agarw...@accelerite.com> > > Sent: Wednesday, March 16, 2016 12:16 AM > > To: users@cloudstack.apache.org > > Subject: Re: Unable to create a New VM - Error message: Unable to start > > instance due to Unable to get answer that is of class > > com.cloud.agent.api.StartAnswer > > > > Hi Nezar, > > > > You can find host name on detail view of SSVM in UI . > > You can also use list api command listSystemVms to get the host details > of > > system vms. > > > > Thanks > > Shweta > > > > > > ________________________________________ > > From: Nezar Madbouh <nezar.madb...@ipscape.com.au> > > Sent: Wednesday, March 16, 2016 9:58 AM > > To: users@cloudstack.apache.org > > Subject: Re: Unable to create a New VM - Error message: Unable to start > > instance due to Unable to get answer that is of class > > com.cloud.agent.api.StartAnswer > > > > Hi Simon, > > > > How can I find out which host the SSVM is on? > > > > Thanks & Regards, > > Nezar. > > > > On 16 March 2016 at 12:28, Simon Weller <swel...@ena.com> wrote: > > > > > Can you ssh into the Secondary Storage VM? > > > > > > This can be done by getting the link-local-ip address from the > > > infrastructure menu in the UI and then sshing into the host the SSVM is > > on > > > and issuing this command as root: ssh -i /root/.ssh/id_rsa.cloud -p > 3922 > > > root@link-local-ip (replace link-local-ip with the link local ip). > > > > > > Try running the SSVM diangnostic script: > > > usr/local/cloud/systemvm/ssvm-check.sh > > > > > > - Si > > > ________________________________________ > > > From: Nezar Madbouh <nezar.madb...@ipscape.com.au> > > > Sent: Tuesday, March 15, 2016 7:49 PM > > > To: users@cloudstack.apache.org > > > Subject: Re: Unable to create a New VM - Error message: Unable to start > > > instance due to Unable to get answer that is of class > > > com.cloud.agent.api.StartAnswer > > > > > > Hi Simon, > > > > > > Please see details below: > > > > > > > > > mysql> use cloud; select id, name, status, type from host where id = 5; > > > Reading table information for completion of table and column names > > > You can turn off this feature to get a quicker startup with -A > > > > > > Database changed > > > +----+------------------------------+--------+---------+ > > > | id | name | status | type | > > > +----+------------------------------+--------+---------+ > > > | 5 | eqx-cs-cmp-02.ipscape.com.au | up | routing | > > > +----+------------------------------+--------+---------+ > > > 1 row in set (0.00 sec) > > > > > > > > > > > > cat /var/log/cloudstack/agent/agent.log > > > com.cloud.utils.exception.CloudRuntimeException: > > > com.cloud.utils.exception.CloudRuntimeException: Can't find > > > volume:089ff3ca-9bff-402c-a14e-ea7a35aaad5c > > > at > > > > > > > > > com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getPhysicalDisk(KVMStoragePoolManager.java:262) > > > at > > > > > > > > > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVbd(LibvirtComputingResource.java:3921) > > > at > > > > > > > > > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:3789) > > > at > > > > > > > > > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1333) > > > at com.cloud.agent.Agent.processRequest(Agent.java:501) > > > at > > com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:808) > > > at com.cloud.utils.nio.Task.run(Task.java:84) > > > at > > > > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > > at > > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > > at java.lang.Thread.run(Thread.java:745) > > > agent.log.2016-03-03.gz (END) > > > > > > > > > > > > [root@eqx-cs-cmp-02 agent]# virsh pool-list > > > Name State Autostart > > > ----------------------------------------- > > > 2d0967f8-50d6-4d6d-87b7-4da0596f6924 active no > > > b3647480-5709-36eb-9ddd-e2c666f30b57 active no > > > e3831246-ff55-4af4-9964-a17565edaccc active no > > > > > > > > > > > > > > > [Root @ eqx-cs-cmp-02 agent] # df -h > > > Filesystem Size Used Avail Use% Mounted on > > > /dev/mapper/vg_root-lv_root > > > 203G 215G 1.3G 1% / > > > tmpfs 48g 48g 0 0% / dev / shm > > > / Dev / sda1 477m 427m 25M 6% / boot > > > xx.xx.xx.xx:/Cloudstack/primary > > > 7.6T 4.1T 3.5T 54% > > > /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57 > > > > > > > > > > > > > > > I also noticed the following in cloudstack-agent.err > > > > > > libvirt: QEMU Driver error : Domain not found: no domain with matching > > name > > > 'i-11-600-VM' > > > libvirt: Storage Driver error : Storage volume not found: no storage > vol > > > with matching name '089ff3ca-9bff-402c-a14e-ea7a35aaad5c' > > > > > > Your help is much appreciated, > > > Nezar > > > > > > > > > On 16 March 2016 at 11:03, Simon Weller <swel...@ena.com> wrote: > > > > > > > Nezar, > > > > > > > > ok, I just took a look at a pastebin you posted. > > > > > > > > I see it complaining about the inability to find a volume prior to > > > > throwing the host exception. > > > > > > > > Firstly, let's find out who host 5 is. Can you log into your MySQL > > server > > > > and run this: > > > > > > > > use cloud;select id,name,status,type from host where id=5; > > > > > > > > Can you post the results? > > > > > > > > If it's a host, can you post the /var/log/cloudstack/agent/agent.log > > from > > > > that host? > > > > > > > > And also run virsh pool-list on the host in question? > > > > > > > > - Si > > > > > > > > > > > > > > > > > > > > > > > > ________________________________________ > > > > From: Nezar Madbouh <nezar.madb...@ipscape.com.au> > > > > Sent: Tuesday, March 15, 2016 6:11 PM > > > > To: users@cloudstack.apache.org > > > > Subject: Re: Unable to create a New VM - Error message: Unable to > start > > > > instance due to Unable to get answer that is of class > > > > com.cloud.agent.api.StartAnswer > > > > > > > > Hi Simon, > > > > > > > > When you say "double check your routers tied to the network", do you > > mean > > > > the physical routers or the virtual routers? > > > > > > > > When I try to create a VM in Prod Environment it fails. I can't ping > > the > > > > Prod virtual router from management server or Host. > > > > > > > > But I can ping Dev virtual router from management server and Host > and I > > > was > > > > able to create a VM in Dev Environment. > > > > > > > > How can I troubleshoot the Prod Virtual router. I launched the > console > > > but > > > > couldn't login. > > > > > > > > > > > > Please note that we need to take into consideration that the affected > > > > network is the Production network and we need to minimise service > > outage > > > as > > > > much as possible. > > > > > > > > Your help is much appreciated, > > > > Nezar > > > > > > > > > > > > > > > > On 16 March 2016 at 08:37, Simon Weller <swel...@ena.com> wrote: > > > > > > > > > I've seen this before where a Virtual Router has crashed and the > > > > > management server can't communicate with it. > > > > > Are you using an advanced network setup? > > > > > > > > > > If so, double check your routers tied to the network where you are > > > > > attempting to start an instance. > > > > > > > > > > - Si > > > > > > > > > > > > > > > ________________________________________ > > > > > From: Nezar Madbouh <nezar.madb...@ipscape.com.au> > > > > > Sent: Tuesday, March 15, 2016 4:02 PM > > > > > To: users@cloudstack.apache.org > > > > > Subject: Re: Unable to create a New VM - Error message: Unable to > > start > > > > > instance due to Unable to get answer that is of class > > > > > com.cloud.agent.api.StartAnswer > > > > > > > > > > Hi Glenn, > > > > > > > > > > We have 16 hosts, all KVM. > > > > > > > > > > The error is not specific to Host 5. Every time I try to create a > VM > > it > > > > > gives me the same error message but on different host. > > > > > > > > > > How do I restart the agent on the host. Is it service affecting? > > > > > > > > > > Thanks very much for your help. > > > > > Neza > > > > > > > > > > On 15 March 2016 at 17:37, Glenn Wagner < > glenn.wag...@shapeblue.com> > > > > > wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > Can you explain your environment first , how many hosts , what is > > > your > > > > > > hypervisors (KVM,Xenserver,VMware) > > > > > > > > > > > > The error seems to be with your Host 5 Hypervisor > > > > > > Resource [Host:5] is unreachable: > > > > > > > > > > > > Go to the infrastructure section and see what host5 is reporting, > > > > > > Have you tried to restart the agent on the Host? > > > > > > > > > > > > Thanks > > > > > > Glenn > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [image: ShapeBlue] <http://www.shapeblue.com> > > > > > > Glenn Wagner > > > > > > Senior Consultant , ShapeBlue > > > > > > d: * | s: +27 21 527 0091* <%7C%20s:%20+27%2021%20527%200091> > | > > > m: > > > > > > *+27 73 917 4111* <+27%2073%20917%204111> > > > > > > e: *glenn.wag...@shapeblue.com | t: * > > > > > > <glenn.wag...@shapeblue.com%20%7C%20t:> | w: * > www.shapeblue.com > > * > > > > > > <http://www.shapeblue.com> > > > > > > a: > > > > > > 2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West Cape Town > > 7130 > > > > > South Africa > > > > > > Shape Blue Ltd is a company incorporated in England & Wales. > > > ShapeBlue > > > > > > Services India LLP is a company incorporated in India and is > > operated > > > > > under > > > > > > license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda > is > > a > > > > > > company incorporated in Brasil and is operated under license from > > > Shape > > > > > > Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The > > > Republic > > > > of > > > > > > South Africa and is traded under license from Shape Blue Ltd. > > > ShapeBlue > > > > > is > > > > > > a registered trademark. > > > > > > This email and any attachments to it may be confidential and are > > > > intended > > > > > > solely for the use of the individual to whom it is addressed. Any > > > views > > > > > or > > > > > > opinions expressed are solely those of the author and do not > > > > necessarily > > > > > > represent those of Shape Blue Ltd or related companies. If you > are > > > not > > > > > the > > > > > > intended recipient of this email, you must neither take any > action > > > > based > > > > > > upon its contents, nor copy or show it to anyone. Please contact > > the > > > > > sender > > > > > > if you believe you have received this email in error. > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > From: Nezar Madbouh [mailto:nezar.madb...@ipscape.com.au] > > > > > > Sent: Tuesday, 15 March 2016 5:45 AM > > > > > > To: users@cloudstack.apache.org > > > > > > Subject: Re: Unable to create a New VM - Error message: Unable to > > > start > > > > > > instance due to Unable to get answer that is of class > > > > > > com.cloud.agent.api.StartAnswer > > > > > > > > > > > > Hi Jeremy, > > > > > > > > > > > > I have uploaded the contents to http://pastebin.com/1J3mmgTh > > > > > > > > > > > > your help is much appreciated, > > > > > > Nezar. > > > > > > > > > > > > On 15 March 2016 at 13:46, Jeremy Peterson < > jpeter...@acentek.net> > > > > > wrote: > > > > > > > > > > > > > I don't see any attachments can you create a pastebin. That's > > > easier > > > > > > > to look through as I think attachments are removed before being > > > sent > > > > > > > to user group. > > > > > > > > > > > > > > Jeremy > > > > > > > > > > > > > > > > > > > > > -------- Original message -------- > > > > > > > From: Nezar Madbouh <nezar.madb...@ipscape.com.au> > > > > > > > Date: 3/14/2016 6:38 PM (GMT-06:00) > > > > > > > To: users@cloudstack.apache.org > > > > > > > Subject: Unable to create a New VM - Error message: Unable to > > start > > > > > > > instance due to Unable to get answer that is of class > > > > > > > com.cloud.agent.api.StartAnswer > > > > > > > > > > > > > > Hi Shweta and Sadhu, > > > > > > > > > > > > > > Please find attached managerment-server.log which contains the > > logs > > > > > > > since I started creating a new VM (VM id 600). > > > > > > > > > > > > > > Please note that we don't have agent.log files > > > > > > > > > > > > > > Your help is much appreciated. > > > > > > > > > > > > > > Regards, > > > > > > > Nezar. > > > > > > > > > > > > > Find out more about ShapeBlue and our range of CloudStack related > > > > > services: > > > > > > IaaS Cloud Design & Build > > > > > > <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge – > > > rapid > > > > > > IaaS deployment framework <http://shapeblue.com/csforge/> > > > > > > CloudStack Consulting < > > http://shapeblue.com/cloudstack-consultancy/> > > > | > > > > > CloudStack > > > > > > Software Engineering > > > > > > <http://shapeblue.com/cloudstack-software-engineering/> > > > > > > CloudStack Infrastructure Support > > > > > > <http://shapeblue.com/cloudstack-infrastructure-support/> | > > > CloudStack > > > > > > Bootcamp Training Courses < > > http://shapeblue.com/cloudstack-training/ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > DISCLAIMER > > ========== > > This e-mail may contain privileged and confidential information which is > > the property of Accelerite, a Persistent Systems business. It is intended > > only for the use of the individual or entity to which it is addressed. If > > you are not the intended recipient, you are not authorized to read, > retain, > > copy, print, distribute or use this message. If you have received this > > communication in error, please notify the sender and delete all copies of > > this message. Accelerite, a Persistent Systems business does not accept > any > > liability for virus infected mails. > > >
