Also, what is the Cloudstack version you are running and what is the OS you're running it on?
Are you using actimeo=0 in your NFS mount line? ________________________________________ From: Simon Weller <swel...@ena.com> Sent: Friday, March 18, 2016 7:40 AM To: users@cloudstack.apache.org Subject: Re: Unable to create a New VM - Error message: Unable to start instance due to Unable to get answer that is of class com.cloud.agent.api.StartAnswer Nexar, That's normal behaviour. You can only ssh to the link local on the host where the VM resides. Can you screenshot and post somewhere your secondary storage configuration and your primary storage configuration from the UI with the details? - Si ________________________________________ From: Nezar Madbouh <nezar.madb...@ipscape.com.au> Sent: Thursday, March 17, 2016 8:19 PM To: users@cloudstack.apache.org Subject: Re: Unable to create a New VM - Error message: Unable to start instance due to Unable to get answer that is of class com.cloud.agent.api.StartAnswer Hi Simon, I just noticed that from other Hosts I can't ssh into the SSVM: [root@eqx-cs-cmp-01 ~]# ssh -i /root/.ssh/id_rsa.cloud -p 3922 root@169.254.3.161 ssh: connect to host 169.254.3.161 port 3922: No route to host I can only ssh from one Host: [root@eqx-cs-cmp-06 ~]# ssh -i /root/.ssh/id_rsa.cloud -p 3922 root@169.254.3.161 Linux s-11-VM 3.2.0-4-amd64 #1 SMP Debian 3.2.57-3 x86_64 Thanks & Regards, Nezar. On 18 March 2016 at 10:36, Nezar Madbouh <nezar.madb...@ipscape.com.au> wrote: > Hi Simon, > > Yes existing VMs are successfully running. Yesterday I was able to create > a VM in Dev Environment but I can't create VMs in Prod environment. > > Thanks, > Nezar. > > On 18 March 2016 at 09:12, Simon Weller <swel...@ena.com> wrote: > >> I'm assuming you have other VMs successfully running, correct? So you can >> write to your primary storage. >> >> >> >> ________________________________________ >> From: Nezar Madbouh <nezar.madb...@ipscape.com.au> >> Sent: Thursday, March 17, 2016 4:43 PM >> To: users@cloudstack.apache.org >> Subject: Re: Unable to create a New VM - Error message: Unable to start >> instance due to Unable to get answer that is of class >> com.cloud.agent.api.StartAnswer >> >> Hi Simon and Shweta, >> >> Please see the result below: >> >> root@s-11-VM:~# /usr/local/cloud/systemvm/ssvm-check.sh >> ================================================ >> First DNS server is 8.8.8.8 >> PING 8.8.8.8 (8.8.8.8): 48 data bytes >> 56 bytes from 8.8.8.8: icmp_seq=0 ttl=57 time=5.209 ms >> 56 bytes from 8.8.8.8: icmp_seq=1 ttl=57 time=0.936 ms >> --- 8.8.8.8 ping statistics --- >> 2 packets transmitted, 2 packets received, 0% packet loss >> round-trip min/avg/max/stddev = 0.936/3.072/5.209/2.137 ms >> Good: Can ping DNS server >> ================================================ >> Good: DNS resolves download.cloud.com >> ================================================ >> nfs is currently mounted >> Mount point is /mnt/SecStorage/d11d5c92-6950-30e8-a0b5-3cd46a17ec14 >> Good: Can write to mount point >> ================================================ >> Management server is 10.61.153.20. Checking connectivity. >> Good: Can connect to management server port 8250 >> ================================================ >> Good: Java process is running >> ================================================ >> Tests Complete. Look for ERROR or WARNING above. >> root@s-11-VM:~# >> >> Thanks & Regards, >> Nezar. >> >> On 17 March 2016 at 09:42, Simon Weller <swel...@ena.com> wrote: >> >> > Nezar, >> > >> > Log into the Cloudstack UI. >> > >> > Click on "Infrastructure" on the left menu. Click on View All below >> > "System VMs". find the item marked "Secondary Storage VM" and then >> click >> > on the Name of that VM on the left (it will be s-XXXX-VM). You'll see a >> > bunch of information about the VM. Find the Host line and also the Link >> > Local IP Address line and then follow the original instructions in my >> > previous email. >> > >> > - Si >> > >> > >> > >> > >> > >> > ________________________________________ >> > From: Shweta Agarwal <shweta.agarw...@accelerite.com> >> > Sent: Wednesday, March 16, 2016 12:16 AM >> > To: users@cloudstack.apache.org >> > Subject: Re: Unable to create a New VM - Error message: Unable to start >> > instance due to Unable to get answer that is of class >> > com.cloud.agent.api.StartAnswer >> > >> > Hi Nezar, >> > >> > You can find host name on detail view of SSVM in UI . >> > You can also use list api command listSystemVms to get the host details >> of >> > system vms. >> > >> > Thanks >> > Shweta >> > >> > >> > ________________________________________ >> > From: Nezar Madbouh <nezar.madb...@ipscape.com.au> >> > Sent: Wednesday, March 16, 2016 9:58 AM >> > To: users@cloudstack.apache.org >> > Subject: Re: Unable to create a New VM - Error message: Unable to start >> > instance due to Unable to get answer that is of class >> > com.cloud.agent.api.StartAnswer >> > >> > Hi Simon, >> > >> > How can I find out which host the SSVM is on? >> > >> > Thanks & Regards, >> > Nezar. >> > >> > On 16 March 2016 at 12:28, Simon Weller <swel...@ena.com> wrote: >> > >> > > Can you ssh into the Secondary Storage VM? >> > > >> > > This can be done by getting the link-local-ip address from the >> > > infrastructure menu in the UI and then sshing into the host the SSVM >> is >> > on >> > > and issuing this command as root: ssh -i /root/.ssh/id_rsa.cloud -p >> 3922 >> > > root@link-local-ip (replace link-local-ip with the link local ip). >> > > >> > > Try running the SSVM diangnostic script: >> > > usr/local/cloud/systemvm/ssvm-check.sh >> > > >> > > - Si >> > > ________________________________________ >> > > From: Nezar Madbouh <nezar.madb...@ipscape.com.au> >> > > Sent: Tuesday, March 15, 2016 7:49 PM >> > > To: users@cloudstack.apache.org >> > > Subject: Re: Unable to create a New VM - Error message: Unable to >> start >> > > instance due to Unable to get answer that is of class >> > > com.cloud.agent.api.StartAnswer >> > > >> > > Hi Simon, >> > > >> > > Please see details below: >> > > >> > > >> > > mysql> use cloud; select id, name, status, type from host where id = >> 5; >> > > Reading table information for completion of table and column names >> > > You can turn off this feature to get a quicker startup with -A >> > > >> > > Database changed >> > > +----+------------------------------+--------+---------+ >> > > | id | name | status | type | >> > > +----+------------------------------+--------+---------+ >> > > | 5 | eqx-cs-cmp-02.ipscape.com.au | up | routing | >> > > +----+------------------------------+--------+---------+ >> > > 1 row in set (0.00 sec) >> > > >> > > >> > > >> > > cat /var/log/cloudstack/agent/agent.log >> > > com.cloud.utils.exception.CloudRuntimeException: >> > > com.cloud.utils.exception.CloudRuntimeException: Can't find >> > > volume:089ff3ca-9bff-402c-a14e-ea7a35aaad5c >> > > at >> > > >> > > >> > >> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getPhysicalDisk(KVMStoragePoolManager.java:262) >> > > at >> > > >> > > >> > >> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVbd(LibvirtComputingResource.java:3921) >> > > at >> > > >> > > >> > >> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:3789) >> > > at >> > > >> > > >> > >> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1333) >> > > at com.cloud.agent.Agent.processRequest(Agent.java:501) >> > > at >> > com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:808) >> > > at com.cloud.utils.nio.Task.run(Task.java:84) >> > > at >> > > >> > > >> > >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> > > at >> > > >> > > >> > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> > > at java.lang.Thread.run(Thread.java:745) >> > > agent.log.2016-03-03.gz (END) >> > > >> > > >> > > >> > > [root@eqx-cs-cmp-02 agent]# virsh pool-list >> > > Name State Autostart >> > > ----------------------------------------- >> > > 2d0967f8-50d6-4d6d-87b7-4da0596f6924 active no >> > > b3647480-5709-36eb-9ddd-e2c666f30b57 active no >> > > e3831246-ff55-4af4-9964-a17565edaccc active no >> > > >> > > >> > > >> > > >> > > [Root @ eqx-cs-cmp-02 agent] # df -h >> > > Filesystem Size Used Avail Use% Mounted on >> > > /dev/mapper/vg_root-lv_root >> > > 203G 215G 1.3G 1% / >> > > tmpfs 48g 48g 0 0% / dev / shm >> > > / Dev / sda1 477m 427m 25M 6% / boot >> > > xx.xx.xx.xx:/Cloudstack/primary >> > > 7.6T 4.1T 3.5T 54% >> > > /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57 >> > > >> > > >> > > >> > > >> > > I also noticed the following in cloudstack-agent.err >> > > >> > > libvirt: QEMU Driver error : Domain not found: no domain with matching >> > name >> > > 'i-11-600-VM' >> > > libvirt: Storage Driver error : Storage volume not found: no storage >> vol >> > > with matching name '089ff3ca-9bff-402c-a14e-ea7a35aaad5c' >> > > >> > > Your help is much appreciated, >> > > Nezar >> > > >> > > >> > > On 16 March 2016 at 11:03, Simon Weller <swel...@ena.com> wrote: >> > > >> > > > Nezar, >> > > > >> > > > ok, I just took a look at a pastebin you posted. >> > > > >> > > > I see it complaining about the inability to find a volume prior to >> > > > throwing the host exception. >> > > > >> > > > Firstly, let's find out who host 5 is. Can you log into your MySQL >> > server >> > > > and run this: >> > > > >> > > > use cloud;select id,name,status,type from host where id=5; >> > > > >> > > > Can you post the results? >> > > > >> > > > If it's a host, can you post the /var/log/cloudstack/agent/agent.log >> > from >> > > > that host? >> > > > >> > > > And also run virsh pool-list on the host in question? >> > > > >> > > > - Si >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > ________________________________________ >> > > > From: Nezar Madbouh <nezar.madb...@ipscape.com.au> >> > > > Sent: Tuesday, March 15, 2016 6:11 PM >> > > > To: users@cloudstack.apache.org >> > > > Subject: Re: Unable to create a New VM - Error message: Unable to >> start >> > > > instance due to Unable to get answer that is of class >> > > > com.cloud.agent.api.StartAnswer >> > > > >> > > > Hi Simon, >> > > > >> > > > When you say "double check your routers tied to the network", do you >> > mean >> > > > the physical routers or the virtual routers? >> > > > >> > > > When I try to create a VM in Prod Environment it fails. I can't ping >> > the >> > > > Prod virtual router from management server or Host. >> > > > >> > > > But I can ping Dev virtual router from management server and Host >> and I >> > > was >> > > > able to create a VM in Dev Environment. >> > > > >> > > > How can I troubleshoot the Prod Virtual router. I launched the >> console >> > > but >> > > > couldn't login. >> > > > >> > > > >> > > > Please note that we need to take into consideration that the >> affected >> > > > network is the Production network and we need to minimise service >> > outage >> > > as >> > > > much as possible. >> > > > >> > > > Your help is much appreciated, >> > > > Nezar >> > > > >> > > > >> > > > >> > > > On 16 March 2016 at 08:37, Simon Weller <swel...@ena.com> wrote: >> > > > >> > > > > I've seen this before where a Virtual Router has crashed and the >> > > > > management server can't communicate with it. >> > > > > Are you using an advanced network setup? >> > > > > >> > > > > If so, double check your routers tied to the network where you are >> > > > > attempting to start an instance. >> > > > > >> > > > > - Si >> > > > > >> > > > > >> > > > > ________________________________________ >> > > > > From: Nezar Madbouh <nezar.madb...@ipscape.com.au> >> > > > > Sent: Tuesday, March 15, 2016 4:02 PM >> > > > > To: users@cloudstack.apache.org >> > > > > Subject: Re: Unable to create a New VM - Error message: Unable to >> > start >> > > > > instance due to Unable to get answer that is of class >> > > > > com.cloud.agent.api.StartAnswer >> > > > > >> > > > > Hi Glenn, >> > > > > >> > > > > We have 16 hosts, all KVM. >> > > > > >> > > > > The error is not specific to Host 5. Every time I try to create a >> VM >> > it >> > > > > gives me the same error message but on different host. >> > > > > >> > > > > How do I restart the agent on the host. Is it service affecting? >> > > > > >> > > > > Thanks very much for your help. >> > > > > Neza >> > > > > >> > > > > On 15 March 2016 at 17:37, Glenn Wagner < >> glenn.wag...@shapeblue.com> >> > > > > wrote: >> > > > > >> > > > > > Hi, >> > > > > > >> > > > > > Can you explain your environment first , how many hosts , what >> is >> > > your >> > > > > > hypervisors (KVM,Xenserver,VMware) >> > > > > > >> > > > > > The error seems to be with your Host 5 Hypervisor >> > > > > > Resource [Host:5] is unreachable: >> > > > > > >> > > > > > Go to the infrastructure section and see what host5 is >> reporting, >> > > > > > Have you tried to restart the agent on the Host? >> > > > > > >> > > > > > Thanks >> > > > > > Glenn >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > [image: ShapeBlue] <http://www.shapeblue.com> >> > > > > > Glenn Wagner >> > > > > > Senior Consultant , ShapeBlue >> > > > > > d: * | s: +27 21 527 0091* >> <%7C%20s:%20+27%2021%20527%200091> | >> > > m: >> > > > > > *+27 73 917 4111* <+27%2073%20917%204111> >> > > > > > e: *glenn.wag...@shapeblue.com | t: * >> > > > > > <glenn.wag...@shapeblue.com%20%7C%20t:> | w: * >> www.shapeblue.com >> > * >> > > > > > <http://www.shapeblue.com> >> > > > > > a: >> > > > > > 2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West Cape Town >> > 7130 >> > > > > South Africa >> > > > > > Shape Blue Ltd is a company incorporated in England & Wales. >> > > ShapeBlue >> > > > > > Services India LLP is a company incorporated in India and is >> > operated >> > > > > under >> > > > > > license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda >> is >> > a >> > > > > > company incorporated in Brasil and is operated under license >> from >> > > Shape >> > > > > > Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The >> > > Republic >> > > > of >> > > > > > South Africa and is traded under license from Shape Blue Ltd. >> > > ShapeBlue >> > > > > is >> > > > > > a registered trademark. >> > > > > > This email and any attachments to it may be confidential and are >> > > > intended >> > > > > > solely for the use of the individual to whom it is addressed. >> Any >> > > views >> > > > > or >> > > > > > opinions expressed are solely those of the author and do not >> > > > necessarily >> > > > > > represent those of Shape Blue Ltd or related companies. If you >> are >> > > not >> > > > > the >> > > > > > intended recipient of this email, you must neither take any >> action >> > > > based >> > > > > > upon its contents, nor copy or show it to anyone. Please contact >> > the >> > > > > sender >> > > > > > if you believe you have received this email in error. >> > > > > > >> > > > > > >> > > > > > -----Original Message----- >> > > > > > From: Nezar Madbouh [mailto:nezar.madb...@ipscape.com.au] >> > > > > > Sent: Tuesday, 15 March 2016 5:45 AM >> > > > > > To: users@cloudstack.apache.org >> > > > > > Subject: Re: Unable to create a New VM - Error message: Unable >> to >> > > start >> > > > > > instance due to Unable to get answer that is of class >> > > > > > com.cloud.agent.api.StartAnswer >> > > > > > >> > > > > > Hi Jeremy, >> > > > > > >> > > > > > I have uploaded the contents to http://pastebin.com/1J3mmgTh >> > > > > > >> > > > > > your help is much appreciated, >> > > > > > Nezar. >> > > > > > >> > > > > > On 15 March 2016 at 13:46, Jeremy Peterson < >> jpeter...@acentek.net> >> > > > > wrote: >> > > > > > >> > > > > > > I don't see any attachments can you create a pastebin. That's >> > > easier >> > > > > > > to look through as I think attachments are removed before >> being >> > > sent >> > > > > > > to user group. >> > > > > > > >> > > > > > > Jeremy >> > > > > > > >> > > > > > > >> > > > > > > -------- Original message -------- >> > > > > > > From: Nezar Madbouh <nezar.madb...@ipscape.com.au> >> > > > > > > Date: 3/14/2016 6:38 PM (GMT-06:00) >> > > > > > > To: users@cloudstack.apache.org >> > > > > > > Subject: Unable to create a New VM - Error message: Unable to >> > start >> > > > > > > instance due to Unable to get answer that is of class >> > > > > > > com.cloud.agent.api.StartAnswer >> > > > > > > >> > > > > > > Hi Shweta and Sadhu, >> > > > > > > >> > > > > > > Please find attached managerment-server.log which contains the >> > logs >> > > > > > > since I started creating a new VM (VM id 600). >> > > > > > > >> > > > > > > Please note that we don't have agent.log files >> > > > > > > >> > > > > > > Your help is much appreciated. >> > > > > > > >> > > > > > > Regards, >> > > > > > > Nezar. >> > > > > > > >> > > > > > Find out more about ShapeBlue and our range of CloudStack >> related >> > > > > services: >> > > > > > IaaS Cloud Design & Build >> > > > > > <http://shapeblue.com/iaas-cloud-design-and-build//> | CSForge >> – >> > > rapid >> > > > > > IaaS deployment framework <http://shapeblue.com/csforge/> >> > > > > > CloudStack Consulting < >> > http://shapeblue.com/cloudstack-consultancy/> >> > > | >> > > > > CloudStack >> > > > > > Software Engineering >> > > > > > <http://shapeblue.com/cloudstack-software-engineering/> >> > > > > > CloudStack Infrastructure Support >> > > > > > <http://shapeblue.com/cloudstack-infrastructure-support/> | >> > > CloudStack >> > > > > > Bootcamp Training Courses < >> > http://shapeblue.com/cloudstack-training/ >> > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> > >> > >> > DISCLAIMER >> > ========== >> > This e-mail may contain privileged and confidential information which is >> > the property of Accelerite, a Persistent Systems business. It is >> intended >> > only for the use of the individual or entity to which it is addressed. >> If >> > you are not the intended recipient, you are not authorized to read, >> retain, >> > copy, print, distribute or use this message. If you have received this >> > communication in error, please notify the sender and delete all copies >> of >> > this message. Accelerite, a Persistent Systems business does not accept >> any >> > liability for virus infected mails. >> > >> > >
