Hi Ilya, I shutdown all the VMs in Pre-prod Environmnet expecting the Pre-prod Virtual router to be deleted automatically because no VM is running on this vlan but it wasn't deleted.
Am I right to expect the VR to be deleted automatically if no VM is running in that vlan? Thanks, Nezar. On 15 April 2016 at 09:53, Nezar Madbouh <nezar.madb...@ipscape.com.au> wrote: > Hi Ilya, > > I have run the script and it didn't return any results: > > [root@eqx-cs-mgt-01 ~]# ./async_jobs.sh > [root@eqx-cs-mgt-01 ~]# > > > I also ran the command inside mysql: > > mysql> SELECT id, user_id, account_id, instance_type, instance_id, > job_cmd, job_cmd_info, job_dispatcher, job_pending_signals FROM > cloud.async_job where job_status=0 and job_dispatcher not like > "pseudoJobDispatcher"; > Empty set (0.00 sec) > > your help is much appreciated, > Nezar. > > > On 14 April 2016 at 16:49, ilya <ilya.mailing.li...@gmail.com> wrote: > >> It appears you have pending jobs that are not letting you go through. >> >> Usually stop and starting management service should be enough. >> >> However, try running this script from your management server to show us >> what you have running in async_job table. >> >> >> >> > #!/bin/bash >> > DATESTAMP=$(date +%m%d%y-%H%M%S) >> > DBPASS=$(java -classpath >> /usr/share/cloudstack-common/lib/jasypt-1.9.0.jar >> org.jasypt.intf.cli.JasyptPBEStringDecryptionCLI input="$(cat >> /etc/cloudstack/management/db.properties | grep db.cloud.password | awk >> -F'(' '{print $2}' | sed 's/)//g')" password="$(cat >> /etc/cloudstack/management/key)" | grep -A2 OUTPUT | tail -1) >> > DBHOST=$(cat /etc/cloudstack/management/db.properties | grep >> db.cloud.host | awk -F'=' '{print $2}' | tail -1 ) >> > DBUSER=$(cat /etc/cloudstack/management/db.properties | grep >> db.cloud.username | awk -F'=' '{print $2}') >> > DB=$(cat /etc/cloudstack/management/db.properties | grep db.cloud.name >> | awk -F'=' '{print $2}') >> > DBPORT=$(cat /etc/cloudstack/management/db.properties | grep >> db.cloud.port | awk -F'=' '{print $2}') >> > MYSQLCMD="mysql -h $DBHOST -u $DBUSER -P $DBPORT -p$DBPASS $DB" >> > echo 'SELECT id, user_id, account_id, instance_type, instance_id, >> job_cmd, job_cmd_info, job_dispatcher, job_pending_signals FROM >> cloud.async_job where job_status=0 and job_dispatcher not like >> "pseudoJobDispatcher"' | $MYSQLCMD >> >> >> >> >> On 4/13/16 5:16 PM, Nezar Madbouh wrote: >> > Hi Simon, >> > >> > The following work was done but unfortunately the problem is not fixed: >> > >> > >> > on management server: >> > >> > service cloudstack-management stop >> > >> > >> > on the host: >> > >> > service cloudstack-agent stop >> > >> > service libvirtd restart >> > >> > service cloudstack-agent start >> > >> > >> > on management server: >> > >> > service cloudstack-management start >> > >> > Regards, >> > Nezar. >> > >> > On 24 March 2016 at 05:06, Simon Weller <swel...@ena.com> wrote: >> > >> >> Yes, I think that order is fine. Make sure the processes are indeed >> >> shutdown before moving on to the next step. >> >> >> >> >> >> >> >> >> >> >> >> ________________________________________ >> >> From: Nezar Madbouh <nezar.madb...@ipscape.com.au> >> >> Sent: Tuesday, March 22, 2016 5:33 PM >> >> To: users@cloudstack.apache.org >> >> Subject: Re: Unable to create a New VM - Error message: Unable to start >> >> instance due to Unable to get answer that is of class >> >> com.cloud.agent.api.StartAnswer >> >> >> >> Hi Simon, >> >> >> >> Can you please check if the following sequence of action is correct: >> >> >> >> on management server: >> >> >> >> service cloudstack-management stop >> >> >> >> >> >> on the host: >> >> >> >> service cloudstack-agent stop >> >> >> >> service libvirtd restart >> >> >> >> service cloudstack-agent start >> >> >> >> >> >> on management server: >> >> >> >> service cloudstack-management start >> >> >> >> Thanks very much for your help. >> >> Nezar. >> >> >> >> On 23 March 2016 at 08:39, Simon Weller <swel...@ena.com> wrote: >> >> >> >>> Shutting down all the management services should clear the jobs. >> >>> >> >>> >> >>> ________________________________________ >> >>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au> >> >>> Sent: Tuesday, March 22, 2016 4:28 PM >> >>> To: users@cloudstack.apache.org >> >>> Subject: Re: Unable to create a New VM - Error message: Unable to >> start >> >>> instance due to Unable to get answer that is of class >> >>> com.cloud.agent.api.StartAnswer >> >>> >> >>> Hi Simon, >> >>> >> >>> I will plan to restart agents, libvirtd and management services. >> >>> >> >>> How do I clear hung jobs. >> >>> >> >>> Your help is much appreciated, >> >>> Nezar. >> >>> >> >>> On 22 March 2016 at 23:03, Simon Weller <swel...@ena.com> wrote: >> >>> >> >>>> Nezar, >> >>>> >> >>>> I've looked through the code related to your release and it seems >> that >> >> in >> >>>> order to throw this exception, 10 attempts have been made to retrieve >> >> the >> >>>> disk information prior to the exception being thrown. >> >>>> >> >>>> It appears there is something really odd going on here. I'm starting >> >> to >> >>>> wonder whether it might be worth while for you to stop all of your >> >>> agents, >> >>>> plus management server processes, restart libvirtd on each of the >> hosts >> >>> and >> >>>> then bring the agents back up prior to starting the management >> servers. >> >>>> >> >>>> There are a bunch of logs the pastebin you posted that seems to >> >> indicate >> >>>> you have some jobs hung, for example: >> >>>> >> >>>> 2016-03-14 21:45:31,828 WARN [o.a.c.f.j.i.AsyncJobMonitor] >> >>>> (Timer-1:ctx-0c090bd7) Task (job-9066) has been pending for 1660851 >> >>> seconds >> >>>> 2016-03-14 21:45:31,828 WARN [o.a.c.f.j.i.AsyncJobMonitor] >> >>>> (Timer-1:ctx-0c090bd7) Task (job-8133) has been pending for 4253003 >> >>> seconds >> >>>> >> >>>> Restarting the agents, libvirtd and management will not cause an >> >> outage >> >>>> to your cloud, but I would suggest, if you decided to do this, you >> >> should >> >>>> complete it in a maintenance window. >> >>>> >> >>>> - Si >> >>>> >> >>>> ________________________________________ >> >>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au> >> >>>> Sent: Sunday, March 20, 2016 10:27 PM >> >>>> To: users@cloudstack.apache.org >> >>>> Subject: Re: Unable to create a New VM - Error message: Unable to >> start >> >>>> instance due to Unable to get answer that is of class >> >>>> com.cloud.agent.api.StartAnswer >> >>>> >> >>>> Hi Simon, >> >>>> >> >>>> Please see answers to your questions below. I am not sure if I >> provided >> >>> the >> >>>> correct answer for the primary and secondary storage configuration. >> >>> Please >> >>>> let me know if this is not what you are after: >> >>>> >> >>>> >> >>>> Can you screenshot and post somewhere your secondary storage >> >>> configuration >> >>>> and your primary storage configuration from the UI with the details? >> >>>> >> >>>> mysql> select * from configuration where name like '%secondary%'\G >> >>>> *************************** 1. row *************************** >> >>>> category: Account Defaults >> >>>> instance: DEFAULT >> >>>> component: management-server >> >>>> name: max.account.secondary.storage >> >>>> value: 400 >> >>>> description: The default maximum secondary storage space (in GiB) >> >> that >> >>>> can be used for an account >> >>>> default_value: 400 >> >>>> updated: NULL >> >>>> scope: NULL >> >>>> is_dynamic: 0 >> >>>> *************************** 2. row *************************** >> >>>> category: Project Defaults >> >>>> instance: DEFAULT >> >>>> component: management-server >> >>>> name: max.project.secondary.storage >> >>>> value: 400 >> >>>> description: The default maximum secondary storage space (in GiB) >> >> that >> >>>> can be used for a project >> >>>> default_value: 400 >> >>>> updated: NULL >> >>>> scope: NULL >> >>>> is_dynamic: 0 >> >>>> *************************** 3. row *************************** >> >>>> category: Hidden >> >>>> instance: DEFAULT >> >>>> component: management-server >> >>>> name: secondary.storage.vm >> >>>> value: +rwRhJOXPymbQOuAMLnhiw== >> >>>> description: Deploys a VM per zone to manage secondary storage if >> >> true, >> >>>> otherwise secondary storage is mounted on management server >> >>>> default_value: NULL >> >>>> updated: NULL >> >>>> scope: NULL >> >>>> is_dynamic: 0 >> >>>> *************************** 4. row *************************** >> >>>> category: Advanced >> >>>> instance: DEFAULT >> >>>> component: ExtensionRegistry >> >>>> name: secondary.storage.vm.allocators.exclude >> >>>> value: NULL >> >>>> description: Extensions to exclude from being registered >> >>>> default_value: NULL >> >>>> updated: 2014-12-04 00:18:15 >> >>>> scope: Global >> >>>> is_dynamic: 0 >> >>>> *************************** 5. row *************************** >> >>>> category: NetworkManager >> >>>> instance: DEFAULT >> >>>> component: management-server >> >>>> name: vm.network.nic.max.secondary.ipaddresses >> >>>> value: NULL >> >>>> description: Specify the number of secondary ip addresses per nic >> per >> >>> vm >> >>>> default_value: 256 >> >>>> updated: NULL >> >>>> scope: NULL >> >>>> is_dynamic: 0 >> >>>> 5 rows in set (0.00 sec) >> >>>> >> >>>> >> >>>> mysql> select * from configuration where name like '%primary%'\G >> >>>> *************************** 1. row *************************** >> >>>> category: Account Defaults >> >>>> instance: DEFAULT >> >>>> component: management-server >> >>>> name: max.account.primary.storage >> >>>> value: 200 >> >>>> description: The default maximum primary storage space (in GiB) >> that >> >>> can >> >>>> be used for an account >> >>>> default_value: 200 >> >>>> updated: NULL >> >>>> scope: NULL >> >>>> is_dynamic: 0 >> >>>> *************************** 2. row *************************** >> >>>> category: Project Defaults >> >>>> instance: DEFAULT >> >>>> component: management-server >> >>>> name: max.project.primary.storage >> >>>> value: 200 >> >>>> description: The default maximum primary storage space (in GiB) >> that >> >>> can >> >>>> be used for a project >> >>>> default_value: 200 >> >>>> updated: NULL >> >>>> scope: NULL >> >>>> is_dynamic: 0 >> >>>> *************************** 3. row *************************** >> >>>> category: Storage >> >>>> instance: DEFAULT >> >>>> component: TemplateManager >> >>>> name: primary.storage.download.wait >> >>>> value: 10800 >> >>>> description: In second, timeout for download template to primary >> >>> storage >> >>>> default_value: NULL >> >>>> updated: NULL >> >>>> scope: NULL >> >>>> is_dynamic: 0 >> >>>> 3 rows in set (0.00 sec) >> >>>> >> >>>> >> >>>> >> >>>> Also, what is the Cloudstack version you are running and what is the >> OS >> >>>> you're running it on? >> >>>> >> >>>> Cloudstack 4.4.1 >> >>>> >> >>>> Centos 6.6 >> >>>> >> >>>> Are you using actimeo=0 in your NFS mount line? >> >>>> >> >>>> Primary storage on Hypervisors has the following parameters >> >>>> 10.61.154.10:/Cloudstack/primary on >> >>>> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57 type nfs >> >>>> (rw,noatime,nfsvers=3,proto=tcp,hard,nolock,intr,addr=10.61.154.10) >> >>>> >> >>>> >> >>>> Secondary storage on SSVM has the following parameters >> >>>> 10.61.154.10:/Cloudstack/secondary on >> >>>> /mnt/SecStorage/d11d5c92-6950-30e8-a0b5-3cd46a17ec14 type nfs4 >> >>>> >> >>>> >> >>> >> >> >> (rw,relatime,vers=4,rsize=262144,wsize=262144,namlen=255,acdirmin=0,acdirmax=0,soft,proto=tcp,port=0,timeo=133,retrans= >> 2147483647 >> ,sec=sys,clientaddr=10.61.154.70,minorversion=0,local_lock=none,addr=10.61.154.10) >> >>>> >> >>>> >> >>>> Thanks & Regards, >> >>>> Nezar. >> >>>> >> >>>> On 18 March 2016 at 23:50, Simon Weller <swel...@ena.com> wrote: >> >>>> >> >>>>> Also, what is the Cloudstack version you are running and what is the >> >> OS >> >>>>> you're running it on? >> >>>>> >> >>>>> Are you using actimeo=0 in your NFS mount line? >> >>>>> >> >>>>> ________________________________________ >> >>>>> From: Simon Weller <swel...@ena.com> >> >>>>> Sent: Friday, March 18, 2016 7:40 AM >> >>>>> To: users@cloudstack.apache.org >> >>>>> Subject: Re: Unable to create a New VM - Error message: Unable to >> >> start >> >>>>> instance due to Unable to get answer that is of class >> >>>>> com.cloud.agent.api.StartAnswer >> >>>>> >> >>>>> Nexar, >> >>>>> >> >>>>> That's normal behaviour. You can only ssh to the link local on the >> >> host >> >>>>> where the VM resides. >> >>>>> >> >>>>> Can you screenshot and post somewhere your secondary storage >> >>>> configuration >> >>>>> and your primary storage configuration from the UI with the details? >> >>>>> >> >>>>> - Si >> >>>>> >> >>>>> ________________________________________ >> >>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au> >> >>>>> Sent: Thursday, March 17, 2016 8:19 PM >> >>>>> To: users@cloudstack.apache.org >> >>>>> Subject: Re: Unable to create a New VM - Error message: Unable to >> >> start >> >>>>> instance due to Unable to get answer that is of class >> >>>>> com.cloud.agent.api.StartAnswer >> >>>>> >> >>>>> Hi Simon, >> >>>>> >> >>>>> I just noticed that from other Hosts I can't ssh into the SSVM: >> >>>>> >> >>>>> [root@eqx-cs-cmp-01 ~]# ssh -i /root/.ssh/id_rsa.cloud -p 3922 >> >>>>> root@169.254.3.161 >> >>>>> ssh: connect to host 169.254.3.161 port 3922: No route to host >> >>>>> >> >>>>> >> >>>>> I can only ssh from one Host: >> >>>>> [root@eqx-cs-cmp-06 ~]# ssh -i /root/.ssh/id_rsa.cloud -p 3922 >> >>>>> root@169.254.3.161 >> >>>>> Linux s-11-VM 3.2.0-4-amd64 #1 SMP Debian 3.2.57-3 x86_64 >> >>>>> >> >>>>> Thanks & Regards, >> >>>>> Nezar. >> >>>>> >> >>>>> On 18 March 2016 at 10:36, Nezar Madbouh < >> >> nezar.madb...@ipscape.com.au >> >>>> >> >>>>> wrote: >> >>>>> >> >>>>>> Hi Simon, >> >>>>>> >> >>>>>> Yes existing VMs are successfully running. Yesterday I was able to >> >>>> create >> >>>>>> a VM in Dev Environment but I can't create VMs in Prod environment. >> >>>>>> >> >>>>>> Thanks, >> >>>>>> Nezar. >> >>>>>> >> >>>>>> On 18 March 2016 at 09:12, Simon Weller <swel...@ena.com> wrote: >> >>>>>> >> >>>>>>> I'm assuming you have other VMs successfully running, correct? So >> >>> you >> >>>>> can >> >>>>>>> write to your primary storage. >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> ________________________________________ >> >>>>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au> >> >>>>>>> Sent: Thursday, March 17, 2016 4:43 PM >> >>>>>>> To: users@cloudstack.apache.org >> >>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to >> >>>> start >> >>>>>>> instance due to Unable to get answer that is of class >> >>>>>>> com.cloud.agent.api.StartAnswer >> >>>>>>> >> >>>>>>> Hi Simon and Shweta, >> >>>>>>> >> >>>>>>> Please see the result below: >> >>>>>>> >> >>>>>>> root@s-11-VM:~# /usr/local/cloud/systemvm/ssvm-check.sh >> >>>>>>> ================================================ >> >>>>>>> First DNS server is 8.8.8.8 >> >>>>>>> PING 8.8.8.8 (8.8.8.8): 48 data bytes >> >>>>>>> 56 bytes from 8.8.8.8: icmp_seq=0 ttl=57 time=5.209 ms >> >>>>>>> 56 bytes from 8.8.8.8: icmp_seq=1 ttl=57 time=0.936 ms >> >>>>>>> --- 8.8.8.8 ping statistics --- >> >>>>>>> 2 packets transmitted, 2 packets received, 0% packet loss >> >>>>>>> round-trip min/avg/max/stddev = 0.936/3.072/5.209/2.137 ms >> >>>>>>> Good: Can ping DNS server >> >>>>>>> ================================================ >> >>>>>>> Good: DNS resolves download.cloud.com >> >>>>>>> ================================================ >> >>>>>>> nfs is currently mounted >> >>>>>>> Mount point is >> >> /mnt/SecStorage/d11d5c92-6950-30e8-a0b5-3cd46a17ec14 >> >>>>>>> Good: Can write to mount point >> >>>>>>> ================================================ >> >>>>>>> Management server is 10.61.153.20. Checking connectivity. >> >>>>>>> Good: Can connect to management server port 8250 >> >>>>>>> ================================================ >> >>>>>>> Good: Java process is running >> >>>>>>> ================================================ >> >>>>>>> Tests Complete. Look for ERROR or WARNING above. >> >>>>>>> root@s-11-VM:~# >> >>>>>>> >> >>>>>>> Thanks & Regards, >> >>>>>>> Nezar. >> >>>>>>> >> >>>>>>> On 17 March 2016 at 09:42, Simon Weller <swel...@ena.com> wrote: >> >>>>>>> >> >>>>>>>> Nezar, >> >>>>>>>> >> >>>>>>>> Log into the Cloudstack UI. >> >>>>>>>> >> >>>>>>>> Click on "Infrastructure" on the left menu. Click on View All >> >>> below >> >>>>>>>> "System VMs". find the item marked "Secondary Storage VM" and >> >>> then >> >>>>>>> click >> >>>>>>>> on the Name of that VM on the left (it will be s-XXXX-VM). >> >> You'll >> >>>> see >> >>>>> a >> >>>>>>>> bunch of information about the VM. Find the Host line and also >> >> the >> >>>>> Link >> >>>>>>>> Local IP Address line and then follow the original instructions >> >> in >> >>>> my >> >>>>>>>> previous email. >> >>>>>>>> >> >>>>>>>> - Si >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> ________________________________________ >> >>>>>>>> From: Shweta Agarwal <shweta.agarw...@accelerite.com> >> >>>>>>>> Sent: Wednesday, March 16, 2016 12:16 AM >> >>>>>>>> To: users@cloudstack.apache.org >> >>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable >> >> to >> >>>>> start >> >>>>>>>> instance due to Unable to get answer that is of class >> >>>>>>>> com.cloud.agent.api.StartAnswer >> >>>>>>>> >> >>>>>>>> Hi Nezar, >> >>>>>>>> >> >>>>>>>> You can find host name on detail view of SSVM in UI . >> >>>>>>>> You can also use list api command listSystemVms to get the host >> >>>>> details >> >>>>>>> of >> >>>>>>>> system vms. >> >>>>>>>> >> >>>>>>>> Thanks >> >>>>>>>> Shweta >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> ________________________________________ >> >>>>>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au> >> >>>>>>>> Sent: Wednesday, March 16, 2016 9:58 AM >> >>>>>>>> To: users@cloudstack.apache.org >> >>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable >> >> to >> >>>>> start >> >>>>>>>> instance due to Unable to get answer that is of class >> >>>>>>>> com.cloud.agent.api.StartAnswer >> >>>>>>>> >> >>>>>>>> Hi Simon, >> >>>>>>>> >> >>>>>>>> How can I find out which host the SSVM is on? >> >>>>>>>> >> >>>>>>>> Thanks & Regards, >> >>>>>>>> Nezar. >> >>>>>>>> >> >>>>>>>> On 16 March 2016 at 12:28, Simon Weller <swel...@ena.com> >> >> wrote: >> >>>>>>>> >> >>>>>>>>> Can you ssh into the Secondary Storage VM? >> >>>>>>>>> >> >>>>>>>>> This can be done by getting the link-local-ip address from the >> >>>>>>>>> infrastructure menu in the UI and then sshing into the host >> >> the >> >>>> SSVM >> >>>>>>> is >> >>>>>>>> on >> >>>>>>>>> and issuing this command as root: ssh -i >> >> /root/.ssh/id_rsa.cloud >> >>>> -p >> >>>>>>> 3922 >> >>>>>>>>> root@link-local-ip (replace link-local-ip with the link >> >> local >> >>>> ip). >> >>>>>>>>> >> >>>>>>>>> Try running the SSVM diangnostic script: >> >>>>>>>>> usr/local/cloud/systemvm/ssvm-check.sh >> >>>>>>>>> >> >>>>>>>>> - Si >> >>>>>>>>> ________________________________________ >> >>>>>>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au> >> >>>>>>>>> Sent: Tuesday, March 15, 2016 7:49 PM >> >>>>>>>>> To: users@cloudstack.apache.org >> >>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable >> >>> to >> >>>>>>> start >> >>>>>>>>> instance due to Unable to get answer that is of class >> >>>>>>>>> com.cloud.agent.api.StartAnswer >> >>>>>>>>> >> >>>>>>>>> Hi Simon, >> >>>>>>>>> >> >>>>>>>>> Please see details below: >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> mysql> use cloud; select id, name, status, type from host >> >> where >> >>>> id = >> >>>>>>> 5; >> >>>>>>>>> Reading table information for completion of table and column >> >>> names >> >>>>>>>>> You can turn off this feature to get a quicker startup with -A >> >>>>>>>>> >> >>>>>>>>> Database changed >> >>>>>>>>> +----+------------------------------+--------+---------+ >> >>>>>>>>> | id | name | status | type | >> >>>>>>>>> +----+------------------------------+--------+---------+ >> >>>>>>>>> | 5 | eqx-cs-cmp-02.ipscape.com.au | up | routing | >> >>>>>>>>> +----+------------------------------+--------+---------+ >> >>>>>>>>> 1 row in set (0.00 sec) >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> cat /var/log/cloudstack/agent/agent.log >> >>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: >> >>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Can't find >> >>>>>>>>> volume:089ff3ca-9bff-402c-a14e-ea7a35aaad5c >> >>>>>>>>> at >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>> >> >>>>>>> >> >>>>> >> >>>> >> >>> >> >> >> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getPhysicalDisk(KVMStoragePoolManager.java:262) >> >>>>>>>>> at >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>> >> >>>>>>> >> >>>>> >> >>>> >> >>> >> >> >> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVbd(LibvirtComputingResource.java:3921) >> >>>>>>>>> at >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>> >> >>>>>>> >> >>>>> >> >>>> >> >>> >> >> >> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:3789) >> >>>>>>>>> at >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>> >> >>>>>>> >> >>>>> >> >>>> >> >>> >> >> >> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1333) >> >>>>>>>>> at >> >> com.cloud.agent.Agent.processRequest(Agent.java:501) >> >>>>>>>>> at >> >>>>>>>> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:808) >> >>>>>>>>> at com.cloud.utils.nio.Task.run(Task.java:84) >> >>>>>>>>> at >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>> >> >>>>>>> >> >>>>> >> >>>> >> >>> >> >> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> >>>>>>>>> at >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>> >> >>>>>>> >> >>>>> >> >>>> >> >>> >> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> >>>>>>>>> at java.lang.Thread.run(Thread.java:745) >> >>>>>>>>> agent.log.2016-03-03.gz (END) >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> [root@eqx-cs-cmp-02 agent]# virsh pool-list >> >>>>>>>>> Name State Autostart >> >>>>>>>>> ----------------------------------------- >> >>>>>>>>> 2d0967f8-50d6-4d6d-87b7-4da0596f6924 active no >> >>>>>>>>> b3647480-5709-36eb-9ddd-e2c666f30b57 active no >> >>>>>>>>> e3831246-ff55-4af4-9964-a17565edaccc active no >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> [Root @ eqx-cs-cmp-02 agent] # df -h >> >>>>>>>>> Filesystem Size Used Avail Use% Mounted on >> >>>>>>>>> /dev/mapper/vg_root-lv_root >> >>>>>>>>> 203G 215G 1.3G 1% / >> >>>>>>>>> tmpfs 48g 48g 0 0% / dev / shm >> >>>>>>>>> / Dev / sda1 477m 427m 25M 6% / boot >> >>>>>>>>> xx.xx.xx.xx:/Cloudstack/primary >> >>>>>>>>> 7.6T 4.1T 3.5T 54% >> >>>>>>>>> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57 >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> I also noticed the following in cloudstack-agent.err >> >>>>>>>>> >> >>>>>>>>> libvirt: QEMU Driver error : Domain not found: no domain with >> >>>>> matching >> >>>>>>>> name >> >>>>>>>>> 'i-11-600-VM' >> >>>>>>>>> libvirt: Storage Driver error : Storage volume not found: no >> >>>> storage >> >>>>>>> vol >> >>>>>>>>> with matching name '089ff3ca-9bff-402c-a14e-ea7a35aaad5c' >> >>>>>>>>> >> >>>>>>>>> Your help is much appreciated, >> >>>>>>>>> Nezar >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> On 16 March 2016 at 11:03, Simon Weller <swel...@ena.com> >> >>> wrote: >> >>>>>>>>> >> >>>>>>>>>> Nezar, >> >>>>>>>>>> >> >>>>>>>>>> ok, I just took a look at a pastebin you posted. >> >>>>>>>>>> >> >>>>>>>>>> I see it complaining about the inability to find a volume >> >>> prior >> >>>> to >> >>>>>>>>>> throwing the host exception. >> >>>>>>>>>> >> >>>>>>>>>> Firstly, let's find out who host 5 is. Can you log into your >> >>>> MySQL >> >>>>>>>> server >> >>>>>>>>>> and run this: >> >>>>>>>>>> >> >>>>>>>>>> use cloud;select id,name,status,type from host where id=5; >> >>>>>>>>>> >> >>>>>>>>>> Can you post the results? >> >>>>>>>>>> >> >>>>>>>>>> If it's a host, can you post the >> >>>>> /var/log/cloudstack/agent/agent.log >> >>>>>>>> from >> >>>>>>>>>> that host? >> >>>>>>>>>> >> >>>>>>>>>> And also run virsh pool-list on the host in question? >> >>>>>>>>>> >> >>>>>>>>>> - Si >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> ________________________________________ >> >>>>>>>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au> >> >>>>>>>>>> Sent: Tuesday, March 15, 2016 6:11 PM >> >>>>>>>>>> To: users@cloudstack.apache.org >> >>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: >> >> Unable >> >>>> to >> >>>>>>> start >> >>>>>>>>>> instance due to Unable to get answer that is of class >> >>>>>>>>>> com.cloud.agent.api.StartAnswer >> >>>>>>>>>> >> >>>>>>>>>> Hi Simon, >> >>>>>>>>>> >> >>>>>>>>>> When you say "double check your routers tied to the >> >> network", >> >>> do >> >>>>> you >> >>>>>>>> mean >> >>>>>>>>>> the physical routers or the virtual routers? >> >>>>>>>>>> >> >>>>>>>>>> When I try to create a VM in Prod Environment it fails. I >> >>> can't >> >>>>> ping >> >>>>>>>> the >> >>>>>>>>>> Prod virtual router from management server or Host. >> >>>>>>>>>> >> >>>>>>>>>> But I can ping Dev virtual router from management server and >> >>>> Host >> >>>>>>> and I >> >>>>>>>>> was >> >>>>>>>>>> able to create a VM in Dev Environment. >> >>>>>>>>>> >> >>>>>>>>>> How can I troubleshoot the Prod Virtual router. I launched >> >> the >> >>>>>>> console >> >>>>>>>>> but >> >>>>>>>>>> couldn't login. >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> Please note that we need to take into consideration that the >> >>>>>>> affected >> >>>>>>>>>> network is the Production network and we need to minimise >> >>>> service >> >>>>>>>> outage >> >>>>>>>>> as >> >>>>>>>>>> much as possible. >> >>>>>>>>>> >> >>>>>>>>>> Your help is much appreciated, >> >>>>>>>>>> Nezar >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> On 16 March 2016 at 08:37, Simon Weller <swel...@ena.com> >> >>>> wrote: >> >>>>>>>>>> >> >>>>>>>>>>> I've seen this before where a Virtual Router has crashed >> >> and >> >>>> the >> >>>>>>>>>>> management server can't communicate with it. >> >>>>>>>>>>> Are you using an advanced network setup? >> >>>>>>>>>>> >> >>>>>>>>>>> If so, double check your routers tied to the network where >> >>> you >> >>>>> are >> >>>>>>>>>>> attempting to start an instance. >> >>>>>>>>>>> >> >>>>>>>>>>> - Si >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> ________________________________________ >> >>>>>>>>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au> >> >>>>>>>>>>> Sent: Tuesday, March 15, 2016 4:02 PM >> >>>>>>>>>>> To: users@cloudstack.apache.org >> >>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: >> >>> Unable >> >>>>> to >> >>>>>>>> start >> >>>>>>>>>>> instance due to Unable to get answer that is of class >> >>>>>>>>>>> com.cloud.agent.api.StartAnswer >> >>>>>>>>>>> >> >>>>>>>>>>> Hi Glenn, >> >>>>>>>>>>> >> >>>>>>>>>>> We have 16 hosts, all KVM. >> >>>>>>>>>>> >> >>>>>>>>>>> The error is not specific to Host 5. Every time I try to >> >>>> create >> >>>>> a >> >>>>>>> VM >> >>>>>>>> it >> >>>>>>>>>>> gives me the same error message but on different host. >> >>>>>>>>>>> >> >>>>>>>>>>> How do I restart the agent on the host. Is it service >> >>>> affecting? >> >>>>>>>>>>> >> >>>>>>>>>>> Thanks very much for your help. >> >>>>>>>>>>> Neza >> >>>>>>>>>>> >> >>>>>>>>>>> On 15 March 2016 at 17:37, Glenn Wagner < >> >>>>>>> glenn.wag...@shapeblue.com> >> >>>>>>>>>>> wrote: >> >>>>>>>>>>> >> >>>>>>>>>>>> Hi, >> >>>>>>>>>>>> >> >>>>>>>>>>>> Can you explain your environment first , how many hosts >> >> , >> >>>> what >> >>>>>>> is >> >>>>>>>>> your >> >>>>>>>>>>>> hypervisors (KVM,Xenserver,VMware) >> >>>>>>>>>>>> >> >>>>>>>>>>>> The error seems to be with your Host 5 Hypervisor >> >>>>>>>>>>>> Resource [Host:5] is unreachable: >> >>>>>>>>>>>> >> >>>>>>>>>>>> Go to the infrastructure section and see what host5 is >> >>>>>>> reporting, >> >>>>>>>>>>>> Have you tried to restart the agent on the Host? >> >>>>>>>>>>>> >> >>>>>>>>>>>> Thanks >> >>>>>>>>>>>> Glenn >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> [image: ShapeBlue] <http://www.shapeblue.com> >> >>>>>>>>>>>> Glenn Wagner >> >>>>>>>>>>>> Senior Consultant , ShapeBlue >> >>>>>>>>>>>> d: * | s: +27 21 527 0091* >> >>>>>>> <%7C%20s:%20+27%2021%20527%200091> | >> >>>>>>>>> m: >> >>>>>>>>>>>> *+27 73 917 4111* <+27%2073%20917%204111> >> >>>>>>>>>>>> e: *glenn.wag...@shapeblue.com | t: * >> >>>>>>>>>>>> <glenn.wag...@shapeblue.com%20%7C%20t:> | w: * >> >>>>>>> www.shapeblue.com >> >>>>>>>> * >> >>>>>>>>>>>> <http://www.shapeblue.com> >> >>>>>>>>>>>> a: >> >>>>>>>>>>>> 2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West >> >>> Cape >> >>>>> Town >> >>>>>>>> 7130 >> >>>>>>>>>>> South Africa >> >>>>>>>>>>>> Shape Blue Ltd is a company incorporated in England & >> >>> Wales. >> >>>>>>>>> ShapeBlue >> >>>>>>>>>>>> Services India LLP is a company incorporated in India >> >> and >> >>> is >> >>>>>>>> operated >> >>>>>>>>>>> under >> >>>>>>>>>>>> license from Shape Blue Ltd. Shape Blue Brasil >> >> Consultoria >> >>>>> Ltda >> >>>>>>> is >> >>>>>>>> a >> >>>>>>>>>>>> company incorporated in Brasil and is operated under >> >>> license >> >>>>>>> from >> >>>>>>>>> Shape >> >>>>>>>>>>>> Blue Ltd. ShapeBlue SA Pty Ltd is a company registered >> >> by >> >>>> The >> >>>>>>>>> Republic >> >>>>>>>>>> of >> >>>>>>>>>>>> South Africa and is traded under license from Shape Blue >> >>>> Ltd. >> >>>>>>>>> ShapeBlue >> >>>>>>>>>>> is >> >>>>>>>>>>>> a registered trademark. >> >>>>>>>>>>>> This email and any attachments to it may be confidential >> >>> and >> >>>>> are >> >>>>>>>>>> intended >> >>>>>>>>>>>> solely for the use of the individual to whom it is >> >>>> addressed. >> >>>>>>> Any >> >>>>>>>>> views >> >>>>>>>>>>> or >> >>>>>>>>>>>> opinions expressed are solely those of the author and do >> >>> not >> >>>>>>>>>> necessarily >> >>>>>>>>>>>> represent those of Shape Blue Ltd or related companies. >> >> If >> >>>> you >> >>>>>>> are >> >>>>>>>>> not >> >>>>>>>>>>> the >> >>>>>>>>>>>> intended recipient of this email, you must neither take >> >>> any >> >>>>>>> action >> >>>>>>>>>> based >> >>>>>>>>>>>> upon its contents, nor copy or show it to anyone. Please >> >>>>> contact >> >>>>>>>> the >> >>>>>>>>>>> sender >> >>>>>>>>>>>> if you believe you have received this email in error. >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> -----Original Message----- >> >>>>>>>>>>>> From: Nezar Madbouh [mailto: >> >> nezar.madb...@ipscape.com.au] >> >>>>>>>>>>>> Sent: Tuesday, 15 March 2016 5:45 AM >> >>>>>>>>>>>> To: users@cloudstack.apache.org >> >>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: >> >>>> Unable >> >>>>>>> to >> >>>>>>>>> start >> >>>>>>>>>>>> instance due to Unable to get answer that is of class >> >>>>>>>>>>>> com.cloud.agent.api.StartAnswer >> >>>>>>>>>>>> >> >>>>>>>>>>>> Hi Jeremy, >> >>>>>>>>>>>> >> >>>>>>>>>>>> I have uploaded the contents to >> >>>> http://pastebin.com/1J3mmgTh >> >>>>>>>>>>>> >> >>>>>>>>>>>> your help is much appreciated, >> >>>>>>>>>>>> Nezar. >> >>>>>>>>>>>> >> >>>>>>>>>>>> On 15 March 2016 at 13:46, Jeremy Peterson < >> >>>>>>> jpeter...@acentek.net> >> >>>>>>>>>>> wrote: >> >>>>>>>>>>>> >> >>>>>>>>>>>>> I don't see any attachments can you create a pastebin. >> >>>>> That's >> >>>>>>>>> easier >> >>>>>>>>>>>>> to look through as I think attachments are removed >> >>> before >> >>>>>>> being >> >>>>>>>>> sent >> >>>>>>>>>>>>> to user group. >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> Jeremy >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> -------- Original message -------- >> >>>>>>>>>>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au> >> >>>>>>>>>>>>> Date: 3/14/2016 6:38 PM (GMT-06:00) >> >>>>>>>>>>>>> To: users@cloudstack.apache.org >> >>>>>>>>>>>>> Subject: Unable to create a New VM - Error message: >> >>> Unable >> >>>>> to >> >>>>>>>> start >> >>>>>>>>>>>>> instance due to Unable to get answer that is of class >> >>>>>>>>>>>>> com.cloud.agent.api.StartAnswer >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> Hi Shweta and Sadhu, >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> Please find attached managerment-server.log which >> >>> contains >> >>>>> the >> >>>>>>>> logs >> >>>>>>>>>>>>> since I started creating a new VM (VM id 600). >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> Please note that we don't have agent.log files >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> Your help is much appreciated. >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> Regards, >> >>>>>>>>>>>>> Nezar. >> >>>>>>>>>>>>> >> >>>>>>>>>>>> Find out more about ShapeBlue and our range of >> >> CloudStack >> >>>>>>> related >> >>>>>>>>>>> services: >> >>>>>>>>>>>> IaaS Cloud Design & Build >> >>>>>>>>>>>> <http://shapeblue.com/iaas-cloud-design-and-build//> | >> >>>>> CSForge >> >>>>>>> - >> >>>>>>>>> rapid >> >>>>>>>>>>>> IaaS deployment framework < >> >> http://shapeblue.com/csforge/> >> >>>>>>>>>>>> CloudStack Consulting < >> >>>>>>>> http://shapeblue.com/cloudstack-consultancy/> >> >>>>>>>>> | >> >>>>>>>>>>> CloudStack >> >>>>>>>>>>>> Software Engineering >> >>>>>>>>>>>> <http://shapeblue.com/cloudstack-software-engineering/> >> >>>>>>>>>>>> CloudStack Infrastructure Support >> >>>>>>>>>>>> < >> >> http://shapeblue.com/cloudstack-infrastructure-support/> >> >>> | >> >>>>>>>>> CloudStack >> >>>>>>>>>>>> Bootcamp Training Courses < >> >>>>>>>> http://shapeblue.com/cloudstack-training/ >> >>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> DISCLAIMER >> >>>>>>>> ========== >> >>>>>>>> This e-mail may contain privileged and confidential information >> >>>> which >> >>>>> is >> >>>>>>>> the property of Accelerite, a Persistent Systems business. It is >> >>>>>>> intended >> >>>>>>>> only for the use of the individual or entity to which it is >> >>>> addressed. >> >>>>>>> If >> >>>>>>>> you are not the intended recipient, you are not authorized to >> >>> read, >> >>>>>>> retain, >> >>>>>>>> copy, print, distribute or use this message. If you have >> >> received >> >>>> this >> >>>>>>>> communication in error, please notify the sender and delete all >> >>>> copies >> >>>>>>> of >> >>>>>>>> this message. Accelerite, a Persistent Systems business does not >> >>>>> accept >> >>>>>>> any >> >>>>>>>> liability for virus infected mails. >> >>>>>>>> >> >>>>>>> >> >>>>>> >> >>>>>> >> >>>>> >> >>>> >> >>> >> >> >> >> >> > >> > >