Can you goto primary storage in cloudstack UI, is this what you have defined for your NFS Server?
"SharedMountPoint","host":"localhost","path":"/mnt/storage" If so, please use IP or FQDN instead.. On 4/14/16 10:39 PM, Nezar Madbouh wrote: > Hi Ilya, > > Here is the output of mount command: > > [root@eqx-cs-cmp-01 ~]# mount > /dev/mapper/vg_root-lv_root on / type ext4 (rw) > proc on /proc type proc (rw) > sysfs on /sys type sysfs (rw) > devpts on /dev/pts type devpts (rw,gid=5,mode=620) > tmpfs on /dev/shm type tmpfs (rw) > /dev/sda1 on /boot type ext4 (rw) > none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) > sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) > nfsd on /proc/fs/nfsd type nfsd (rw) > 10.61.154.10:/Cloudstack/primary on > /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57 type nfs > (rw,noatime,nfsvers=3,proto=tcp,hard,nolock,intr,addr=10.61.154.10) > [root@eqx-cs-cmp-01 ~]# > > > On 15 April 2016 at 15:22, Nezar Madbouh <nezar.madb...@ipscape.com.au> > wrote: > >> Hi Ilya, >> >> Thanks for the instructions. I followed them and pasted the information on >> pastebin: >> >> management-server logs job-14938 >> http://pastebin.com/GAiy8u5H >> >> Agent logs >> http://pastebin.com/mfiADsJc >> >> >> Your help is much appreciated. >> >> Nezar. >> >> On 15 April 2016 at 13:58, ilya <ilya.mailing.li...@gmail.com> wrote: >> >>> Here is how i would troubleshoot this: >>> >>> 1) ssh to MS host, run >>> >>> tail -f /var/cloudstack/management/management-server.log | grep -i >>> StartVMCmdByAdmin >>> >>> 2) on cloudstack UI, press start on the VM and watch the tail log >>> >>> 3) on the tail log, you should see a StartVirtualMachine api call show >>> up with job ID. For example: >>> >>>> 2016-04-15 03:41:57,193 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] >>> (API-Job-Executor-24:ctx-7892387a job-291575) Executing AsyncJobVO >>> {id:291575, userId: 157, accountId: 2, instanceType: VirtualMachine, >>> instanceId: 7648, cmd: >>> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin, cmdInfo: >>> {"id":"7648","response":"json","sessionkey":"mMJsduS9KLKApqH4dPpIlGZdt1Y\u003d","ctxDetails":"{\"com.cloud.vm.VirtualMachine\":7648,\"com.cloud.uservm.UserVm\":7648,\"com.cloud.network.router.VirtualRouter\":7648}","cmdEventType":"VM.START","expires":"2016-04-15T03:51:57+0000","ctxUserId":"157","signatureversion":"3","httpmethod":"GET","uuid":"7648","ctxAccountId":"2","ctxStartEventId":"314416"}, >>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, >>> result: null, initMsid: 345051337894, completeMsid: null, lastUpdated: >>> null, lastPolled: null, created: null} >>> >>> Each task in cloudstack spins up its own job, in this case its >>> *job-291575* >>> >>> 4) now execute >>>> grep -A3 -B3 job-291575 >>> /var/cloudstack/management/management-server.log | tee job-291575.log >>>> grep job-291575 /var/cloudstack/management/management-server.log | grep >>> via | head -1 >>> >>> you should see a response like so, >>> >>>> 2016-04-15 03:41:59,259 DEBUG [c.c.a.m.ClusteredAgentAttache] >>> (Work-Job-Executor-42:ctx-b2c4af6b job-291575/job-291576 ctx-0988152e) Seq >>> 441-4181310779036795138: Forwarding Seq 441-4181310779036795138: { Cmd , >>> MgmtId: 345051337894, via: 441(my-kvm-host01.example.com), Ver: v1, >>> Flags: 100011, >>> [{"com.cloud.agent.api.routing.DhcpEntryCommand":{"vmMac":"06:fe:4e:00:12:ad","vmIpAddress":"10.10.27.178","vmName":"ol6-test-ilya1","defaultRouter":"10.10.27.1","defaultDns":"10.17.27.166","duid":"00:03:00:01:06:fe:4e:00:12:ad","isDefault":true,"executeInSequence":false,"accessDetails":{"zone.network.type":"Advanced","router.guest.ip":"10.10.27.166","router.ip":"169.254.0.238"," >>> router.name":"r-7301-VM"},"wait":0}}] } to 345051337893 >>> >>> We are looking for this line "via: 441(my-kvm-host01.example.com)" >>> >>> SSH to my-kvm-host01.example.com, and copy the >>> /var/log/cloudstack/agent.log to pastebin >>> >>> also copy job-291575.log to pastebin >>> >>> I'll try to have a look, though when i get chance. >>> >>> >>> Also paste the output of mount command on kvm host in question. I'm >>> suspecting one of your storage pools maybe bad. >>> >>> Lastly, what distro are you running for KVM? >>> >>> Regards >>> ilya >>> >>> On 4/14/16 5:05 PM, Nezar Madbouh wrote: >>>> Hi Ilya, >>>> >>>> I shutdown all the VMs in Pre-prod Environmnet expecting the Pre-prod >>>> Virtual router to be deleted automatically because no VM is running on >>> this >>>> vlan but it wasn't deleted. >>>> >>>> Am I right to expect the VR to be deleted automatically if no VM is >>> running >>>> in that vlan? >>>> >>>> Thanks, >>>> Nezar. >>>> >>>> On 15 April 2016 at 09:53, Nezar Madbouh <nezar.madb...@ipscape.com.au> >>>> wrote: >>>> >>>>> Hi Ilya, >>>>> >>>>> I have run the script and it didn't return any results: >>>>> >>>>> [root@eqx-cs-mgt-01 ~]# ./async_jobs.sh >>>>> [root@eqx-cs-mgt-01 ~]# >>>>> >>>>> >>>>> I also ran the command inside mysql: >>>>> >>>>> mysql> SELECT id, user_id, account_id, instance_type, instance_id, >>>>> job_cmd, job_cmd_info, job_dispatcher, job_pending_signals FROM >>>>> cloud.async_job where job_status=0 and job_dispatcher not like >>>>> "pseudoJobDispatcher"; >>>>> Empty set (0.00 sec) >>>>> >>>>> your help is much appreciated, >>>>> Nezar. >>>>> >>>>> >>>>> On 14 April 2016 at 16:49, ilya <ilya.mailing.li...@gmail.com> wrote: >>>>> >>>>>> It appears you have pending jobs that are not letting you go through. >>>>>> >>>>>> Usually stop and starting management service should be enough. >>>>>> >>>>>> However, try running this script from your management server to show >>> us >>>>>> what you have running in async_job table. >>>>>> >>>>>> >>>>>> >>>>>>> #!/bin/bash >>>>>>> DATESTAMP=$(date +%m%d%y-%H%M%S) >>>>>>> DBPASS=$(java -classpath >>>>>> /usr/share/cloudstack-common/lib/jasypt-1.9.0.jar >>>>>> org.jasypt.intf.cli.JasyptPBEStringDecryptionCLI input="$(cat >>>>>> /etc/cloudstack/management/db.properties | grep db.cloud.password | >>> awk >>>>>> -F'(' '{print $2}' | sed 's/)//g')" password="$(cat >>>>>> /etc/cloudstack/management/key)" | grep -A2 OUTPUT | tail -1) >>>>>>> DBHOST=$(cat /etc/cloudstack/management/db.properties | grep >>>>>> db.cloud.host | awk -F'=' '{print $2}' | tail -1 ) >>>>>>> DBUSER=$(cat /etc/cloudstack/management/db.properties | grep >>>>>> db.cloud.username | awk -F'=' '{print $2}') >>>>>>> DB=$(cat /etc/cloudstack/management/db.properties | grep >>> db.cloud.name >>>>>> | awk -F'=' '{print $2}') >>>>>>> DBPORT=$(cat /etc/cloudstack/management/db.properties | grep >>>>>> db.cloud.port | awk -F'=' '{print $2}') >>>>>>> MYSQLCMD="mysql -h $DBHOST -u $DBUSER -P $DBPORT -p$DBPASS $DB" >>>>>>> echo 'SELECT id, user_id, account_id, instance_type, instance_id, >>>>>> job_cmd, job_cmd_info, job_dispatcher, job_pending_signals FROM >>>>>> cloud.async_job where job_status=0 and job_dispatcher not like >>>>>> "pseudoJobDispatcher"' | $MYSQLCMD >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On 4/13/16 5:16 PM, Nezar Madbouh wrote: >>>>>>> Hi Simon, >>>>>>> >>>>>>> The following work was done but unfortunately the problem is not >>> fixed: >>>>>>> >>>>>>> >>>>>>> on management server: >>>>>>> >>>>>>> service cloudstack-management stop >>>>>>> >>>>>>> >>>>>>> on the host: >>>>>>> >>>>>>> service cloudstack-agent stop >>>>>>> >>>>>>> service libvirtd restart >>>>>>> >>>>>>> service cloudstack-agent start >>>>>>> >>>>>>> >>>>>>> on management server: >>>>>>> >>>>>>> service cloudstack-management start >>>>>>> >>>>>>> Regards, >>>>>>> Nezar. >>>>>>> >>>>>>> On 24 March 2016 at 05:06, Simon Weller <swel...@ena.com> wrote: >>>>>>> >>>>>>>> Yes, I think that order is fine. Make sure the processes are indeed >>>>>>>> shutdown before moving on to the next step. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> ________________________________________ >>>>>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au> >>>>>>>> Sent: Tuesday, March 22, 2016 5:33 PM >>>>>>>> To: users@cloudstack.apache.org >>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to >>> start >>>>>>>> instance due to Unable to get answer that is of class >>>>>>>> com.cloud.agent.api.StartAnswer >>>>>>>> >>>>>>>> Hi Simon, >>>>>>>> >>>>>>>> Can you please check if the following sequence of action is correct: >>>>>>>> >>>>>>>> on management server: >>>>>>>> >>>>>>>> service cloudstack-management stop >>>>>>>> >>>>>>>> >>>>>>>> on the host: >>>>>>>> >>>>>>>> service cloudstack-agent stop >>>>>>>> >>>>>>>> service libvirtd restart >>>>>>>> >>>>>>>> service cloudstack-agent start >>>>>>>> >>>>>>>> >>>>>>>> on management server: >>>>>>>> >>>>>>>> service cloudstack-management start >>>>>>>> >>>>>>>> Thanks very much for your help. >>>>>>>> Nezar. >>>>>>>> >>>>>>>> On 23 March 2016 at 08:39, Simon Weller <swel...@ena.com> wrote: >>>>>>>> >>>>>>>>> Shutting down all the management services should clear the jobs. >>>>>>>>> >>>>>>>>> >>>>>>>>> ________________________________________ >>>>>>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au> >>>>>>>>> Sent: Tuesday, March 22, 2016 4:28 PM >>>>>>>>> To: users@cloudstack.apache.org >>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to >>>>>> start >>>>>>>>> instance due to Unable to get answer that is of class >>>>>>>>> com.cloud.agent.api.StartAnswer >>>>>>>>> >>>>>>>>> Hi Simon, >>>>>>>>> >>>>>>>>> I will plan to restart agents, libvirtd and management services. >>>>>>>>> >>>>>>>>> How do I clear hung jobs. >>>>>>>>> >>>>>>>>> Your help is much appreciated, >>>>>>>>> Nezar. >>>>>>>>> >>>>>>>>> On 22 March 2016 at 23:03, Simon Weller <swel...@ena.com> wrote: >>>>>>>>> >>>>>>>>>> Nezar, >>>>>>>>>> >>>>>>>>>> I've looked through the code related to your release and it seems >>>>>> that >>>>>>>> in >>>>>>>>>> order to throw this exception, 10 attempts have been made to >>> retrieve >>>>>>>> the >>>>>>>>>> disk information prior to the exception being thrown. >>>>>>>>>> >>>>>>>>>> It appears there is something really odd going on here. I'm >>> starting >>>>>>>> to >>>>>>>>>> wonder whether it might be worth while for you to stop all of your >>>>>>>>> agents, >>>>>>>>>> plus management server processes, restart libvirtd on each of the >>>>>> hosts >>>>>>>>> and >>>>>>>>>> then bring the agents back up prior to starting the management >>>>>> servers. >>>>>>>>>> >>>>>>>>>> There are a bunch of logs the pastebin you posted that seems to >>>>>>>> indicate >>>>>>>>>> you have some jobs hung, for example: >>>>>>>>>> >>>>>>>>>> 2016-03-14 21:45:31,828 WARN [o.a.c.f.j.i.AsyncJobMonitor] >>>>>>>>>> (Timer-1:ctx-0c090bd7) Task (job-9066) has been pending for >>> 1660851 >>>>>>>>> seconds >>>>>>>>>> 2016-03-14 21:45:31,828 WARN [o.a.c.f.j.i.AsyncJobMonitor] >>>>>>>>>> (Timer-1:ctx-0c090bd7) Task (job-8133) has been pending for >>> 4253003 >>>>>>>>> seconds >>>>>>>>>> >>>>>>>>>> Restarting the agents, libvirtd and management will not cause an >>>>>>>> outage >>>>>>>>>> to your cloud, but I would suggest, if you decided to do this, you >>>>>>>> should >>>>>>>>>> complete it in a maintenance window. >>>>>>>>>> >>>>>>>>>> - Si >>>>>>>>>> >>>>>>>>>> ________________________________________ >>>>>>>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au> >>>>>>>>>> Sent: Sunday, March 20, 2016 10:27 PM >>>>>>>>>> To: users@cloudstack.apache.org >>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to >>>>>> start >>>>>>>>>> instance due to Unable to get answer that is of class >>>>>>>>>> com.cloud.agent.api.StartAnswer >>>>>>>>>> >>>>>>>>>> Hi Simon, >>>>>>>>>> >>>>>>>>>> Please see answers to your questions below. I am not sure if I >>>>>> provided >>>>>>>>> the >>>>>>>>>> correct answer for the primary and secondary storage >>> configuration. >>>>>>>>> Please >>>>>>>>>> let me know if this is not what you are after: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Can you screenshot and post somewhere your secondary storage >>>>>>>>> configuration >>>>>>>>>> and your primary storage configuration from the UI with the >>> details? >>>>>>>>>> >>>>>>>>>> mysql> select * from configuration where name like '%secondary%'\G >>>>>>>>>> *************************** 1. row *************************** >>>>>>>>>> category: Account Defaults >>>>>>>>>> instance: DEFAULT >>>>>>>>>> component: management-server >>>>>>>>>> name: max.account.secondary.storage >>>>>>>>>> value: 400 >>>>>>>>>> description: The default maximum secondary storage space (in >>> GiB) >>>>>>>> that >>>>>>>>>> can be used for an account >>>>>>>>>> default_value: 400 >>>>>>>>>> updated: NULL >>>>>>>>>> scope: NULL >>>>>>>>>> is_dynamic: 0 >>>>>>>>>> *************************** 2. row *************************** >>>>>>>>>> category: Project Defaults >>>>>>>>>> instance: DEFAULT >>>>>>>>>> component: management-server >>>>>>>>>> name: max.project.secondary.storage >>>>>>>>>> value: 400 >>>>>>>>>> description: The default maximum secondary storage space (in >>> GiB) >>>>>>>> that >>>>>>>>>> can be used for a project >>>>>>>>>> default_value: 400 >>>>>>>>>> updated: NULL >>>>>>>>>> scope: NULL >>>>>>>>>> is_dynamic: 0 >>>>>>>>>> *************************** 3. row *************************** >>>>>>>>>> category: Hidden >>>>>>>>>> instance: DEFAULT >>>>>>>>>> component: management-server >>>>>>>>>> name: secondary.storage.vm >>>>>>>>>> value: +rwRhJOXPymbQOuAMLnhiw== >>>>>>>>>> description: Deploys a VM per zone to manage secondary storage >>> if >>>>>>>> true, >>>>>>>>>> otherwise secondary storage is mounted on management server >>>>>>>>>> default_value: NULL >>>>>>>>>> updated: NULL >>>>>>>>>> scope: NULL >>>>>>>>>> is_dynamic: 0 >>>>>>>>>> *************************** 4. row *************************** >>>>>>>>>> category: Advanced >>>>>>>>>> instance: DEFAULT >>>>>>>>>> component: ExtensionRegistry >>>>>>>>>> name: secondary.storage.vm.allocators.exclude >>>>>>>>>> value: NULL >>>>>>>>>> description: Extensions to exclude from being registered >>>>>>>>>> default_value: NULL >>>>>>>>>> updated: 2014-12-04 00:18:15 >>>>>>>>>> scope: Global >>>>>>>>>> is_dynamic: 0 >>>>>>>>>> *************************** 5. row *************************** >>>>>>>>>> category: NetworkManager >>>>>>>>>> instance: DEFAULT >>>>>>>>>> component: management-server >>>>>>>>>> name: vm.network.nic.max.secondary.ipaddresses >>>>>>>>>> value: NULL >>>>>>>>>> description: Specify the number of secondary ip addresses per >>> nic >>>>>> per >>>>>>>>> vm >>>>>>>>>> default_value: 256 >>>>>>>>>> updated: NULL >>>>>>>>>> scope: NULL >>>>>>>>>> is_dynamic: 0 >>>>>>>>>> 5 rows in set (0.00 sec) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> mysql> select * from configuration where name like '%primary%'\G >>>>>>>>>> *************************** 1. row *************************** >>>>>>>>>> category: Account Defaults >>>>>>>>>> instance: DEFAULT >>>>>>>>>> component: management-server >>>>>>>>>> name: max.account.primary.storage >>>>>>>>>> value: 200 >>>>>>>>>> description: The default maximum primary storage space (in GiB) >>>>>> that >>>>>>>>> can >>>>>>>>>> be used for an account >>>>>>>>>> default_value: 200 >>>>>>>>>> updated: NULL >>>>>>>>>> scope: NULL >>>>>>>>>> is_dynamic: 0 >>>>>>>>>> *************************** 2. row *************************** >>>>>>>>>> category: Project Defaults >>>>>>>>>> instance: DEFAULT >>>>>>>>>> component: management-server >>>>>>>>>> name: max.project.primary.storage >>>>>>>>>> value: 200 >>>>>>>>>> description: The default maximum primary storage space (in GiB) >>>>>> that >>>>>>>>> can >>>>>>>>>> be used for a project >>>>>>>>>> default_value: 200 >>>>>>>>>> updated: NULL >>>>>>>>>> scope: NULL >>>>>>>>>> is_dynamic: 0 >>>>>>>>>> *************************** 3. row *************************** >>>>>>>>>> category: Storage >>>>>>>>>> instance: DEFAULT >>>>>>>>>> component: TemplateManager >>>>>>>>>> name: primary.storage.download.wait >>>>>>>>>> value: 10800 >>>>>>>>>> description: In second, timeout for download template to primary >>>>>>>>> storage >>>>>>>>>> default_value: NULL >>>>>>>>>> updated: NULL >>>>>>>>>> scope: NULL >>>>>>>>>> is_dynamic: 0 >>>>>>>>>> 3 rows in set (0.00 sec) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Also, what is the Cloudstack version you are running and what is >>> the >>>>>> OS >>>>>>>>>> you're running it on? >>>>>>>>>> >>>>>>>>>> Cloudstack 4.4.1 >>>>>>>>>> >>>>>>>>>> Centos 6.6 >>>>>>>>>> >>>>>>>>>> Are you using actimeo=0 in your NFS mount line? >>>>>>>>>> >>>>>>>>>> Primary storage on Hypervisors has the following parameters >>>>>>>>>> 10.61.154.10:/Cloudstack/primary on >>>>>>>>>> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57 type nfs >>>>>>>>>> >>> (rw,noatime,nfsvers=3,proto=tcp,hard,nolock,intr,addr=10.61.154.10) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Secondary storage on SSVM has the following parameters >>>>>>>>>> 10.61.154.10:/Cloudstack/secondary on >>>>>>>>>> /mnt/SecStorage/d11d5c92-6950-30e8-a0b5-3cd46a17ec14 type nfs4 >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>> >>> (rw,relatime,vers=4,rsize=262144,wsize=262144,namlen=255,acdirmin=0,acdirmax=0,soft,proto=tcp,port=0,timeo=133,retrans= >>>>>> 2147483647 >>>>>> >>> ,sec=sys,clientaddr=10.61.154.70,minorversion=0,local_lock=none,addr=10.61.154.10) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thanks & Regards, >>>>>>>>>> Nezar. >>>>>>>>>> >>>>>>>>>> On 18 March 2016 at 23:50, Simon Weller <swel...@ena.com> wrote: >>>>>>>>>> >>>>>>>>>>> Also, what is the Cloudstack version you are running and what is >>> the >>>>>>>> OS >>>>>>>>>>> you're running it on? >>>>>>>>>>> >>>>>>>>>>> Are you using actimeo=0 in your NFS mount line? >>>>>>>>>>> >>>>>>>>>>> ________________________________________ >>>>>>>>>>> From: Simon Weller <swel...@ena.com> >>>>>>>>>>> Sent: Friday, March 18, 2016 7:40 AM >>>>>>>>>>> To: users@cloudstack.apache.org >>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to >>>>>>>> start >>>>>>>>>>> instance due to Unable to get answer that is of class >>>>>>>>>>> com.cloud.agent.api.StartAnswer >>>>>>>>>>> >>>>>>>>>>> Nexar, >>>>>>>>>>> >>>>>>>>>>> That's normal behaviour. You can only ssh to the link local on >>> the >>>>>>>> host >>>>>>>>>>> where the VM resides. >>>>>>>>>>> >>>>>>>>>>> Can you screenshot and post somewhere your secondary storage >>>>>>>>>> configuration >>>>>>>>>>> and your primary storage configuration from the UI with the >>> details? >>>>>>>>>>> >>>>>>>>>>> - Si >>>>>>>>>>> >>>>>>>>>>> ________________________________________ >>>>>>>>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au> >>>>>>>>>>> Sent: Thursday, March 17, 2016 8:19 PM >>>>>>>>>>> To: users@cloudstack.apache.org >>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to >>>>>>>> start >>>>>>>>>>> instance due to Unable to get answer that is of class >>>>>>>>>>> com.cloud.agent.api.StartAnswer >>>>>>>>>>> >>>>>>>>>>> Hi Simon, >>>>>>>>>>> >>>>>>>>>>> I just noticed that from other Hosts I can't ssh into the SSVM: >>>>>>>>>>> >>>>>>>>>>> [root@eqx-cs-cmp-01 ~]# ssh -i /root/.ssh/id_rsa.cloud -p 3922 >>>>>>>>>>> root@169.254.3.161 >>>>>>>>>>> ssh: connect to host 169.254.3.161 port 3922: No route to host >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I can only ssh from one Host: >>>>>>>>>>> [root@eqx-cs-cmp-06 ~]# ssh -i /root/.ssh/id_rsa.cloud -p 3922 >>>>>>>>>>> root@169.254.3.161 >>>>>>>>>>> Linux s-11-VM 3.2.0-4-amd64 #1 SMP Debian 3.2.57-3 x86_64 >>>>>>>>>>> >>>>>>>>>>> Thanks & Regards, >>>>>>>>>>> Nezar. >>>>>>>>>>> >>>>>>>>>>> On 18 March 2016 at 10:36, Nezar Madbouh < >>>>>>>> nezar.madb...@ipscape.com.au >>>>>>>>>> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Simon, >>>>>>>>>>>> >>>>>>>>>>>> Yes existing VMs are successfully running. Yesterday I was able >>> to >>>>>>>>>> create >>>>>>>>>>>> a VM in Dev Environment but I can't create VMs in Prod >>> environment. >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> Nezar. >>>>>>>>>>>> >>>>>>>>>>>> On 18 March 2016 at 09:12, Simon Weller <swel...@ena.com> >>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> I'm assuming you have other VMs successfully running, correct? >>> So >>>>>>>>> you >>>>>>>>>>> can >>>>>>>>>>>>> write to your primary storage. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> ________________________________________ >>>>>>>>>>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au> >>>>>>>>>>>>> Sent: Thursday, March 17, 2016 4:43 PM >>>>>>>>>>>>> To: users@cloudstack.apache.org >>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable >>> to >>>>>>>>>> start >>>>>>>>>>>>> instance due to Unable to get answer that is of class >>>>>>>>>>>>> com.cloud.agent.api.StartAnswer >>>>>>>>>>>>> >>>>>>>>>>>>> Hi Simon and Shweta, >>>>>>>>>>>>> >>>>>>>>>>>>> Please see the result below: >>>>>>>>>>>>> >>>>>>>>>>>>> root@s-11-VM:~# /usr/local/cloud/systemvm/ssvm-check.sh >>>>>>>>>>>>> ================================================ >>>>>>>>>>>>> First DNS server is 8.8.8.8 >>>>>>>>>>>>> PING 8.8.8.8 (8.8.8.8): 48 data bytes >>>>>>>>>>>>> 56 bytes from 8.8.8.8: icmp_seq=0 ttl=57 time=5.209 ms >>>>>>>>>>>>> 56 bytes from 8.8.8.8: icmp_seq=1 ttl=57 time=0.936 ms >>>>>>>>>>>>> --- 8.8.8.8 ping statistics --- >>>>>>>>>>>>> 2 packets transmitted, 2 packets received, 0% packet loss >>>>>>>>>>>>> round-trip min/avg/max/stddev = 0.936/3.072/5.209/2.137 ms >>>>>>>>>>>>> Good: Can ping DNS server >>>>>>>>>>>>> ================================================ >>>>>>>>>>>>> Good: DNS resolves download.cloud.com >>>>>>>>>>>>> ================================================ >>>>>>>>>>>>> nfs is currently mounted >>>>>>>>>>>>> Mount point is >>>>>>>> /mnt/SecStorage/d11d5c92-6950-30e8-a0b5-3cd46a17ec14 >>>>>>>>>>>>> Good: Can write to mount point >>>>>>>>>>>>> ================================================ >>>>>>>>>>>>> Management server is 10.61.153.20. Checking connectivity. >>>>>>>>>>>>> Good: Can connect to management server port 8250 >>>>>>>>>>>>> ================================================ >>>>>>>>>>>>> Good: Java process is running >>>>>>>>>>>>> ================================================ >>>>>>>>>>>>> Tests Complete. Look for ERROR or WARNING above. >>>>>>>>>>>>> root@s-11-VM:~# >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks & Regards, >>>>>>>>>>>>> Nezar. >>>>>>>>>>>>> >>>>>>>>>>>>> On 17 March 2016 at 09:42, Simon Weller <swel...@ena.com> >>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Nezar, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Log into the Cloudstack UI. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Click on "Infrastructure" on the left menu. Click on View All >>>>>>>>> below >>>>>>>>>>>>>> "System VMs". find the item marked "Secondary Storage VM" and >>>>>>>>> then >>>>>>>>>>>>> click >>>>>>>>>>>>>> on the Name of that VM on the left (it will be s-XXXX-VM). >>>>>>>> You'll >>>>>>>>>> see >>>>>>>>>>> a >>>>>>>>>>>>>> bunch of information about the VM. Find the Host line and also >>>>>>>> the >>>>>>>>>>> Link >>>>>>>>>>>>>> Local IP Address line and then follow the original >>> instructions >>>>>>>> in >>>>>>>>>> my >>>>>>>>>>>>>> previous email. >>>>>>>>>>>>>> >>>>>>>>>>>>>> - Si >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> ________________________________________ >>>>>>>>>>>>>> From: Shweta Agarwal <shweta.agarw...@accelerite.com> >>>>>>>>>>>>>> Sent: Wednesday, March 16, 2016 12:16 AM >>>>>>>>>>>>>> To: users@cloudstack.apache.org >>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable >>>>>>>> to >>>>>>>>>>> start >>>>>>>>>>>>>> instance due to Unable to get answer that is of class >>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Nezar, >>>>>>>>>>>>>> >>>>>>>>>>>>>> You can find host name on detail view of SSVM in UI . >>>>>>>>>>>>>> You can also use list api command listSystemVms to get the >>> host >>>>>>>>>>> details >>>>>>>>>>>>> of >>>>>>>>>>>>>> system vms. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>> Shweta >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> ________________________________________ >>>>>>>>>>>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au> >>>>>>>>>>>>>> Sent: Wednesday, March 16, 2016 9:58 AM >>>>>>>>>>>>>> To: users@cloudstack.apache.org >>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable >>>>>>>> to >>>>>>>>>>> start >>>>>>>>>>>>>> instance due to Unable to get answer that is of class >>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Simon, >>>>>>>>>>>>>> >>>>>>>>>>>>>> How can I find out which host the SSVM is on? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks & Regards, >>>>>>>>>>>>>> Nezar. >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 16 March 2016 at 12:28, Simon Weller <swel...@ena.com> >>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Can you ssh into the Secondary Storage VM? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This can be done by getting the link-local-ip address from >>> the >>>>>>>>>>>>>>> infrastructure menu in the UI and then sshing into the host >>>>>>>> the >>>>>>>>>> SSVM >>>>>>>>>>>>> is >>>>>>>>>>>>>> on >>>>>>>>>>>>>>> and issuing this command as root: ssh -i >>>>>>>> /root/.ssh/id_rsa.cloud >>>>>>>>>> -p >>>>>>>>>>>>> 3922 >>>>>>>>>>>>>>> root@link-local-ip (replace link-local-ip with the link >>>>>>>> local >>>>>>>>>> ip). >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Try running the SSVM diangnostic script: >>>>>>>>>>>>>>> usr/local/cloud/systemvm/ssvm-check.sh >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> - Si >>>>>>>>>>>>>>> ________________________________________ >>>>>>>>>>>>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au> >>>>>>>>>>>>>>> Sent: Tuesday, March 15, 2016 7:49 PM >>>>>>>>>>>>>>> To: users@cloudstack.apache.org >>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: >>> Unable >>>>>>>>> to >>>>>>>>>>>>> start >>>>>>>>>>>>>>> instance due to Unable to get answer that is of class >>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Simon, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Please see details below: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> mysql> use cloud; select id, name, status, type from host >>>>>>>> where >>>>>>>>>> id = >>>>>>>>>>>>> 5; >>>>>>>>>>>>>>> Reading table information for completion of table and column >>>>>>>>> names >>>>>>>>>>>>>>> You can turn off this feature to get a quicker startup with >>> -A >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Database changed >>>>>>>>>>>>>>> +----+------------------------------+--------+---------+ >>>>>>>>>>>>>>> | id | name | status | type | >>>>>>>>>>>>>>> +----+------------------------------+--------+---------+ >>>>>>>>>>>>>>> | 5 | eqx-cs-cmp-02.ipscape.com.au | up | routing | >>>>>>>>>>>>>>> +----+------------------------------+--------+---------+ >>>>>>>>>>>>>>> 1 row in set (0.00 sec) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> cat /var/log/cloudstack/agent/agent.log >>>>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: >>>>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Can't find >>>>>>>>>>>>>>> volume:089ff3ca-9bff-402c-a14e-ea7a35aaad5c >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>> >>> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getPhysicalDisk(KVMStoragePoolManager.java:262) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>> >>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVbd(LibvirtComputingResource.java:3921) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>> >>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:3789) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>> >>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1333) >>>>>>>>>>>>>>> at >>>>>>>> com.cloud.agent.Agent.processRequest(Agent.java:501) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>> >>> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:808) >>>>>>>>>>>>>>> at com.cloud.utils.nio.Task.run(Task.java:84) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>> >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>>>>>>>>>>>> at >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>> >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>>>>>>>>>> agent.log.2016-03-03.gz (END) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [root@eqx-cs-cmp-02 agent]# virsh pool-list >>>>>>>>>>>>>>> Name State Autostart >>>>>>>>>>>>>>> ----------------------------------------- >>>>>>>>>>>>>>> 2d0967f8-50d6-4d6d-87b7-4da0596f6924 active no >>>>>>>>>>>>>>> b3647480-5709-36eb-9ddd-e2c666f30b57 active no >>>>>>>>>>>>>>> e3831246-ff55-4af4-9964-a17565edaccc active no >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [Root @ eqx-cs-cmp-02 agent] # df -h >>>>>>>>>>>>>>> Filesystem Size Used Avail Use% Mounted on >>>>>>>>>>>>>>> /dev/mapper/vg_root-lv_root >>>>>>>>>>>>>>> 203G 215G 1.3G 1% / >>>>>>>>>>>>>>> tmpfs 48g 48g 0 0% / dev / shm >>>>>>>>>>>>>>> / Dev / sda1 477m 427m 25M 6% / boot >>>>>>>>>>>>>>> xx.xx.xx.xx:/Cloudstack/primary >>>>>>>>>>>>>>> 7.6T 4.1T 3.5T 54% >>>>>>>>>>>>>>> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I also noticed the following in cloudstack-agent.err >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> libvirt: QEMU Driver error : Domain not found: no domain with >>>>>>>>>>> matching >>>>>>>>>>>>>> name >>>>>>>>>>>>>>> 'i-11-600-VM' >>>>>>>>>>>>>>> libvirt: Storage Driver error : Storage volume not found: no >>>>>>>>>> storage >>>>>>>>>>>>> vol >>>>>>>>>>>>>>> with matching name '089ff3ca-9bff-402c-a14e-ea7a35aaad5c' >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Your help is much appreciated, >>>>>>>>>>>>>>> Nezar >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 16 March 2016 at 11:03, Simon Weller <swel...@ena.com> >>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Nezar, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> ok, I just took a look at a pastebin you posted. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I see it complaining about the inability to find a volume >>>>>>>>> prior >>>>>>>>>> to >>>>>>>>>>>>>>>> throwing the host exception. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Firstly, let's find out who host 5 is. Can you log into your >>>>>>>>>> MySQL >>>>>>>>>>>>>> server >>>>>>>>>>>>>>>> and run this: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> use cloud;select id,name,status,type from host where id=5; >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Can you post the results? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> If it's a host, can you post the >>>>>>>>>>> /var/log/cloudstack/agent/agent.log >>>>>>>>>>>>>> from >>>>>>>>>>>>>>>> that host? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> And also run virsh pool-list on the host in question? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> - Si >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> ________________________________________ >>>>>>>>>>>>>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au> >>>>>>>>>>>>>>>> Sent: Tuesday, March 15, 2016 6:11 PM >>>>>>>>>>>>>>>> To: users@cloudstack.apache.org >>>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: >>>>>>>> Unable >>>>>>>>>> to >>>>>>>>>>>>> start >>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class >>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Simon, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> When you say "double check your routers tied to the >>>>>>>> network", >>>>>>>>> do >>>>>>>>>>> you >>>>>>>>>>>>>> mean >>>>>>>>>>>>>>>> the physical routers or the virtual routers? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> When I try to create a VM in Prod Environment it fails. I >>>>>>>>> can't >>>>>>>>>>> ping >>>>>>>>>>>>>> the >>>>>>>>>>>>>>>> Prod virtual router from management server or Host. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> But I can ping Dev virtual router from management server and >>>>>>>>>> Host >>>>>>>>>>>>> and I >>>>>>>>>>>>>>> was >>>>>>>>>>>>>>>> able to create a VM in Dev Environment. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> How can I troubleshoot the Prod Virtual router. I launched >>>>>>>> the >>>>>>>>>>>>> console >>>>>>>>>>>>>>> but >>>>>>>>>>>>>>>> couldn't login. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Please note that we need to take into consideration that the >>>>>>>>>>>>> affected >>>>>>>>>>>>>>>> network is the Production network and we need to minimise >>>>>>>>>> service >>>>>>>>>>>>>> outage >>>>>>>>>>>>>>> as >>>>>>>>>>>>>>>> much as possible. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Your help is much appreciated, >>>>>>>>>>>>>>>> Nezar >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 16 March 2016 at 08:37, Simon Weller <swel...@ena.com> >>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I've seen this before where a Virtual Router has crashed >>>>>>>> and >>>>>>>>>> the >>>>>>>>>>>>>>>>> management server can't communicate with it. >>>>>>>>>>>>>>>>> Are you using an advanced network setup? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> If so, double check your routers tied to the network where >>>>>>>>> you >>>>>>>>>>> are >>>>>>>>>>>>>>>>> attempting to start an instance. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> - Si >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> ________________________________________ >>>>>>>>>>>>>>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au> >>>>>>>>>>>>>>>>> Sent: Tuesday, March 15, 2016 4:02 PM >>>>>>>>>>>>>>>>> To: users@cloudstack.apache.org >>>>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: >>>>>>>>> Unable >>>>>>>>>>> to >>>>>>>>>>>>>> start >>>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class >>>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi Glenn, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> We have 16 hosts, all KVM. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> The error is not specific to Host 5. Every time I try to >>>>>>>>>> create >>>>>>>>>>> a >>>>>>>>>>>>> VM >>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>> gives me the same error message but on different host. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> How do I restart the agent on the host. Is it service >>>>>>>>>> affecting? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks very much for your help. >>>>>>>>>>>>>>>>> Neza >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On 15 March 2016 at 17:37, Glenn Wagner < >>>>>>>>>>>>> glenn.wag...@shapeblue.com> >>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Can you explain your environment first , how many hosts >>>>>>>> , >>>>>>>>>> what >>>>>>>>>>>>> is >>>>>>>>>>>>>>> your >>>>>>>>>>>>>>>>>> hypervisors (KVM,Xenserver,VMware) >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> The error seems to be with your Host 5 Hypervisor >>>>>>>>>>>>>>>>>> Resource [Host:5] is unreachable: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Go to the infrastructure section and see what host5 is >>>>>>>>>>>>> reporting, >>>>>>>>>>>>>>>>>> Have you tried to restart the agent on the Host? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>>>> Glenn >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> [image: ShapeBlue] <http://www.shapeblue.com> >>>>>>>>>>>>>>>>>> Glenn Wagner >>>>>>>>>>>>>>>>>> Senior Consultant , ShapeBlue >>>>>>>>>>>>>>>>>> d: * | s: +27 21 527 0091* >>>>>>>>>>>>> <%7C%20s:%20+27%2021%20527%200091> | >>>>>>>>>>>>>>> m: >>>>>>>>>>>>>>>>>> *+27 73 917 4111* <+27%2073%20917%204111> >>>>>>>>>>>>>>>>>> e: *glenn.wag...@shapeblue.com | t: * >>>>>>>>>>>>>>>>>> <glenn.wag...@shapeblue.com%20%7C%20t:> | w: * >>>>>>>>>>>>> www.shapeblue.com >>>>>>>>>>>>>> * >>>>>>>>>>>>>>>>>> <http://www.shapeblue.com> >>>>>>>>>>>>>>>>>> a: >>>>>>>>>>>>>>>>>> 2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West >>>>>>>>> Cape >>>>>>>>>>> Town >>>>>>>>>>>>>> 7130 >>>>>>>>>>>>>>>>> South Africa >>>>>>>>>>>>>>>>>> Shape Blue Ltd is a company incorporated in England & >>>>>>>>> Wales. >>>>>>>>>>>>>>> ShapeBlue >>>>>>>>>>>>>>>>>> Services India LLP is a company incorporated in India >>>>>>>> and >>>>>>>>> is >>>>>>>>>>>>>> operated >>>>>>>>>>>>>>>>> under >>>>>>>>>>>>>>>>>> license from Shape Blue Ltd. Shape Blue Brasil >>>>>>>> Consultoria >>>>>>>>>>> Ltda >>>>>>>>>>>>> is >>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>>> company incorporated in Brasil and is operated under >>>>>>>>> license >>>>>>>>>>>>> from >>>>>>>>>>>>>>> Shape >>>>>>>>>>>>>>>>>> Blue Ltd. ShapeBlue SA Pty Ltd is a company registered >>>>>>>> by >>>>>>>>>> The >>>>>>>>>>>>>>> Republic >>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>> South Africa and is traded under license from Shape Blue >>>>>>>>>> Ltd. >>>>>>>>>>>>>>> ShapeBlue >>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>> a registered trademark. >>>>>>>>>>>>>>>>>> This email and any attachments to it may be confidential >>>>>>>>> and >>>>>>>>>>> are >>>>>>>>>>>>>>>> intended >>>>>>>>>>>>>>>>>> solely for the use of the individual to whom it is >>>>>>>>>> addressed. >>>>>>>>>>>>> Any >>>>>>>>>>>>>>> views >>>>>>>>>>>>>>>>> or >>>>>>>>>>>>>>>>>> opinions expressed are solely those of the author and do >>>>>>>>> not >>>>>>>>>>>>>>>> necessarily >>>>>>>>>>>>>>>>>> represent those of Shape Blue Ltd or related companies. >>>>>>>> If >>>>>>>>>> you >>>>>>>>>>>>> are >>>>>>>>>>>>>>> not >>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>> intended recipient of this email, you must neither take >>>>>>>>> any >>>>>>>>>>>>> action >>>>>>>>>>>>>>>> based >>>>>>>>>>>>>>>>>> upon its contents, nor copy or show it to anyone. Please >>>>>>>>>>> contact >>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>> sender >>>>>>>>>>>>>>>>>> if you believe you have received this email in error. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>>>>>>>> From: Nezar Madbouh [mailto: >>>>>>>> nezar.madb...@ipscape.com.au] >>>>>>>>>>>>>>>>>> Sent: Tuesday, 15 March 2016 5:45 AM >>>>>>>>>>>>>>>>>> To: users@cloudstack.apache.org >>>>>>>>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message: >>>>>>>>>> Unable >>>>>>>>>>>>> to >>>>>>>>>>>>>>> start >>>>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class >>>>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hi Jeremy, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I have uploaded the contents to >>>>>>>>>> http://pastebin.com/1J3mmgTh >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> your help is much appreciated, >>>>>>>>>>>>>>>>>> Nezar. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On 15 March 2016 at 13:46, Jeremy Peterson < >>>>>>>>>>>>> jpeter...@acentek.net> >>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I don't see any attachments can you create a pastebin. >>>>>>>>>>> That's >>>>>>>>>>>>>>> easier >>>>>>>>>>>>>>>>>>> to look through as I think attachments are removed >>>>>>>>> before >>>>>>>>>>>>> being >>>>>>>>>>>>>>> sent >>>>>>>>>>>>>>>>>>> to user group. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Jeremy >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> -------- Original message -------- >>>>>>>>>>>>>>>>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au> >>>>>>>>>>>>>>>>>>> Date: 3/14/2016 6:38 PM (GMT-06:00) >>>>>>>>>>>>>>>>>>> To: users@cloudstack.apache.org >>>>>>>>>>>>>>>>>>> Subject: Unable to create a New VM - Error message: >>>>>>>>> Unable >>>>>>>>>>> to >>>>>>>>>>>>>> start >>>>>>>>>>>>>>>>>>> instance due to Unable to get answer that is of class >>>>>>>>>>>>>>>>>>> com.cloud.agent.api.StartAnswer >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Hi Shweta and Sadhu, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Please find attached managerment-server.log which >>>>>>>>> contains >>>>>>>>>>> the >>>>>>>>>>>>>> logs >>>>>>>>>>>>>>>>>>> since I started creating a new VM (VM id 600). >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Please note that we don't have agent.log files >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Your help is much appreciated. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>>>> Nezar. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Find out more about ShapeBlue and our range of >>>>>>>> CloudStack >>>>>>>>>>>>> related >>>>>>>>>>>>>>>>> services: >>>>>>>>>>>>>>>>>> IaaS Cloud Design & Build >>>>>>>>>>>>>>>>>> <http://shapeblue.com/iaas-cloud-design-and-build//> | >>>>>>>>>>> CSForge >>>>>>>>>>>>> - >>>>>>>>>>>>>>> rapid >>>>>>>>>>>>>>>>>> IaaS deployment framework < >>>>>>>> http://shapeblue.com/csforge/> >>>>>>>>>>>>>>>>>> CloudStack Consulting < >>>>>>>>>>>>>> http://shapeblue.com/cloudstack-consultancy/> >>>>>>>>>>>>>>> | >>>>>>>>>>>>>>>>> CloudStack >>>>>>>>>>>>>>>>>> Software Engineering >>>>>>>>>>>>>>>>>> <http://shapeblue.com/cloudstack-software-engineering/> >>>>>>>>>>>>>>>>>> CloudStack Infrastructure Support >>>>>>>>>>>>>>>>>> < >>>>>>>> http://shapeblue.com/cloudstack-infrastructure-support/> >>>>>>>>> | >>>>>>>>>>>>>>> CloudStack >>>>>>>>>>>>>>>>>> Bootcamp Training Courses < >>>>>>>>>>>>>> http://shapeblue.com/cloudstack-training/ >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> DISCLAIMER >>>>>>>>>>>>>> ========== >>>>>>>>>>>>>> This e-mail may contain privileged and confidential >>> information >>>>>>>>>> which >>>>>>>>>>> is >>>>>>>>>>>>>> the property of Accelerite, a Persistent Systems business. It >>> is >>>>>>>>>>>>> intended >>>>>>>>>>>>>> only for the use of the individual or entity to which it is >>>>>>>>>> addressed. >>>>>>>>>>>>> If >>>>>>>>>>>>>> you are not the intended recipient, you are not authorized to >>>>>>>>> read, >>>>>>>>>>>>> retain, >>>>>>>>>>>>>> copy, print, distribute or use this message. If you have >>>>>>>> received >>>>>>>>>> this >>>>>>>>>>>>>> communication in error, please notify the sender and delete >>> all >>>>>>>>>> copies >>>>>>>>>>>>> of >>>>>>>>>>>>>> this message. Accelerite, a Persistent Systems business does >>> not >>>>>>>>>>> accept >>>>>>>>>>>>> any >>>>>>>>>>>>>> liability for virus infected mails. >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>>> >>>> >>> >>> >> >
