Hi Ilya,

I shutdown all the VMs in Pre-prod Environmnet expecting the Pre-prod
Virtual router to be deleted automatically because no VM is running on this
vlan but it wasn't deleted.

Am I right to expect the VR to be deleted automatically if no VM is running
in that vlan?

Thanks,
Nezar.

On 15 April 2016 at 09:53, Nezar Madbouh <nezar.madb...@ipscape.com.au>
wrote:

> Hi Ilya,
>
> I have run the script and it didn't return any results:
>
> [root@eqx-cs-mgt-01 ~]# ./async_jobs.sh
> [root@eqx-cs-mgt-01 ~]#
>
>
> I also ran the command inside mysql:
>
> mysql> SELECT id, user_id, account_id, instance_type, instance_id,
> job_cmd, job_cmd_info, job_dispatcher, job_pending_signals  FROM
> cloud.async_job where job_status=0 and job_dispatcher not like
> "pseudoJobDispatcher";
> Empty set (0.00 sec)
>
> your help is much appreciated,
> Nezar.
>
>
> On 14 April 2016 at 16:49, ilya <ilya.mailing.li...@gmail.com> wrote:
>
>> It appears you have pending jobs that are not letting you go through.
>>
>> Usually stop and starting management service should be enough.
>>
>> However, try running this script from your management server to show us
>> what you have running in async_job table.
>>
>>
>>
>> > #!/bin/bash
>> > DATESTAMP=$(date +%m%d%y-%H%M%S)
>> > DBPASS=$(java -classpath
>> /usr/share/cloudstack-common/lib/jasypt-1.9.0.jar
>> org.jasypt.intf.cli.JasyptPBEStringDecryptionCLI input="$(cat
>> /etc/cloudstack/management/db.properties | grep db.cloud.password | awk
>> -F'(' '{print $2}' | sed 's/)//g')" password="$(cat
>> /etc/cloudstack/management/key)" | grep -A2 OUTPUT | tail -1)
>> > DBHOST=$(cat /etc/cloudstack/management/db.properties | grep
>> db.cloud.host | awk -F'=' '{print $2}' | tail -1 )
>> > DBUSER=$(cat /etc/cloudstack/management/db.properties | grep
>> db.cloud.username | awk -F'=' '{print $2}')
>> > DB=$(cat /etc/cloudstack/management/db.properties | grep db.cloud.name
>> | awk -F'=' '{print $2}')
>> > DBPORT=$(cat /etc/cloudstack/management/db.properties | grep
>> db.cloud.port | awk -F'=' '{print $2}')
>> > MYSQLCMD="mysql -h $DBHOST -u $DBUSER -P $DBPORT -p$DBPASS $DB"
>> > echo 'SELECT id, user_id, account_id, instance_type, instance_id,
>> job_cmd, job_cmd_info, job_dispatcher, job_pending_signals  FROM
>> cloud.async_job where job_status=0 and job_dispatcher not like
>> "pseudoJobDispatcher"' | $MYSQLCMD
>>
>>
>>
>>
>> On 4/13/16 5:16 PM, Nezar Madbouh wrote:
>> > Hi Simon,
>> >
>> > The following work was done but unfortunately the problem is not fixed:
>> >
>> >
>> > on management server:
>> >
>> > service cloudstack-management stop
>> >
>> >
>> > on the host:
>> >
>> > service cloudstack-agent stop
>> >
>> > service libvirtd restart
>> >
>> > service cloudstack-agent start
>> >
>> >
>> > on management server:
>> >
>> > service cloudstack-management start
>> >
>> > Regards,
>> > Nezar.
>> >
>> > On 24 March 2016 at 05:06, Simon Weller <swel...@ena.com> wrote:
>> >
>> >> Yes, I think that order is fine. Make sure the processes are indeed
>> >> shutdown before moving on to the next step.
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> ________________________________________
>> >> From: Nezar Madbouh <nezar.madb...@ipscape.com.au>
>> >> Sent: Tuesday, March 22, 2016 5:33 PM
>> >> To: users@cloudstack.apache.org
>> >> Subject: Re: Unable to create a New VM - Error message: Unable to start
>> >> instance due to Unable to get answer that is of class
>> >> com.cloud.agent.api.StartAnswer
>> >>
>> >> Hi Simon,
>> >>
>> >> Can you please check if the following sequence of action is correct:
>> >>
>> >> on management server:
>> >>
>> >> service cloudstack-management stop
>> >>
>> >>
>> >> on the host:
>> >>
>> >> service cloudstack-agent stop
>> >>
>> >> service libvirtd restart
>> >>
>> >> service cloudstack-agent start
>> >>
>> >>
>> >> on management server:
>> >>
>> >> service cloudstack-management start
>> >>
>> >> Thanks very much for your help.
>> >> Nezar.
>> >>
>> >> On 23 March 2016 at 08:39, Simon Weller <swel...@ena.com> wrote:
>> >>
>> >>> Shutting down all the management services should clear the jobs.
>> >>>
>> >>>
>> >>> ________________________________________
>> >>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au>
>> >>> Sent: Tuesday, March 22, 2016 4:28 PM
>> >>> To: users@cloudstack.apache.org
>> >>> Subject: Re: Unable to create a New VM - Error message: Unable to
>> start
>> >>> instance due to Unable to get answer that is of class
>> >>> com.cloud.agent.api.StartAnswer
>> >>>
>> >>> Hi Simon,
>> >>>
>> >>> I will plan to restart agents, libvirtd and management services.
>> >>>
>> >>> How do I clear hung jobs.
>> >>>
>> >>> Your help is much appreciated,
>> >>> Nezar.
>> >>>
>> >>> On 22 March 2016 at 23:03, Simon Weller <swel...@ena.com> wrote:
>> >>>
>> >>>> Nezar,
>> >>>>
>> >>>> I've looked through the code related to your release and it seems
>> that
>> >> in
>> >>>> order to throw this exception, 10 attempts have been made to retrieve
>> >> the
>> >>>> disk information prior to the exception being thrown.
>> >>>>
>> >>>> It appears there is something really odd going on here.  I'm starting
>> >> to
>> >>>> wonder whether it might be worth while for you to stop all of your
>> >>> agents,
>> >>>> plus management server processes, restart libvirtd on each of the
>> hosts
>> >>> and
>> >>>> then bring the agents back up prior to starting the management
>> servers.
>> >>>>
>> >>>> There are a bunch of logs the pastebin you posted that seems to
>> >> indicate
>> >>>> you have some jobs hung, for example:
>> >>>>
>> >>>> 2016-03-14 21:45:31,828 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
>> >>>> (Timer-1:ctx-0c090bd7) Task (job-9066) has been pending for 1660851
>> >>> seconds
>> >>>> 2016-03-14 21:45:31,828 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
>> >>>> (Timer-1:ctx-0c090bd7) Task (job-8133) has been pending for 4253003
>> >>> seconds
>> >>>>
>> >>>> Restarting the  agents, libvirtd and management will not cause an
>> >> outage
>> >>>> to your cloud, but I would suggest, if you decided to do this, you
>> >> should
>> >>>> complete it in a maintenance window.
>> >>>>
>> >>>> - Si
>> >>>>
>> >>>> ________________________________________
>> >>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au>
>> >>>> Sent: Sunday, March 20, 2016 10:27 PM
>> >>>> To: users@cloudstack.apache.org
>> >>>> Subject: Re: Unable to create a New VM - Error message: Unable to
>> start
>> >>>> instance due to Unable to get answer that is of class
>> >>>> com.cloud.agent.api.StartAnswer
>> >>>>
>> >>>> Hi Simon,
>> >>>>
>> >>>> Please see answers to your questions below. I am not sure if I
>> provided
>> >>> the
>> >>>> correct answer for the primary and secondary storage configuration.
>> >>> Please
>> >>>> let me know if this is not what you are after:
>> >>>>
>> >>>>
>> >>>> Can you screenshot and post somewhere your secondary storage
>> >>> configuration
>> >>>> and your primary storage configuration from the UI with the details?
>> >>>>
>> >>>> mysql> select * from configuration where name like '%secondary%'\G
>> >>>> *************************** 1. row ***************************
>> >>>>      category: Account Defaults
>> >>>>      instance: DEFAULT
>> >>>>     component: management-server
>> >>>>          name: max.account.secondary.storage
>> >>>>         value: 400
>> >>>>   description: The default maximum secondary storage space (in GiB)
>> >> that
>> >>>> can be used for an account
>> >>>> default_value: 400
>> >>>>       updated: NULL
>> >>>>         scope: NULL
>> >>>>    is_dynamic: 0
>> >>>> *************************** 2. row ***************************
>> >>>>      category: Project Defaults
>> >>>>      instance: DEFAULT
>> >>>>     component: management-server
>> >>>>          name: max.project.secondary.storage
>> >>>>         value: 400
>> >>>>   description: The default maximum secondary storage space (in GiB)
>> >> that
>> >>>> can be used for a project
>> >>>> default_value: 400
>> >>>>       updated: NULL
>> >>>>         scope: NULL
>> >>>>    is_dynamic: 0
>> >>>> *************************** 3. row ***************************
>> >>>>      category: Hidden
>> >>>>      instance: DEFAULT
>> >>>>     component: management-server
>> >>>>          name: secondary.storage.vm
>> >>>>         value: +rwRhJOXPymbQOuAMLnhiw==
>> >>>>   description: Deploys a VM per zone to manage secondary storage if
>> >> true,
>> >>>> otherwise secondary storage is mounted on management server
>> >>>> default_value: NULL
>> >>>>       updated: NULL
>> >>>>         scope: NULL
>> >>>>    is_dynamic: 0
>> >>>> *************************** 4. row ***************************
>> >>>>      category: Advanced
>> >>>>      instance: DEFAULT
>> >>>>     component: ExtensionRegistry
>> >>>>          name: secondary.storage.vm.allocators.exclude
>> >>>>         value: NULL
>> >>>>   description: Extensions to exclude from being registered
>> >>>> default_value: NULL
>> >>>>       updated: 2014-12-04 00:18:15
>> >>>>         scope: Global
>> >>>>    is_dynamic: 0
>> >>>> *************************** 5. row ***************************
>> >>>>      category: NetworkManager
>> >>>>      instance: DEFAULT
>> >>>>     component: management-server
>> >>>>          name: vm.network.nic.max.secondary.ipaddresses
>> >>>>         value: NULL
>> >>>>   description: Specify the number of secondary ip addresses per nic
>> per
>> >>> vm
>> >>>> default_value: 256
>> >>>>       updated: NULL
>> >>>>         scope: NULL
>> >>>>    is_dynamic: 0
>> >>>> 5 rows in set (0.00 sec)
>> >>>>
>> >>>>
>> >>>> mysql> select * from configuration where name like '%primary%'\G
>> >>>> *************************** 1. row ***************************
>> >>>>      category: Account Defaults
>> >>>>      instance: DEFAULT
>> >>>>     component: management-server
>> >>>>          name: max.account.primary.storage
>> >>>>         value: 200
>> >>>>   description: The default maximum primary storage space (in GiB)
>> that
>> >>> can
>> >>>> be used for an account
>> >>>> default_value: 200
>> >>>>       updated: NULL
>> >>>>         scope: NULL
>> >>>>    is_dynamic: 0
>> >>>> *************************** 2. row ***************************
>> >>>>      category: Project Defaults
>> >>>>      instance: DEFAULT
>> >>>>     component: management-server
>> >>>>          name: max.project.primary.storage
>> >>>>         value: 200
>> >>>>   description: The default maximum primary storage space (in GiB)
>> that
>> >>> can
>> >>>> be used for a project
>> >>>> default_value: 200
>> >>>>       updated: NULL
>> >>>>         scope: NULL
>> >>>>    is_dynamic: 0
>> >>>> *************************** 3. row ***************************
>> >>>>      category: Storage
>> >>>>      instance: DEFAULT
>> >>>>     component: TemplateManager
>> >>>>          name: primary.storage.download.wait
>> >>>>         value: 10800
>> >>>>   description: In second, timeout for download template to primary
>> >>> storage
>> >>>> default_value: NULL
>> >>>>       updated: NULL
>> >>>>         scope: NULL
>> >>>>    is_dynamic: 0
>> >>>> 3 rows in set (0.00 sec)
>> >>>>
>> >>>>
>> >>>>
>> >>>> Also, what is the Cloudstack version you are running and what is the
>> OS
>> >>>> you're running it on?
>> >>>>
>> >>>> Cloudstack 4.4.1
>> >>>>
>> >>>> Centos 6.6
>> >>>>
>> >>>> Are you using actimeo=0 in your NFS mount line?
>> >>>>
>> >>>> Primary storage on Hypervisors has the following parameters
>> >>>> 10.61.154.10:/Cloudstack/primary on
>> >>>> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57 type nfs
>> >>>> (rw,noatime,nfsvers=3,proto=tcp,hard,nolock,intr,addr=10.61.154.10)
>> >>>>
>> >>>>
>> >>>> Secondary storage on SSVM has the following parameters
>> >>>> 10.61.154.10:/Cloudstack/secondary on
>> >>>> /mnt/SecStorage/d11d5c92-6950-30e8-a0b5-3cd46a17ec14 type nfs4
>> >>>>
>> >>>>
>> >>>
>> >>
>> (rw,relatime,vers=4,rsize=262144,wsize=262144,namlen=255,acdirmin=0,acdirmax=0,soft,proto=tcp,port=0,timeo=133,retrans=
>> 2147483647
>> ,sec=sys,clientaddr=10.61.154.70,minorversion=0,local_lock=none,addr=10.61.154.10)
>> >>>>
>> >>>>
>> >>>> Thanks & Regards,
>> >>>> Nezar.
>> >>>>
>> >>>> On 18 March 2016 at 23:50, Simon Weller <swel...@ena.com> wrote:
>> >>>>
>> >>>>> Also, what is the Cloudstack version you are running and what is the
>> >> OS
>> >>>>> you're running it on?
>> >>>>>
>> >>>>> Are you using actimeo=0 in your NFS mount line?
>> >>>>>
>> >>>>> ________________________________________
>> >>>>> From: Simon Weller <swel...@ena.com>
>> >>>>> Sent: Friday, March 18, 2016 7:40 AM
>> >>>>> To: users@cloudstack.apache.org
>> >>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
>> >> start
>> >>>>> instance due to Unable to get answer that is of class
>> >>>>> com.cloud.agent.api.StartAnswer
>> >>>>>
>> >>>>> Nexar,
>> >>>>>
>> >>>>> That's normal behaviour. You can only ssh to the link local on the
>> >> host
>> >>>>> where the VM resides.
>> >>>>>
>> >>>>> Can you screenshot and post somewhere your secondary storage
>> >>>> configuration
>> >>>>> and your primary storage configuration from the UI with the details?
>> >>>>>
>> >>>>> - Si
>> >>>>>
>> >>>>> ________________________________________
>> >>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au>
>> >>>>> Sent: Thursday, March 17, 2016 8:19 PM
>> >>>>> To: users@cloudstack.apache.org
>> >>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
>> >> start
>> >>>>> instance due to Unable to get answer that is of class
>> >>>>> com.cloud.agent.api.StartAnswer
>> >>>>>
>> >>>>> Hi Simon,
>> >>>>>
>> >>>>> I just noticed that from other Hosts I can't ssh into the SSVM:
>> >>>>>
>> >>>>> [root@eqx-cs-cmp-01 ~]# ssh -i /root/.ssh/id_rsa.cloud -p 3922
>> >>>>> root@169.254.3.161
>> >>>>> ssh: connect to host 169.254.3.161 port 3922: No route to host
>> >>>>>
>> >>>>>
>> >>>>> I can only ssh from one Host:
>> >>>>> [root@eqx-cs-cmp-06 ~]# ssh -i /root/.ssh/id_rsa.cloud -p 3922
>> >>>>> root@169.254.3.161
>> >>>>> Linux s-11-VM 3.2.0-4-amd64 #1 SMP Debian 3.2.57-3 x86_64
>> >>>>>
>> >>>>> Thanks & Regards,
>> >>>>> Nezar.
>> >>>>>
>> >>>>> On 18 March 2016 at 10:36, Nezar Madbouh <
>> >> nezar.madb...@ipscape.com.au
>> >>>>
>> >>>>> wrote:
>> >>>>>
>> >>>>>> Hi Simon,
>> >>>>>>
>> >>>>>> Yes existing VMs are successfully running. Yesterday I was able to
>> >>>> create
>> >>>>>> a VM in Dev Environment but I can't create VMs in Prod environment.
>> >>>>>>
>> >>>>>> Thanks,
>> >>>>>> Nezar.
>> >>>>>>
>> >>>>>> On 18 March 2016 at 09:12, Simon Weller <swel...@ena.com> wrote:
>> >>>>>>
>> >>>>>>> I'm assuming you have other VMs successfully running, correct? So
>> >>> you
>> >>>>> can
>> >>>>>>> write to your primary storage.
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> ________________________________________
>> >>>>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au>
>> >>>>>>> Sent: Thursday, March 17, 2016 4:43 PM
>> >>>>>>> To: users@cloudstack.apache.org
>> >>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable to
>> >>>> start
>> >>>>>>> instance due to Unable to get answer that is of class
>> >>>>>>> com.cloud.agent.api.StartAnswer
>> >>>>>>>
>> >>>>>>> Hi Simon and Shweta,
>> >>>>>>>
>> >>>>>>> Please see the result below:
>> >>>>>>>
>> >>>>>>> root@s-11-VM:~# /usr/local/cloud/systemvm/ssvm-check.sh
>> >>>>>>> ================================================
>> >>>>>>> First DNS server is  8.8.8.8
>> >>>>>>> PING 8.8.8.8 (8.8.8.8): 48 data bytes
>> >>>>>>> 56 bytes from 8.8.8.8: icmp_seq=0 ttl=57 time=5.209 ms
>> >>>>>>> 56 bytes from 8.8.8.8: icmp_seq=1 ttl=57 time=0.936 ms
>> >>>>>>> --- 8.8.8.8 ping statistics ---
>> >>>>>>> 2 packets transmitted, 2 packets received, 0% packet loss
>> >>>>>>> round-trip min/avg/max/stddev = 0.936/3.072/5.209/2.137 ms
>> >>>>>>> Good: Can ping DNS server
>> >>>>>>> ================================================
>> >>>>>>> Good: DNS resolves download.cloud.com
>> >>>>>>> ================================================
>> >>>>>>> nfs is currently mounted
>> >>>>>>> Mount point is
>> >> /mnt/SecStorage/d11d5c92-6950-30e8-a0b5-3cd46a17ec14
>> >>>>>>> Good: Can write to mount point
>> >>>>>>> ================================================
>> >>>>>>> Management server is 10.61.153.20. Checking connectivity.
>> >>>>>>> Good: Can connect to management server port 8250
>> >>>>>>> ================================================
>> >>>>>>> Good: Java process is running
>> >>>>>>> ================================================
>> >>>>>>> Tests Complete. Look for ERROR or WARNING above.
>> >>>>>>> root@s-11-VM:~#
>> >>>>>>>
>> >>>>>>> Thanks & Regards,
>> >>>>>>> Nezar.
>> >>>>>>>
>> >>>>>>> On 17 March 2016 at 09:42, Simon Weller <swel...@ena.com> wrote:
>> >>>>>>>
>> >>>>>>>> Nezar,
>> >>>>>>>>
>> >>>>>>>> Log into the Cloudstack UI.
>> >>>>>>>>
>> >>>>>>>> Click on "Infrastructure" on the left menu. Click on  View All
>> >>> below
>> >>>>>>>> "System VMs". find the item marked "Secondary Storage VM"  and
>> >>> then
>> >>>>>>> click
>> >>>>>>>> on the Name of that VM on the left (it will be s-XXXX-VM).
>> >> You'll
>> >>>> see
>> >>>>> a
>> >>>>>>>> bunch of information about the VM. Find the Host line and also
>> >> the
>> >>>>> Link
>> >>>>>>>> Local IP Address line and then follow the original instructions
>> >> in
>> >>>> my
>> >>>>>>>> previous email.
>> >>>>>>>>
>> >>>>>>>> - Si
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> ________________________________________
>> >>>>>>>> From: Shweta Agarwal <shweta.agarw...@accelerite.com>
>> >>>>>>>> Sent: Wednesday, March 16, 2016 12:16 AM
>> >>>>>>>> To: users@cloudstack.apache.org
>> >>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable
>> >> to
>> >>>>> start
>> >>>>>>>> instance due to Unable to get answer that is of class
>> >>>>>>>> com.cloud.agent.api.StartAnswer
>> >>>>>>>>
>> >>>>>>>> Hi Nezar,
>> >>>>>>>>
>> >>>>>>>> You can find host name on detail view of SSVM in UI .
>> >>>>>>>> You can also use list api command listSystemVms to get the host
>> >>>>> details
>> >>>>>>> of
>> >>>>>>>> system vms.
>> >>>>>>>>
>> >>>>>>>> Thanks
>> >>>>>>>> Shweta
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> ________________________________________
>> >>>>>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au>
>> >>>>>>>> Sent: Wednesday, March 16, 2016 9:58 AM
>> >>>>>>>> To: users@cloudstack.apache.org
>> >>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable
>> >> to
>> >>>>> start
>> >>>>>>>> instance due to Unable to get answer that is of class
>> >>>>>>>> com.cloud.agent.api.StartAnswer
>> >>>>>>>>
>> >>>>>>>> Hi Simon,
>> >>>>>>>>
>> >>>>>>>> How can I find out which host the SSVM is on?
>> >>>>>>>>
>> >>>>>>>> Thanks & Regards,
>> >>>>>>>> Nezar.
>> >>>>>>>>
>> >>>>>>>> On 16 March 2016 at 12:28, Simon Weller <swel...@ena.com>
>> >> wrote:
>> >>>>>>>>
>> >>>>>>>>> Can you ssh into the Secondary Storage VM?
>> >>>>>>>>>
>> >>>>>>>>> This can be done by getting the link-local-ip address from the
>> >>>>>>>>> infrastructure menu in the UI and then sshing into the host
>> >> the
>> >>>> SSVM
>> >>>>>>> is
>> >>>>>>>> on
>> >>>>>>>>> and issuing this command as root: ssh -i
>> >> /root/.ssh/id_rsa.cloud
>> >>>> -p
>> >>>>>>> 3922
>> >>>>>>>>> root@link-local-ip  (replace link-local-ip with the link
>> >> local
>> >>>> ip).
>> >>>>>>>>>
>> >>>>>>>>> Try  running the SSVM diangnostic script:
>> >>>>>>>>> usr/local/cloud/systemvm/ssvm-check.sh
>> >>>>>>>>>
>> >>>>>>>>> - Si
>> >>>>>>>>> ________________________________________
>> >>>>>>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au>
>> >>>>>>>>> Sent: Tuesday, March 15, 2016 7:49 PM
>> >>>>>>>>> To: users@cloudstack.apache.org
>> >>>>>>>>> Subject: Re: Unable to create a New VM - Error message: Unable
>> >>> to
>> >>>>>>> start
>> >>>>>>>>> instance due to Unable to get answer that is of class
>> >>>>>>>>> com.cloud.agent.api.StartAnswer
>> >>>>>>>>>
>> >>>>>>>>> Hi Simon,
>> >>>>>>>>>
>> >>>>>>>>> Please see details below:
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> mysql> use cloud; select id, name, status, type from host
>> >> where
>> >>>> id =
>> >>>>>>> 5;
>> >>>>>>>>> Reading table information for completion of table and column
>> >>> names
>> >>>>>>>>> You can turn off this feature to get a quicker startup with -A
>> >>>>>>>>>
>> >>>>>>>>> Database changed
>> >>>>>>>>> +----+------------------------------+--------+---------+
>> >>>>>>>>> | id | name                         | status | type    |
>> >>>>>>>>> +----+------------------------------+--------+---------+
>> >>>>>>>>> | 5 | eqx-cs-cmp-02.ipscape.com.au | up | routing |
>> >>>>>>>>> +----+------------------------------+--------+---------+
>> >>>>>>>>> 1 row in set (0.00 sec)
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> cat /var/log/cloudstack/agent/agent.log
>> >>>>>>>>> com.cloud.utils.exception.CloudRuntimeException:
>> >>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Can't find
>> >>>>>>>>> volume:089ff3ca-9bff-402c-a14e-ea7a35aaad5c
>> >>>>>>>>>         at
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getPhysicalDisk(KVMStoragePoolManager.java:262)
>> >>>>>>>>>         at
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.createVbd(LibvirtComputingResource.java:3921)
>> >>>>>>>>>         at
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:3789)
>> >>>>>>>>>         at
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1333)
>> >>>>>>>>>         at
>> >> com.cloud.agent.Agent.processRequest(Agent.java:501)
>> >>>>>>>>>         at
>> >>>>>>>> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:808)
>> >>>>>>>>>         at com.cloud.utils.nio.Task.run(Task.java:84)
>> >>>>>>>>>         at
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> >>>>>>>>>         at
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> >>>>>>>>>         at java.lang.Thread.run(Thread.java:745)
>> >>>>>>>>> agent.log.2016-03-03.gz (END)
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> [root@eqx-cs-cmp-02 agent]# virsh pool-list
>> >>>>>>>>> Name                 State      Autostart
>> >>>>>>>>> -----------------------------------------
>> >>>>>>>>> 2d0967f8-50d6-4d6d-87b7-4da0596f6924 active     no
>> >>>>>>>>> b3647480-5709-36eb-9ddd-e2c666f30b57 active     no
>> >>>>>>>>> e3831246-ff55-4af4-9964-a17565edaccc active no
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> [Root @ eqx-cs-cmp-02 agent] # df -h
>> >>>>>>>>> Filesystem            Size  Used Avail Use% Mounted on
>> >>>>>>>>> /dev/mapper/vg_root-lv_root
>> >>>>>>>>>                       203G 215G 1.3G 1% /
>> >>>>>>>>> tmpfs 48g 48g 0 0% / dev / shm
>> >>>>>>>>> / Dev / sda1 477m 427m 25M 6% / boot
>> >>>>>>>>> xx.xx.xx.xx:/Cloudstack/primary
>> >>>>>>>>>                       7.6T  4.1T  3.5T  54%
>> >>>>>>>>> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> I also noticed the following in cloudstack-agent.err
>> >>>>>>>>>
>> >>>>>>>>> libvirt: QEMU Driver error : Domain not found: no domain with
>> >>>>> matching
>> >>>>>>>> name
>> >>>>>>>>> 'i-11-600-VM'
>> >>>>>>>>> libvirt: Storage Driver error : Storage volume not found: no
>> >>>> storage
>> >>>>>>> vol
>> >>>>>>>>> with matching name '089ff3ca-9bff-402c-a14e-ea7a35aaad5c'
>> >>>>>>>>>
>> >>>>>>>>> Your help is much appreciated,
>> >>>>>>>>> Nezar
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> On 16 March 2016 at 11:03, Simon Weller <swel...@ena.com>
>> >>> wrote:
>> >>>>>>>>>
>> >>>>>>>>>> Nezar,
>> >>>>>>>>>>
>> >>>>>>>>>> ok, I just took a look at a pastebin you posted.
>> >>>>>>>>>>
>> >>>>>>>>>> I see it complaining about the inability to find a volume
>> >>> prior
>> >>>> to
>> >>>>>>>>>> throwing the host exception.
>> >>>>>>>>>>
>> >>>>>>>>>> Firstly, let's find out who host 5 is. Can you log into your
>> >>>> MySQL
>> >>>>>>>> server
>> >>>>>>>>>> and run this:
>> >>>>>>>>>>
>> >>>>>>>>>> use cloud;select id,name,status,type from host where id=5;
>> >>>>>>>>>>
>> >>>>>>>>>> Can you post the results?
>> >>>>>>>>>>
>> >>>>>>>>>> If it's a host, can you post the
>> >>>>> /var/log/cloudstack/agent/agent.log
>> >>>>>>>> from
>> >>>>>>>>>> that host?
>> >>>>>>>>>>
>> >>>>>>>>>> And also run virsh pool-list on the host in question?
>> >>>>>>>>>>
>> >>>>>>>>>> - Si
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> ________________________________________
>> >>>>>>>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au>
>> >>>>>>>>>> Sent: Tuesday, March 15, 2016 6:11 PM
>> >>>>>>>>>> To: users@cloudstack.apache.org
>> >>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
>> >> Unable
>> >>>> to
>> >>>>>>> start
>> >>>>>>>>>> instance due to Unable to get answer that is of class
>> >>>>>>>>>> com.cloud.agent.api.StartAnswer
>> >>>>>>>>>>
>> >>>>>>>>>> Hi Simon,
>> >>>>>>>>>>
>> >>>>>>>>>> When you say "double check your routers tied to the
>> >> network",
>> >>> do
>> >>>>> you
>> >>>>>>>> mean
>> >>>>>>>>>> the physical routers or the virtual routers?
>> >>>>>>>>>>
>> >>>>>>>>>> When I try to create a VM in Prod Environment it fails. I
>> >>> can't
>> >>>>> ping
>> >>>>>>>> the
>> >>>>>>>>>> Prod virtual router from management server or Host.
>> >>>>>>>>>>
>> >>>>>>>>>> But I can ping Dev virtual router from management server and
>> >>>> Host
>> >>>>>>> and I
>> >>>>>>>>> was
>> >>>>>>>>>> able to create a VM in Dev Environment.
>> >>>>>>>>>>
>> >>>>>>>>>> How can I troubleshoot the Prod Virtual router. I launched
>> >> the
>> >>>>>>> console
>> >>>>>>>>> but
>> >>>>>>>>>> couldn't login.
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> Please note that we need to take into consideration that the
>> >>>>>>> affected
>> >>>>>>>>>> network is the Production network and we need to minimise
>> >>>> service
>> >>>>>>>> outage
>> >>>>>>>>> as
>> >>>>>>>>>> much as possible.
>> >>>>>>>>>>
>> >>>>>>>>>> Your help is much appreciated,
>> >>>>>>>>>> Nezar
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> On 16 March 2016 at 08:37, Simon Weller <swel...@ena.com>
>> >>>> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>>> I've seen this before where a Virtual Router has crashed
>> >> and
>> >>>> the
>> >>>>>>>>>>> management server can't communicate with it.
>> >>>>>>>>>>> Are you using an advanced network setup?
>> >>>>>>>>>>>
>> >>>>>>>>>>> If so, double check your routers tied to the network where
>> >>> you
>> >>>>> are
>> >>>>>>>>>>> attempting to start an instance.
>> >>>>>>>>>>>
>> >>>>>>>>>>> - Si
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> ________________________________________
>> >>>>>>>>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au>
>> >>>>>>>>>>> Sent: Tuesday, March 15, 2016 4:02 PM
>> >>>>>>>>>>> To: users@cloudstack.apache.org
>> >>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
>> >>> Unable
>> >>>>> to
>> >>>>>>>> start
>> >>>>>>>>>>> instance due to Unable to get answer that is of class
>> >>>>>>>>>>> com.cloud.agent.api.StartAnswer
>> >>>>>>>>>>>
>> >>>>>>>>>>> Hi Glenn,
>> >>>>>>>>>>>
>> >>>>>>>>>>> We have 16 hosts, all KVM.
>> >>>>>>>>>>>
>> >>>>>>>>>>> The error is not specific to Host 5. Every time I try to
>> >>>> create
>> >>>>> a
>> >>>>>>> VM
>> >>>>>>>> it
>> >>>>>>>>>>> gives me the same error message but on different host.
>> >>>>>>>>>>>
>> >>>>>>>>>>> How do I restart the agent on the host. Is it service
>> >>>> affecting?
>> >>>>>>>>>>>
>> >>>>>>>>>>> Thanks very much for your help.
>> >>>>>>>>>>> Neza
>> >>>>>>>>>>>
>> >>>>>>>>>>> On 15 March 2016 at 17:37, Glenn Wagner <
>> >>>>>>> glenn.wag...@shapeblue.com>
>> >>>>>>>>>>> wrote:
>> >>>>>>>>>>>
>> >>>>>>>>>>>> Hi,
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Can you explain your environment first , how many hosts
>> >> ,
>> >>>> what
>> >>>>>>> is
>> >>>>>>>>> your
>> >>>>>>>>>>>> hypervisors (KVM,Xenserver,VMware)
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> The error seems to be with your Host 5 Hypervisor
>> >>>>>>>>>>>> Resource [Host:5] is unreachable:
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Go to the infrastructure section and see what host5 is
>> >>>>>>> reporting,
>> >>>>>>>>>>>> Have you tried to restart the agent on the Host?
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Thanks
>> >>>>>>>>>>>> Glenn
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> [image: ShapeBlue] <http://www.shapeblue.com>
>> >>>>>>>>>>>> Glenn Wagner
>> >>>>>>>>>>>> Senior Consultant ,  ShapeBlue
>> >>>>>>>>>>>> d:  * | s: +27 21 527 0091*
>> >>>>>>> <%7C%20s:%20+27%2021%20527%200091>  |
>> >>>>>>>>> m:
>> >>>>>>>>>>>> *+27 73 917 4111* <+27%2073%20917%204111>
>> >>>>>>>>>>>> e:  *glenn.wag...@shapeblue.com | t: *
>> >>>>>>>>>>>> <glenn.wag...@shapeblue.com%20%7C%20t:>  |  w:  *
>> >>>>>>> www.shapeblue.com
>> >>>>>>>> *
>> >>>>>>>>>>>> <http://www.shapeblue.com>
>> >>>>>>>>>>>> a:
>> >>>>>>>>>>>> 2nd Floor, Oudehuis Centre, 122 Main Rd, Somerset West
>> >>> Cape
>> >>>>> Town
>> >>>>>>>> 7130
>> >>>>>>>>>>> South Africa
>> >>>>>>>>>>>> Shape Blue Ltd is a company incorporated in England &
>> >>> Wales.
>> >>>>>>>>> ShapeBlue
>> >>>>>>>>>>>> Services India LLP is a company incorporated in India
>> >> and
>> >>> is
>> >>>>>>>> operated
>> >>>>>>>>>>> under
>> >>>>>>>>>>>> license from Shape Blue Ltd. Shape Blue Brasil
>> >> Consultoria
>> >>>>> Ltda
>> >>>>>>> is
>> >>>>>>>> a
>> >>>>>>>>>>>> company incorporated in Brasil and is operated under
>> >>> license
>> >>>>>>> from
>> >>>>>>>>> Shape
>> >>>>>>>>>>>> Blue Ltd. ShapeBlue SA Pty Ltd is a company registered
>> >> by
>> >>>> The
>> >>>>>>>>> Republic
>> >>>>>>>>>> of
>> >>>>>>>>>>>> South Africa and is traded under license from Shape Blue
>> >>>> Ltd.
>> >>>>>>>>> ShapeBlue
>> >>>>>>>>>>> is
>> >>>>>>>>>>>> a registered trademark.
>> >>>>>>>>>>>> This email and any attachments to it may be confidential
>> >>> and
>> >>>>> are
>> >>>>>>>>>> intended
>> >>>>>>>>>>>> solely for the use of the individual to whom it is
>> >>>> addressed.
>> >>>>>>> Any
>> >>>>>>>>> views
>> >>>>>>>>>>> or
>> >>>>>>>>>>>> opinions expressed are solely those of the author and do
>> >>> not
>> >>>>>>>>>> necessarily
>> >>>>>>>>>>>> represent those of Shape Blue Ltd or related companies.
>> >> If
>> >>>> you
>> >>>>>>> are
>> >>>>>>>>> not
>> >>>>>>>>>>> the
>> >>>>>>>>>>>> intended recipient of this email, you must neither take
>> >>> any
>> >>>>>>> action
>> >>>>>>>>>> based
>> >>>>>>>>>>>> upon its contents, nor copy or show it to anyone. Please
>> >>>>> contact
>> >>>>>>>> the
>> >>>>>>>>>>> sender
>> >>>>>>>>>>>> if you believe you have received this email in error.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> -----Original Message-----
>> >>>>>>>>>>>> From: Nezar Madbouh [mailto:
>> >> nezar.madb...@ipscape.com.au]
>> >>>>>>>>>>>> Sent: Tuesday, 15 March 2016 5:45 AM
>> >>>>>>>>>>>> To: users@cloudstack.apache.org
>> >>>>>>>>>>>> Subject: Re: Unable to create a New VM - Error message:
>> >>>> Unable
>> >>>>>>> to
>> >>>>>>>>> start
>> >>>>>>>>>>>> instance due to Unable to get answer that is of class
>> >>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Hi Jeremy,
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> I have uploaded the contents to
>> >>>> http://pastebin.com/1J3mmgTh
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> your help is much appreciated,
>> >>>>>>>>>>>> Nezar.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> On 15 March 2016 at 13:46, Jeremy Peterson <
>> >>>>>>> jpeter...@acentek.net>
>> >>>>>>>>>>> wrote:
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>> I don't see any attachments can you create a pastebin.
>> >>>>> That's
>> >>>>>>>>> easier
>> >>>>>>>>>>>>> to look through as I think attachments are removed
>> >>> before
>> >>>>>>> being
>> >>>>>>>>> sent
>> >>>>>>>>>>>>> to user group.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Jeremy
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> -------- Original message --------
>> >>>>>>>>>>>>> From: Nezar Madbouh <nezar.madb...@ipscape.com.au>
>> >>>>>>>>>>>>> Date: 3/14/2016 6:38 PM (GMT-06:00)
>> >>>>>>>>>>>>> To: users@cloudstack.apache.org
>> >>>>>>>>>>>>> Subject: Unable to create a New VM - Error message:
>> >>> Unable
>> >>>>> to
>> >>>>>>>> start
>> >>>>>>>>>>>>> instance due to Unable to get answer that is of class
>> >>>>>>>>>>>>> com.cloud.agent.api.StartAnswer
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Hi Shweta and Sadhu,
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Please find attached managerment-server.log which
>> >>> contains
>> >>>>> the
>> >>>>>>>> logs
>> >>>>>>>>>>>>> since I started creating a new VM (VM id 600).
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Please note that we don't have agent.log files
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Your help is much appreciated.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>> Regards,
>> >>>>>>>>>>>>> Nezar.
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>> Find out more about ShapeBlue and our range of
>> >> CloudStack
>> >>>>>>> related
>> >>>>>>>>>>> services:
>> >>>>>>>>>>>> IaaS Cloud Design & Build
>> >>>>>>>>>>>> <http://shapeblue.com/iaas-cloud-design-and-build//> |
>> >>>>> CSForge
>> >>>>>>> -
>> >>>>>>>>> rapid
>> >>>>>>>>>>>> IaaS deployment framework <
>> >> http://shapeblue.com/csforge/>
>> >>>>>>>>>>>> CloudStack Consulting <
>> >>>>>>>> http://shapeblue.com/cloudstack-consultancy/>
>> >>>>>>>>> |
>> >>>>>>>>>>> CloudStack
>> >>>>>>>>>>>> Software Engineering
>> >>>>>>>>>>>> <http://shapeblue.com/cloudstack-software-engineering/>
>> >>>>>>>>>>>> CloudStack Infrastructure Support
>> >>>>>>>>>>>> <
>> >> http://shapeblue.com/cloudstack-infrastructure-support/>
>> >>> |
>> >>>>>>>>> CloudStack
>> >>>>>>>>>>>> Bootcamp Training Courses <
>> >>>>>>>> http://shapeblue.com/cloudstack-training/
>> >>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> DISCLAIMER
>> >>>>>>>> ==========
>> >>>>>>>> This e-mail may contain privileged and confidential information
>> >>>> which
>> >>>>> is
>> >>>>>>>> the property of Accelerite, a Persistent Systems business. It is
>> >>>>>>> intended
>> >>>>>>>> only for the use of the individual or entity to which it is
>> >>>> addressed.
>> >>>>>>> If
>> >>>>>>>> you are not the intended recipient, you are not authorized to
>> >>> read,
>> >>>>>>> retain,
>> >>>>>>>> copy, print, distribute or use this message. If you have
>> >> received
>> >>>> this
>> >>>>>>>> communication in error, please notify the sender and delete all
>> >>>> copies
>> >>>>>>> of
>> >>>>>>>> this message. Accelerite, a Persistent Systems business does not
>> >>>>> accept
>> >>>>>>> any
>> >>>>>>>> liability for virus infected mails.
>> >>>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>> >>
>> >
>>
>
>

Reply via email to