Re: Primary storage not mounted on hosts?

[Simon Weller]

Richard,

The Cloudstack-agent should populate the libvirt pool-list when it starts up.
Have you tried restarting libvirtd and then restarting the Cloudstack-agent?

You may want to turn up debugging on the agent so you get some more detail on 
what's going on.
You can do this by modifying /etc/cloudstack/agent/log4j-cloud.xml
See this wiki article for more details: 
https://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+debug

- Si


From: Richard Klein (RSI) 
Sent: Friday, April 15, 2016 6:54 PM
To: users@cloudstack.apache.org
Subject: Primary storage not mounted on hosts?

I am not sure what happened but our primary storage, which is Gluster, on all 
our hosts is not mounted anymore.  When I do "virsh pool-list" on any host I 
only see the local pool.  Gluster is working fine and there are no problems 
with it because I can mount the Gluster volume manually on any of the hosts and 
see the primary storage.  Instances that are running can write data to the 
local volume and pull data from it.  But if a VM is stopped it can't start 
again.  I get the "Unable to create a New VM - Error message: Unable to start 
instance due to Unable to get answer that is of class 
com.cloud.agent.api.StartAnswer" that I have seen a thread in this mailing list 
and I am sure its primary storage related.

The agent logs on the hosts are issuing the following log snippets which 
confirm its looking for primary storage:

2016-04-15 18:42:34,838 INFO  [kvm.storage.LibvirtStorageAdaptor] 
(agentRequest-Handler-3:null) (logid:ad8ec05a) Trying to fetch storage pool 
c3991ea2-b702-3b1b-bfc5-69cb7d928554 from libvirt
2016-04-15 18:45:19,006 INFO  [kvm.storage.LibvirtStorageAdaptor] 
(agentRequest-Handler-1:null) (logid:4c396753) Trying to fetch storage pool 
c3991ea2-b702-3b1b-bfc5-69cb7d928554 from libvirt
2016-04-15 18:45:49,010 INFO  [kvm.storage.LibvirtStorageAdaptor] 
(agentRequest-Handler-1:null) (logid:4c396753) Trying to fetch storage pool 
c3991ea2-b702-3b1b-bfc5-69cb7d928554 from libvirt

The c3991ea2-b702-3b1b-bfc5-69cb7d928554 is the UUID of our primary storage.

We did have some secondary storage issues (NFS) that caused some NFS mounts to 
secondary storage to hang.  The only way to recover was to reboot the host.  
There were 2 host affected so I put each host in maintenance mode, rebooted and 
then canceled maintenance mode.  I did this one host at a time.  It seems like 
ever since this has happened I have had issues.

Is there a way to get the primary storage remounted and added to libvirt 
pool-list while keeping the VMs up and running?  At this point the only idea I 
have to recover is to power off all VMs, disable primary storage then enable it 
again.  This is a little extreme and is a last resort but I don't know what 
other options I have.

Any suggestions?


Richard Klein  
RSI
5426 Guadalupe, Suite 100
Austin TX 78751

Primary storage not mounted on hosts?

[Richard Klein (RSI)]

I am not sure what happened but our primary storage, which is Gluster, on all 
our hosts is not mounted anymore.  When I do "virsh pool-list" on any host I 
only see the local pool.  Gluster is working fine and there are no problems 
with it because I can mount the Gluster volume manually on any of the hosts and 
see the primary storage.  Instances that are running can write data to the 
local volume and pull data from it.  But if a VM is stopped it can't start 
again.  I get the "Unable to create a New VM - Error message: Unable to start 
instance due to Unable to get answer that is of class 
com.cloud.agent.api.StartAnswer" that I have seen a thread in this mailing list 
and I am sure its primary storage related. 

The agent logs on the hosts are issuing the following log snippets which 
confirm its looking for primary storage:

2016-04-15 18:42:34,838 INFO  [kvm.storage.LibvirtStorageAdaptor] 
(agentRequest-Handler-3:null) (logid:ad8ec05a) Trying to fetch storage pool 
c3991ea2-b702-3b1b-bfc5-69cb7d928554 from libvirt
2016-04-15 18:45:19,006 INFO  [kvm.storage.LibvirtStorageAdaptor] 
(agentRequest-Handler-1:null) (logid:4c396753) Trying to fetch storage pool 
c3991ea2-b702-3b1b-bfc5-69cb7d928554 from libvirt
2016-04-15 18:45:49,010 INFO  [kvm.storage.LibvirtStorageAdaptor] 
(agentRequest-Handler-1:null) (logid:4c396753) Trying to fetch storage pool 
c3991ea2-b702-3b1b-bfc5-69cb7d928554 from libvirt

The c3991ea2-b702-3b1b-bfc5-69cb7d928554 is the UUID of our primary storage.

We did have some secondary storage issues (NFS) that caused some NFS mounts to 
secondary storage to hang.  The only way to recover was to reboot the host.  
There were 2 host affected so I put each host in maintenance mode, rebooted and 
then canceled maintenance mode.  I did this one host at a time.  It seems like 
ever since this has happened I have had issues.

Is there a way to get the primary storage remounted and added to libvirt 
pool-list while keeping the VMs up and running?  At this point the only idea I 
have to recover is to power off all VMs, disable primary storage then enable it 
again.  This is a little extreme and is a last resort but I don't know what 
other options I have.

Any suggestions?


Richard Klein   
RSI 
5426 Guadalupe, Suite 100 
Austin TX 78751 

Re: Cloudstack 4.7 password reset issue - resolved.

[Rafael Weingärtner]

Those are great improvements, please create the PRs that we will review and
help you prepare the code to be merged. Just be aware that we are going to
have a feature freeze starting next Monday (if I am not mistaken) until the
4.9 version is closed.

On Fri, Apr 15, 2016 at 3:12 PM, Richard Klein (RSI) 
wrote:

> Thanks for the replay and I like the 3 PR as well.  The changes are
> related to the environment we are using.  We are using CS 4.7.0 on CentOS7
> and VXLAN with a two 10Mb NICs per host that is bonded via multi-stack LACP
> switch for performance and redundancy.  We ran into the following 3 issues
> with this configuration.
>
> 1) The bond0 interface had 2 bridge interfaces.  The "cloudbr0" is bridged
> to bond0.  It's used for management and VLAN related guest networks. We
> were limited to 1500 MTU in order to be compatible with our infrastructure
> switches.   The "cloudbr1" is bridge to "bond0." where "" is an
> internal VLAN ID that is restricted to Gluster and VXLAN traffic as well as
> using jumbo frames.  The problem was with the logic in the "modifyvxlan.sh"
> script.  When creating a VXLAN isolated network it would always use the
> "cloudbr0" interface even though the traffic label was defined as
> "cloudbr1".  This was due to how the physical interface search was being
> performed.  There was even a "TODO" in the script that suggesting passing
> the traffic label instead of doing a search.  So I modified the Java back
> end code and the "modifyvslan.sh" script to pass the traffic label and this
> corrected the problem and used the "cloudbr1" interface.
>
> 2) We discovered a very odd problem when migrating instances that use
> VXLAN.  For some reason we always got an error like "brbond0" not found.  I
> can look up the exact verbiage but basically it was not able to create the
> proper VXLAN bridges and such on the target host because it couldn't find
> the physical interface of "brbond0" which is true.  The bridge interface
> should have been "brvx-x".  This turned out to be a problem with
> libvirt and not cloudstack.  You could replicate the problem completely
> isolated from the Cloudstack environment.  For some reason I still can't
> explain, libvirt would always fail migrating a VXLAN.  Oddly enough this
> only happened if the VXLAN bridge interface created by CS started with
> "brvx-" (- is the VXLAN ID). Even if manually created and migrated
> (outside of CS control) we found that this would happen so it behaved as
> bug in libvirt not CS.  The solution was quite simple.  We change the code
> that builds the VXLAN bridges to use "vxbr-".  I really can't explain
> why but the name changed resolved the issue.
>
> 3) This last one is the password change issue we've been discussing which
> as you know was resolved by changing the "configure.py" script.
>
> I will try a pull request on number 3 first.  This is the simplest way for
> me to get use to the process.  I will then follow up on the others as time
> permits.  We are about to implement CS in a production environment so
> hopefully I can get to the PR soon.
>
>
> Richard Klein  
> RSI
> 5426 Guadalupe, Suite 100
> Austin TX 78751
>
>
>
>
>
> > -Original Message-
> > From: Rafael Weingärtner [mailto:rafaelweingart...@gmail.com]
> > Sent: Friday, April 15, 2016 12:02 PM
> > To: users@cloudstack.apache.org
> > Subject: Re: Cloudstack 4.7 password reset issue - resolved.
> >
> > Would it be possible for you to explain a little bit these changes?
> >
> > I believe a PR per change would be the best way to go.
> >
> > On Fri, Apr 15, 2016 at 1:31 PM, Richard Klein (RSI) 
> > wrote:
> >
> > > I would be happy to submit a pull request but I am relatively new to
> > > using Git and GitHub.  I have a lot of experience with SVN and CVS.  I
> > > have read the following link about the process:
> > >
> > >* https://cloudstack.apache.org/developers.html
> > >* https://help.github.com/articles/creating-a-pull-request/
> > >
> > > I have forked the apache/cloudstack on GitHub and have been making
> > > changes to the 4.7.0 version on a separate branch.  This branch
> > > contains several code changes we had to make in order to get CS to run
> in
> > our environment.
> > > Since I am not familiar with Mavin I have created some non-standard
> > > version numbers in order to distinguish the RPMs and use a private
> > > repository so we can control the upgrade process.
> > >
> > > I see 2 options on submitting a pull request.  First is to submit it
> > > from the existing branch that contains all the modified code we've
> > > made to 4.7.0.  The only downside is it contains a lot of "pom.xml"
> > > version number changes as well.  The Second option is to create a
> > > branch for each of the 3 types of fixes we have made and do a pull
> request
> > for each one.
> > >
> > > Let me know if there are any additional resources I need to read up on
> > > 

RE: Cloudstack 4.7 password reset issue - resolved.

[Richard Klein (RSI)]

Thanks for the replay and I like the 3 PR as well.  The changes are related to 
the environment we are using.  We are using CS 4.7.0 on CentOS7 and VXLAN with 
a two 10Mb NICs per host that is bonded via multi-stack LACP switch for 
performance and redundancy.  We ran into the following 3 issues with this 
configuration.

1) The bond0 interface had 2 bridge interfaces.  The "cloudbr0" is bridged to 
bond0.  It's used for management and VLAN related guest networks. We were 
limited to 1500 MTU in order to be compatible with our infrastructure switches. 
  The "cloudbr1" is bridge to "bond0." where "" is an internal VLAN ID 
that is restricted to Gluster and VXLAN traffic as well as using jumbo frames.  
The problem was with the logic in the "modifyvxlan.sh" script.  When creating a 
VXLAN isolated network it would always use the "cloudbr0" interface even though 
the traffic label was defined as "cloudbr1".  This was due to how the physical 
interface search was being performed.  There was even a "TODO" in the script 
that suggesting passing the traffic label instead of doing a search.  So I 
modified the Java back end code and the "modifyvslan.sh" script to pass the 
traffic label and this corrected the problem and used the "cloudbr1" interface.

2) We discovered a very odd problem when migrating instances that use VXLAN.  
For some reason we always got an error like "brbond0" not found.  I can look up 
the exact verbiage but basically it was not able to create the proper VXLAN 
bridges and such on the target host because it couldn't find the physical 
interface of "brbond0" which is true.  The bridge interface should have been 
"brvx-x".  This turned out to be a problem with libvirt and not cloudstack. 
 You could replicate the problem completely isolated from the Cloudstack 
environment.  For some reason I still can't explain, libvirt would always fail 
migrating a VXLAN.  Oddly enough this only happened if the VXLAN bridge 
interface created by CS started with "brvx-" (- is the VXLAN ID). Even 
if manually created and migrated (outside of CS control) we found that this 
would happen so it behaved as bug in libvirt not CS.  The solution was quite 
simple.  We change the code that builds the VXLAN bridges to use "vxbr-".  
I really can't explain why but the name changed resolved the issue.

3) This last one is the password change issue we've been discussing which as 
you know was resolved by changing the "configure.py" script.

I will try a pull request on number 3 first.  This is the simplest way for me 
to get use to the process.  I will then follow up on the others as time 
permits.  We are about to implement CS in a production environment so hopefully 
I can get to the PR soon.


Richard Klein   
RSI 
5426 Guadalupe, Suite 100 
Austin TX 78751 





> -Original Message-
> From: Rafael Weingärtner [mailto:rafaelweingart...@gmail.com]
> Sent: Friday, April 15, 2016 12:02 PM
> To: users@cloudstack.apache.org
> Subject: Re: Cloudstack 4.7 password reset issue - resolved.
> 
> Would it be possible for you to explain a little bit these changes?
> 
> I believe a PR per change would be the best way to go.
> 
> On Fri, Apr 15, 2016 at 1:31 PM, Richard Klein (RSI) 
> wrote:
> 
> > I would be happy to submit a pull request but I am relatively new to
> > using Git and GitHub.  I have a lot of experience with SVN and CVS.  I
> > have read the following link about the process:
> >
> >* https://cloudstack.apache.org/developers.html
> >* https://help.github.com/articles/creating-a-pull-request/
> >
> > I have forked the apache/cloudstack on GitHub and have been making
> > changes to the 4.7.0 version on a separate branch.  This branch
> > contains several code changes we had to make in order to get CS to run in
> our environment.
> > Since I am not familiar with Mavin I have created some non-standard
> > version numbers in order to distinguish the RPMs and use a private
> > repository so we can control the upgrade process.
> >
> > I see 2 options on submitting a pull request.  First is to submit it
> > from the existing branch that contains all the modified code we've
> > made to 4.7.0.  The only downside is it contains a lot of "pom.xml"
> > version number changes as well.  The Second option is to create a
> > branch for each of the 3 types of fixes we have made and do a pull request
> for each one.
> >
> > Let me know if there are any additional resources I need to read up on
> > and the proper method of submitting a pull request.
> >
> > Thanks!
> >
> > Richard Klein  
> > RSI
> > 5426 Guadalupe, Suite 100
> > Austin TX 78751
> >
> >
> >
> > > -Original Message-
> > > From: Remi Bergsma [mailto:rberg...@schubergphilis.com]
> > > Sent: Thursday, April 14, 2016 3:35 PM
> > > To: users@cloudstack.apache.org
> > > Subject: Re: Cloudstack 4.7 password reset issue - resolved.
> > >
> > > Hi Richard,
> > >
> > > Great 

Re: Unable to create a New VM - Error message: Unable to start instance due to Unable to get answer that is of class com.cloud.agent.api.StartAnswer

[ilya]

Here is what i need to understand, if you are using SharedMountPoint,
then /mnt/storage must be preset on all KVM hosts in a persistent manner
(i.e. fstab).

However, if you are using NFS, you can just let cloudstack manage the
NFS mount points by adding the storage as NFS type.

On 4/15/16 12:46 AM, ilya wrote:
> Can you goto primary storage in cloudstack UI, is this what you have
> defined for your NFS Server?
> 
> "SharedMountPoint","host":"localhost","path":"/mnt/storage"
> 
> If so, please use IP or FQDN instead..
> 
> 
> On 4/14/16 10:39 PM, Nezar Madbouh wrote:
>> Hi Ilya,
>>
>> Here is the output of mount command:
>>
>> [root@eqx-cs-cmp-01 ~]# mount
>> /dev/mapper/vg_root-lv_root on / type ext4 (rw)
>> proc on /proc type proc (rw)
>> sysfs on /sys type sysfs (rw)
>> devpts on /dev/pts type devpts (rw,gid=5,mode=620)
>> tmpfs on /dev/shm type tmpfs (rw)
>> /dev/sda1 on /boot type ext4 (rw)
>> none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
>> sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
>> nfsd on /proc/fs/nfsd type nfsd (rw)
>> 10.61.154.10:/Cloudstack/primary on
>> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57 type nfs
>> (rw,noatime,nfsvers=3,proto=tcp,hard,nolock,intr,addr=10.61.154.10)
>> [root@eqx-cs-cmp-01 ~]#
>>
>>
>> On 15 April 2016 at 15:22, Nezar Madbouh 
>> wrote:
>>
>>> Hi Ilya,
>>>
>>> Thanks for the instructions. I followed them and pasted the information on
>>> pastebin:
>>>
>>> management-server logs job-14938
>>> http://pastebin.com/GAiy8u5H
>>>
>>> Agent logs
>>> http://pastebin.com/mfiADsJc
>>>
>>>
>>> Your help is much appreciated.
>>>
>>> Nezar.
>>>
>>> On 15 April 2016 at 13:58, ilya  wrote:
>>>
 Here is how i would troubleshoot this:

 1) ssh to MS host, run

 tail -f /var/cloudstack/management/management-server.log | grep -i
 StartVMCmdByAdmin

 2) on cloudstack UI, press start on the VM and watch the tail log

 3) on the tail log, you should see a StartVirtualMachine api call show
 up with job ID. For example:

> 2016-04-15 03:41:57,193 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
 (API-Job-Executor-24:ctx-7892387a job-291575) Executing AsyncJobVO
 {id:291575, userId: 157, accountId: 2, instanceType: VirtualMachine,
 instanceId: 7648, cmd:
 org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin, cmdInfo:
 {"id":"7648","response":"json","sessionkey":"mMJsduS9KLKApqH4dPpIlGZdt1Y\u003d","ctxDetails":"{\"com.cloud.vm.VirtualMachine\":7648,\"com.cloud.uservm.UserVm\":7648,\"com.cloud.network.router.VirtualRouter\":7648}","cmdEventType":"VM.START","expires":"2016-04-15T03:51:57+","ctxUserId":"157","signatureversion":"3","httpmethod":"GET","uuid":"7648","ctxAccountId":"2","ctxStartEventId":"314416"},
 cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
 result: null, initMsid: 345051337894, completeMsid: null, lastUpdated:
 null, lastPolled: null, created: null}

 Each task in cloudstack spins up its own job, in this case its
 *job-291575*

 4) now execute
> grep -A3 -B3 job-291575
 /var/cloudstack/management/management-server.log | tee job-291575.log
> grep job-291575 /var/cloudstack/management/management-server.log | grep
 via | head -1

 you should see a response like so,

> 2016-04-15 03:41:59,259 DEBUG [c.c.a.m.ClusteredAgentAttache]
 (Work-Job-Executor-42:ctx-b2c4af6b job-291575/job-291576 ctx-0988152e) Seq
 441-4181310779036795138: Forwarding Seq 441-4181310779036795138:  { Cmd ,
 MgmtId: 345051337894, via: 441(my-kvm-host01.example.com), Ver: v1,
 Flags: 100011,
 [{"com.cloud.agent.api.routing.DhcpEntryCommand":{"vmMac":"06:fe:4e:00:12:ad","vmIpAddress":"10.10.27.178","vmName":"ol6-test-ilya1","defaultRouter":"10.10.27.1","defaultDns":"10.17.27.166","duid":"00:03:00:01:06:fe:4e:00:12:ad","isDefault":true,"executeInSequence":false,"accessDetails":{"zone.network.type":"Advanced","router.guest.ip":"10.10.27.166","router.ip":"169.254.0.238","
 router.name":"r-7301-VM"},"wait":0}}] } to 345051337893

 We are looking for this line "via: 441(my-kvm-host01.example.com)"

 SSH to my-kvm-host01.example.com, and copy the
 /var/log/cloudstack/agent.log to pastebin

 also copy job-291575.log to pastebin

 I'll try to have a look, though when i get chance.


 Also paste the output of mount command on kvm host in question. I'm
 suspecting one of your storage pools maybe bad.

 Lastly, what distro are you running for KVM?

 Regards
 ilya

 On 4/14/16 5:05 PM, Nezar Madbouh wrote:
> Hi Ilya,
>
> I shutdown all the VMs in Pre-prod Environmnet expecting the Pre-prod
> Virtual router to be deleted automatically because no VM is running on
 this
> vlan but it wasn't deleted.
>
> Am I right to expect the VR to be deleted 

Re: Cloudstack 4.7 password reset issue - resolved.

[Rafael Weingärtner]

Would it be possible for you to explain a little bit these changes?

I believe a PR per change would be the best way to go.

On Fri, Apr 15, 2016 at 1:31 PM, Richard Klein (RSI) 
wrote:

> I would be happy to submit a pull request but I am relatively new to using
> Git and GitHub.  I have a lot of experience with SVN and CVS.  I have read
> the following link about the process:
>
>* https://cloudstack.apache.org/developers.html
>* https://help.github.com/articles/creating-a-pull-request/
>
> I have forked the apache/cloudstack on GitHub and have been making changes
> to the 4.7.0 version on a separate branch.  This branch contains several
> code changes we had to make in order to get CS to run in our environment.
> Since I am not familiar with Mavin I have created some non-standard version
> numbers in order to distinguish the RPMs and use a private repository so we
> can control the upgrade process.
>
> I see 2 options on submitting a pull request.  First is to submit it from
> the existing branch that contains all the modified code we've made to
> 4.7.0.  The only downside is it contains a lot of "pom.xml" version number
> changes as well.  The Second option is to create a branch for each of the 3
> types of fixes we have made and do a pull request for each one.
>
> Let me know if there are any additional resources I need to read up on and
> the proper method of submitting a pull request.
>
> Thanks!
>
> Richard Klein  
> RSI
> 5426 Guadalupe, Suite 100
> Austin TX 78751
>
>
>
> > -Original Message-
> > From: Remi Bergsma [mailto:rberg...@schubergphilis.com]
> > Sent: Thursday, April 14, 2016 3:35 PM
> > To: users@cloudstack.apache.org
> > Subject: Re: Cloudstack 4.7 password reset issue - resolved.
> >
> > Hi Richard,
> >
> > Great you fixed it! Can you send the patch of your fix as a spul request
> on
> > github?
> >
> > Required upgrade is yes when the router reports a version older than the
> > minreq.sysvm.version (or similar) global setting. It's used to upgrade
> systemvm
> > templates.
> >
> > Regards, Remi
> >
> > Sent from my iPhone
> >
> > > On 13 Apr 2016, at 22:23, Richard Klein (RSI) 
> wrote:
> > >
> > > I finally found the problem and resolved the issue.  The problem was
> in the
> > Python code change I made.  I had a flag variable that indicated to save
> data
> > when it was changed while processing a list.  This worked fine as long
> as it
> > executed the logic and defined the flag variable.  The problem was during
> > startup when it doesn't go through the loop and the flag variable was
> > undefined.  This cause the "update_config.py" to fail which then bubbled
> back
> > up as an error and prevent the router from starting.
> > >
> > > Once I changed the code and rebuilt the project all worked well and
> the bug
> > is fixed.  Thanks so much for everyone's help.  This process was very
> > educational and looking forward to learning more.
> > >
> > > I do have one question just out of curiosity.  What makes the "Requires
> > Upgrade" column on the Home->Infrastructure->Virtual Router page
> indicated
> > "Yes"?
> > >
> > > Thanks again,
> > >
> > >
> > > Richard Klein  
> > > RSI
> > > 5426 Guadalupe, Suite 100
> > > Austin TX 78751
> > >
> > >
> > >
> > >
> > >> -Original Message-
> > >> From: Rajani Karuturi [mailto:raj...@apache.org]
> > >> Sent: Tuesday, April 12, 2016 6:15 AM
> > >> To: users@cloudstack.apache.org
> > >> Subject: Re: Cloudstack 4.7 password reset issue.
> > >>
> > >> Can you check the vm_instance table for the VR entry and update
> > >> vm_template_id?
> > >>
> > >> This might be helpful
> > >> https://gist.github.com/terbolous/102ae8edd1cda192561c
> > >>
> > >> ~Rajani
> > >>
> > >> On Sat, Apr 9, 2016 at 5:45 AM, Richard Klein (RSI)
> > >> 
> > >> wrote:
> > >>
> > >>> I found the password reset issue and it ended up being a Python
> > >>> script on the VR.  I ended up modifying the
> > >>> "/opt/cloud/bin/configure.py" to resolve the issue.  Basically there
> is a
> > "/etc/cloud/vmpassword.json"
> > >>> file that is updated with the IP/password pair when the GUI password
> > >>> change is performed.  During the power on process the VM
> > >>> configuration info is sent to the router which reads the
> > >>> vmpassword.json file and sends the password changes to the password
> > >>> server cache file.  When the client retrieved the password it was
> > >>> cleared from the password cache file but not the vmpassword.json
> > >>> file.  So every time a VM started the last password reset was sent
> to the
> > password server again.
> > >>>
> > >>> The question I have now is how do I get the system VM template
> > >>> updated with the change?  Since we are using CS v4.7 we used the
> > >>> system template for v4.6 per the installation instructions for
> > >>> CentOS7 and KVM.  I performed the following steps to use a new system
> > VM template:
> 

RE: Cloudstack 4.7 password reset issue - resolved.

[Richard Klein (RSI)]

I would be happy to submit a pull request but I am relatively new to using Git 
and GitHub.  I have a lot of experience with SVN and CVS.  I have read the 
following link about the process:

   * https://cloudstack.apache.org/developers.html
   * https://help.github.com/articles/creating-a-pull-request/

I have forked the apache/cloudstack on GitHub and have been making changes to 
the 4.7.0 version on a separate branch.  This branch contains several code 
changes we had to make in order to get CS to run in our environment.  Since I 
am not familiar with Mavin I have created some non-standard version numbers in 
order to distinguish the RPMs and use a private repository so we can control 
the upgrade process.

I see 2 options on submitting a pull request.  First is to submit it from the 
existing branch that contains all the modified code we've made to 4.7.0.  The 
only downside is it contains a lot of "pom.xml" version number changes as well. 
 The Second option is to create a branch for each of the 3 types of fixes we 
have made and do a pull request for each one.

Let me know if there are any additional resources I need to read up on and the 
proper method of submitting a pull request.

Thanks!

Richard Klein   
RSI 
5426 Guadalupe, Suite 100 
Austin TX 78751 



> -Original Message-
> From: Remi Bergsma [mailto:rberg...@schubergphilis.com]
> Sent: Thursday, April 14, 2016 3:35 PM
> To: users@cloudstack.apache.org
> Subject: Re: Cloudstack 4.7 password reset issue - resolved.
> 
> Hi Richard,
> 
> Great you fixed it! Can you send the patch of your fix as a spul request on
> github?
> 
> Required upgrade is yes when the router reports a version older than the
> minreq.sysvm.version (or similar) global setting. It's used to upgrade 
> systemvm
> templates.
> 
> Regards, Remi
> 
> Sent from my iPhone
> 
> > On 13 Apr 2016, at 22:23, Richard Klein (RSI)  wrote:
> >
> > I finally found the problem and resolved the issue.  The problem was in the
> Python code change I made.  I had a flag variable that indicated to save data
> when it was changed while processing a list.  This worked fine as long as it
> executed the logic and defined the flag variable.  The problem was during
> startup when it doesn't go through the loop and the flag variable was
> undefined.  This cause the "update_config.py" to fail which then bubbled back
> up as an error and prevent the router from starting.
> >
> > Once I changed the code and rebuilt the project all worked well and the bug
> is fixed.  Thanks so much for everyone's help.  This process was very
> educational and looking forward to learning more.
> >
> > I do have one question just out of curiosity.  What makes the "Requires
> Upgrade" column on the Home->Infrastructure->Virtual Router page indicated
> "Yes"?
> >
> > Thanks again,
> >
> >
> > Richard Klein  
> > RSI
> > 5426 Guadalupe, Suite 100
> > Austin TX 78751
> >
> >
> >
> >
> >> -Original Message-
> >> From: Rajani Karuturi [mailto:raj...@apache.org]
> >> Sent: Tuesday, April 12, 2016 6:15 AM
> >> To: users@cloudstack.apache.org
> >> Subject: Re: Cloudstack 4.7 password reset issue.
> >>
> >> Can you check the vm_instance table for the VR entry and update
> >> vm_template_id?
> >>
> >> This might be helpful
> >> https://gist.github.com/terbolous/102ae8edd1cda192561c
> >>
> >> ~Rajani
> >>
> >> On Sat, Apr 9, 2016 at 5:45 AM, Richard Klein (RSI)
> >> 
> >> wrote:
> >>
> >>> I found the password reset issue and it ended up being a Python
> >>> script on the VR.  I ended up modifying the
> >>> "/opt/cloud/bin/configure.py" to resolve the issue.  Basically there is a
> "/etc/cloud/vmpassword.json"
> >>> file that is updated with the IP/password pair when the GUI password
> >>> change is performed.  During the power on process the VM
> >>> configuration info is sent to the router which reads the
> >>> vmpassword.json file and sends the password changes to the password
> >>> server cache file.  When the client retrieved the password it was
> >>> cleared from the password cache file but not the vmpassword.json
> >>> file.  So every time a VM started the last password reset was sent to the
> password server again.
> >>>
> >>> The question I have now is how do I get the system VM template
> >>> updated with the change?  Since we are using CS v4.7 we used the
> >>> system template for v4.6 per the installation instructions for
> >>> CentOS7 and KVM.  I performed the following steps to use a new system
> VM template:
> >>>
> >>> * I copied the system vm template QCOW2 file from secondary storage
> >>> to a work server and made a backup of it.
> >>> * On the work server I mounted the QCOW2 template file using
> >> "guestmount"
> >>> tools and made the code changes to the template.
> >>> * I then copied this modified template file to a web server and
> >>> registered the template in cloudstack with all checkboxes off except
> >>> for
> 

Re: Error: "Unable to add host"

[Cayetano Rodríguez]

Yes, I'm trying to setup everything on a single machine. I followed that
guide and now I've checked all steps in the KVM section. When I execute  "
lsmod | grep kvm "
 I get the following output:
[root@srvr1 ~]# lsmod | grep kvm
kvm   341551  0

In the guide it appears in the output another line (kvm_intel), am I doing
something wrong so that line doesn't appear?

Thank you

2016-04-15 4:06 GMT+02:00 Pavan Bandarupally <
pavan.bandarupa...@accelerite.com>:

> If you have resources, it is recommended to have management server and
> host on separate machines. But just to troubleshoot your environment you
> can check the below link and see if things work.
>
> From my understanding you are trying to setup everything ( management
> server and hotst) on a single machine. The error you face is generally seen
> if the hypervisor host doesn't have KVM agent installed properly (I might
> be wrong here). Can you please go through the below link and make sure that
> you have followed all the required steps (for KVM installation ) and
> confirm if KVM is properly installed (say by checking lsmod | grep kvm ) ?
>
>
> http://docs.cloudstack.apache.org/projects/cloudstack-installation/en/4.6/qig.html
>
> Regards,
> Pavan
>
>
> -Original Message-
> From: Cayetano Rodríguez [mailto:tfg.cloudcomput...@gmail.com]
> Sent: Friday, April 15, 2016 6:08 AM
> To: users@cloudstack.apache.org
> Subject: Re: Error: "Unable to add host"
>
> The managenent ip address is the same as the host ip address, and the same
> as the machine that I'm using...
> I've done this because I've followed the Quick Installation Guide that is
> focused on doing everything in the same machine, and this guide specifies
> to use the same ip addresses in managenent server and the first host.
>
> I am interested in getting started on CloudStack doing this "trial"
> installation, so I don't need lots of resources (and actually don't have
> them at this moment). For this reason I think doing everything in the same
> machine is the best option for me.
>
> If you could help me doing this that would be great, or if it's not
> possible and you think it's better and easier separating the management
> server and host, I could look for another machine to use.
>
> Thank you very much.
>
> El jueves, 14 de abril de 2016, Simon Weller  escribió:
>
> > Can you ping the ip address you supplied from the management server?
> > Are you able to ssh into it using the credentials (from the management
> > server) you're using in the Cloudstack GUI?
> >
> > ICloudstack is really designed to be deployed on separate servers, so
> > mixing management and hosts can complicate things a fair bit.
> >
> > - Si
> >
> >
> > 
> > From: Cayetano Rodríguez >
> > Sent: Thursday, April 14, 2016 12:18 PM
> > To: users@cloudstack.apache.org 
> > Subject: Error: "Unable to add host"
> >
> > Hi everyone!
> >
> > I've followed the "Quick Installation Guide for Centos 6" guide, to
> > install CloudStack in one machine with Centos 6.5. I've followed all
> > steps and finished installing, but I get an error while configuring
> > it. The error appears when CloudStack is trying to add a host. The
> > error says: "Unable to add the host".
> >
> > I've checked /var/log/cloudstack/management/management-server.log and
> > this is the complete error:
> >
> >
> >
> > 2016-04-14 19:10:36,257 DEBUG [c.c.a.ApiServlet]
> > (catalina-exec-1:ctx-3c617426) (logid:25f4748e) ===START===
> > 192.168.1.2 -- GET
> >
> > command=listNetworkOfferings=Enabled=Shared
> > =json&_=1460653836254
> > 2016-04-14 19:10:36,315 DEBUG [c.c.a.ApiServlet]
> > (catalina-exec-1:ctx-3c617426 ctx-5090361b) (logid:25f4748e) ===END===
> > 192.168.1.2 -- GET
> >
> > command=listNetworkOfferings=Enabled=Shared
> > =json&_=1460653836254
> > 2016-04-14 19:10:36,323 DEBUG [c.c.a.ApiServlet]
> > (catalina-exec-14:ctx-8f5ed97e) (logid:30b82c99) ===START===
> > 192.168.1.2
> > -- POST  command=addHost=json
> > 2016-04-14 19:10:36,335 WARN  [c.c.a.d.ParamGenericValidationWorker]
> > (catalina-exec-14:ctx-8f5ed97e ctx-8c36d11e) (logid:30b82c99) Received
> > unknown parameters for command addHost. Unknown parameters :
> > clustertype
> > 2016-04-14 19:10:36,339 INFO  [c.c.r.ResourceManagerImpl]
> > (catalina-exec-14:ctx-8f5ed97e ctx-8c36d11e) (logid:30b82c99) Trying
> > to add a new host at http://192.168.1.2 in data center 1
> > 2016-04-14 19:10:36,343 WARN  [c.c.h.k.d.LibvirtServerDiscoverer]
> > (catalina-exec-14:ctx-8f5ed97e ctx-8c36d11e) (logid:30b82c99)  can't
> > setup agent, due to java.io.IOException: There was a problem while
> > connecting to
> > 192.168.1.2:22 - There was a problem while connecting to
> > 192.168.1.2:22
> > 2016-04-14 19:10:36,343 WARN  [c.c.r.ResourceManagerImpl]
> > (catalina-exec-14:ctx-8f5ed97e ctx-8c36d11e) (logid:30b82c99) Unable
> > to find the server resources at 

Guid is not updated for cluster with specified cluster id

[Mario Giammarco]

Hello,
I had to delete and insert again two servers in a zone.
Now I was able to insert again the server artkvm2 with ip 10.4.1.3 but when 
I try to add again artkvm1 with ip 10.4.1.2 I only get:

Guid is not updated for cluster with specified cluster id; need to wait for 
hosts in this cluster to come up

I have seen other post in the past about this error but they are not 
helpful for me.

Here is the log:

2016-04-15 12:05:11,671 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-
Handler-3:null) (logid:) SeqA 22-1109: Processing Seq 22-1109:  { Cmd , 
MgmtId: -1, via: 22, Ver: v1, Flags: 11, 
[{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":
{"_proxyVmId":3773,"_loadInfo":"{\n  \"connections\": []\n}","wait":0}}] }
2016-04-15 12:05:11,679 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-
Handler-3:null) (logid:) SeqA 22-1109: Sending Seq 22-1109:  { Ans: , 
MgmtId: 345041274374, via: 22, Ver: v1, Flags: 100010, 
[{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }
2016-04-15 12:05:15,955 DEBUG [c.c.a.ApiServlet] (catalina-exec-7:ctx-
47e15670) (logid:4d7b1671) ===START===  10.1.0.38 -- POST  
command=addHost=json
2016-04-15 12:05:15,966 WARN  [c.c.a.d.ParamGenericValidationWorker] 
(catalina-exec-7:ctx-47e15670 ctx-e6b00129) (logid:4d7b1671) Received 
unknown parameters for command addHost. Unknown parameters : clustertype
2016-04-15 12:05:15,970 ERROR [c.c.a.ApiServer] (catalina-exec-7:ctx-
47e15670 ctx-e6b00129) (logid:4d7b1671) unhandled exception executing api 
command: [Ljava.lang.String;@59a74e3e
com.cloud.utils.exception.CloudRuntimeException: Guid is not updated for 
cluster with specified cluster id; need to wait for hosts in this cluster 
to come up
at 
com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.ja
va:588)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:5
7)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImp
l.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(Aop
Utils.java:317)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoin
t(ReflectiveMethodInvocation.java:183)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflec
tiveMethodInvocation.java:150)
at 
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(Expo
seInvocationInterceptor.java:91)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflec
tiveMethodInvocation.java:172)
at 
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopPr
oxy.java:204)
at com.sun.proxy.$Proxy157.discoverHosts(Unknown Source)
at 
org.apache.cloudstack.api.command.admin.host.AddHostCmd.execute(AddHostCmd.
java:142)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
at com.cloud.api.ApiServer.queueCommand(ApiServer.java:698)
at com.cloud.api.ApiServer.handleRequest(ApiServer.java:529)
at 
com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:297)
at com.cloud.api.ApiServlet$1.run(ApiServlet.java:127)
at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(Def
aultManagedContext.java:56)
at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithCo
ntext(DefaultManagedContext.java:103)
at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithCon
text(DefaultManagedContext.java:53)
at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:124)
at com.cloud.api.ApiServlet.doPost(ApiServlet.java:91)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:643)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applicatio
nFilterChain.java:290)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterC
hain.java:206)
at 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.j
ava:233)
at 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.j
ava:191)
at 
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:12
7)
at 
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:10
3)
at 
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:615)
at 
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.jav
a:109)
at 
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at 
org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java
:891)
at 

Re: Unable to create a New VM - Error message: Unable to start instance due to Unable to get answer that is of class com.cloud.agent.api.StartAnswer

[ilya]

as well as output of mount command..

On 4/14/16 10:39 PM, Nezar Madbouh wrote:
> Hi Ilya,
> 
> Here is the output of mount command:
> 
> [root@eqx-cs-cmp-01 ~]# mount
> /dev/mapper/vg_root-lv_root on / type ext4 (rw)
> proc on /proc type proc (rw)
> sysfs on /sys type sysfs (rw)
> devpts on /dev/pts type devpts (rw,gid=5,mode=620)
> tmpfs on /dev/shm type tmpfs (rw)
> /dev/sda1 on /boot type ext4 (rw)
> none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
> sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
> nfsd on /proc/fs/nfsd type nfsd (rw)
> 10.61.154.10:/Cloudstack/primary on
> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57 type nfs
> (rw,noatime,nfsvers=3,proto=tcp,hard,nolock,intr,addr=10.61.154.10)
> [root@eqx-cs-cmp-01 ~]#
> 
> 
> On 15 April 2016 at 15:22, Nezar Madbouh 
> wrote:
> 
>> Hi Ilya,
>>
>> Thanks for the instructions. I followed them and pasted the information on
>> pastebin:
>>
>> management-server logs job-14938
>> http://pastebin.com/GAiy8u5H
>>
>> Agent logs
>> http://pastebin.com/mfiADsJc
>>
>>
>> Your help is much appreciated.
>>
>> Nezar.
>>
>> On 15 April 2016 at 13:58, ilya  wrote:
>>
>>> Here is how i would troubleshoot this:
>>>
>>> 1) ssh to MS host, run
>>>
>>> tail -f /var/cloudstack/management/management-server.log | grep -i
>>> StartVMCmdByAdmin
>>>
>>> 2) on cloudstack UI, press start on the VM and watch the tail log
>>>
>>> 3) on the tail log, you should see a StartVirtualMachine api call show
>>> up with job ID. For example:
>>>
 2016-04-15 03:41:57,193 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> (API-Job-Executor-24:ctx-7892387a job-291575) Executing AsyncJobVO
>>> {id:291575, userId: 157, accountId: 2, instanceType: VirtualMachine,
>>> instanceId: 7648, cmd:
>>> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin, cmdInfo:
>>> {"id":"7648","response":"json","sessionkey":"mMJsduS9KLKApqH4dPpIlGZdt1Y\u003d","ctxDetails":"{\"com.cloud.vm.VirtualMachine\":7648,\"com.cloud.uservm.UserVm\":7648,\"com.cloud.network.router.VirtualRouter\":7648}","cmdEventType":"VM.START","expires":"2016-04-15T03:51:57+","ctxUserId":"157","signatureversion":"3","httpmethod":"GET","uuid":"7648","ctxAccountId":"2","ctxStartEventId":"314416"},
>>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
>>> result: null, initMsid: 345051337894, completeMsid: null, lastUpdated:
>>> null, lastPolled: null, created: null}
>>>
>>> Each task in cloudstack spins up its own job, in this case its
>>> *job-291575*
>>>
>>> 4) now execute
 grep -A3 -B3 job-291575
>>> /var/cloudstack/management/management-server.log | tee job-291575.log
 grep job-291575 /var/cloudstack/management/management-server.log | grep
>>> via | head -1
>>>
>>> you should see a response like so,
>>>
 2016-04-15 03:41:59,259 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>> (Work-Job-Executor-42:ctx-b2c4af6b job-291575/job-291576 ctx-0988152e) Seq
>>> 441-4181310779036795138: Forwarding Seq 441-4181310779036795138:  { Cmd ,
>>> MgmtId: 345051337894, via: 441(my-kvm-host01.example.com), Ver: v1,
>>> Flags: 100011,
>>> [{"com.cloud.agent.api.routing.DhcpEntryCommand":{"vmMac":"06:fe:4e:00:12:ad","vmIpAddress":"10.10.27.178","vmName":"ol6-test-ilya1","defaultRouter":"10.10.27.1","defaultDns":"10.17.27.166","duid":"00:03:00:01:06:fe:4e:00:12:ad","isDefault":true,"executeInSequence":false,"accessDetails":{"zone.network.type":"Advanced","router.guest.ip":"10.10.27.166","router.ip":"169.254.0.238","
>>> router.name":"r-7301-VM"},"wait":0}}] } to 345051337893
>>>
>>> We are looking for this line "via: 441(my-kvm-host01.example.com)"
>>>
>>> SSH to my-kvm-host01.example.com, and copy the
>>> /var/log/cloudstack/agent.log to pastebin
>>>
>>> also copy job-291575.log to pastebin
>>>
>>> I'll try to have a look, though when i get chance.
>>>
>>>
>>> Also paste the output of mount command on kvm host in question. I'm
>>> suspecting one of your storage pools maybe bad.
>>>
>>> Lastly, what distro are you running for KVM?
>>>
>>> Regards
>>> ilya
>>>
>>> On 4/14/16 5:05 PM, Nezar Madbouh wrote:
 Hi Ilya,

 I shutdown all the VMs in Pre-prod Environmnet expecting the Pre-prod
 Virtual router to be deleted automatically because no VM is running on
>>> this
 vlan but it wasn't deleted.

 Am I right to expect the VR to be deleted automatically if no VM is
>>> running
 in that vlan?

 Thanks,
 Nezar.

 On 15 April 2016 at 09:53, Nezar Madbouh 
 wrote:

> Hi Ilya,
>
> I have run the script and it didn't return any results:
>
> [root@eqx-cs-mgt-01 ~]# ./async_jobs.sh
> [root@eqx-cs-mgt-01 ~]#
>
>
> I also ran the command inside mysql:
>
> mysql> SELECT id, user_id, account_id, instance_type, instance_id,
> job_cmd, job_cmd_info, job_dispatcher, job_pending_signals  FROM
> cloud.async_job where job_status=0 and 

Re: Unable to create a New VM - Error message: Unable to start instance due to Unable to get answer that is of class com.cloud.agent.api.StartAnswer

[ilya]

Can you goto primary storage in cloudstack UI, is this what you have
defined for your NFS Server?

"SharedMountPoint","host":"localhost","path":"/mnt/storage"

If so, please use IP or FQDN instead..


On 4/14/16 10:39 PM, Nezar Madbouh wrote:
> Hi Ilya,
> 
> Here is the output of mount command:
> 
> [root@eqx-cs-cmp-01 ~]# mount
> /dev/mapper/vg_root-lv_root on / type ext4 (rw)
> proc on /proc type proc (rw)
> sysfs on /sys type sysfs (rw)
> devpts on /dev/pts type devpts (rw,gid=5,mode=620)
> tmpfs on /dev/shm type tmpfs (rw)
> /dev/sda1 on /boot type ext4 (rw)
> none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
> sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
> nfsd on /proc/fs/nfsd type nfsd (rw)
> 10.61.154.10:/Cloudstack/primary on
> /mnt/b3647480-5709-36eb-9ddd-e2c666f30b57 type nfs
> (rw,noatime,nfsvers=3,proto=tcp,hard,nolock,intr,addr=10.61.154.10)
> [root@eqx-cs-cmp-01 ~]#
> 
> 
> On 15 April 2016 at 15:22, Nezar Madbouh 
> wrote:
> 
>> Hi Ilya,
>>
>> Thanks for the instructions. I followed them and pasted the information on
>> pastebin:
>>
>> management-server logs job-14938
>> http://pastebin.com/GAiy8u5H
>>
>> Agent logs
>> http://pastebin.com/mfiADsJc
>>
>>
>> Your help is much appreciated.
>>
>> Nezar.
>>
>> On 15 April 2016 at 13:58, ilya  wrote:
>>
>>> Here is how i would troubleshoot this:
>>>
>>> 1) ssh to MS host, run
>>>
>>> tail -f /var/cloudstack/management/management-server.log | grep -i
>>> StartVMCmdByAdmin
>>>
>>> 2) on cloudstack UI, press start on the VM and watch the tail log
>>>
>>> 3) on the tail log, you should see a StartVirtualMachine api call show
>>> up with job ID. For example:
>>>
 2016-04-15 03:41:57,193 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>> (API-Job-Executor-24:ctx-7892387a job-291575) Executing AsyncJobVO
>>> {id:291575, userId: 157, accountId: 2, instanceType: VirtualMachine,
>>> instanceId: 7648, cmd:
>>> org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin, cmdInfo:
>>> {"id":"7648","response":"json","sessionkey":"mMJsduS9KLKApqH4dPpIlGZdt1Y\u003d","ctxDetails":"{\"com.cloud.vm.VirtualMachine\":7648,\"com.cloud.uservm.UserVm\":7648,\"com.cloud.network.router.VirtualRouter\":7648}","cmdEventType":"VM.START","expires":"2016-04-15T03:51:57+","ctxUserId":"157","signatureversion":"3","httpmethod":"GET","uuid":"7648","ctxAccountId":"2","ctxStartEventId":"314416"},
>>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
>>> result: null, initMsid: 345051337894, completeMsid: null, lastUpdated:
>>> null, lastPolled: null, created: null}
>>>
>>> Each task in cloudstack spins up its own job, in this case its
>>> *job-291575*
>>>
>>> 4) now execute
 grep -A3 -B3 job-291575
>>> /var/cloudstack/management/management-server.log | tee job-291575.log
 grep job-291575 /var/cloudstack/management/management-server.log | grep
>>> via | head -1
>>>
>>> you should see a response like so,
>>>
 2016-04-15 03:41:59,259 DEBUG [c.c.a.m.ClusteredAgentAttache]
>>> (Work-Job-Executor-42:ctx-b2c4af6b job-291575/job-291576 ctx-0988152e) Seq
>>> 441-4181310779036795138: Forwarding Seq 441-4181310779036795138:  { Cmd ,
>>> MgmtId: 345051337894, via: 441(my-kvm-host01.example.com), Ver: v1,
>>> Flags: 100011,
>>> [{"com.cloud.agent.api.routing.DhcpEntryCommand":{"vmMac":"06:fe:4e:00:12:ad","vmIpAddress":"10.10.27.178","vmName":"ol6-test-ilya1","defaultRouter":"10.10.27.1","defaultDns":"10.17.27.166","duid":"00:03:00:01:06:fe:4e:00:12:ad","isDefault":true,"executeInSequence":false,"accessDetails":{"zone.network.type":"Advanced","router.guest.ip":"10.10.27.166","router.ip":"169.254.0.238","
>>> router.name":"r-7301-VM"},"wait":0}}] } to 345051337893
>>>
>>> We are looking for this line "via: 441(my-kvm-host01.example.com)"
>>>
>>> SSH to my-kvm-host01.example.com, and copy the
>>> /var/log/cloudstack/agent.log to pastebin
>>>
>>> also copy job-291575.log to pastebin
>>>
>>> I'll try to have a look, though when i get chance.
>>>
>>>
>>> Also paste the output of mount command on kvm host in question. I'm
>>> suspecting one of your storage pools maybe bad.
>>>
>>> Lastly, what distro are you running for KVM?
>>>
>>> Regards
>>> ilya
>>>
>>> On 4/14/16 5:05 PM, Nezar Madbouh wrote:
 Hi Ilya,

 I shutdown all the VMs in Pre-prod Environmnet expecting the Pre-prod
 Virtual router to be deleted automatically because no VM is running on
>>> this
 vlan but it wasn't deleted.

 Am I right to expect the VR to be deleted automatically if no VM is
>>> running
 in that vlan?

 Thanks,
 Nezar.

 On 15 April 2016 at 09:53, Nezar Madbouh 
 wrote:

> Hi Ilya,
>
> I have run the script and it didn't return any results:
>
> [root@eqx-cs-mgt-01 ~]# ./async_jobs.sh
> [root@eqx-cs-mgt-01 ~]#
>
>
> I also ran the command inside mysql:
>
> mysql> SELECT id, user_id,