SOLVED: Re: Upgrade from 4.5.2 to 4.7 Vmware broken for me and now 4.8 as well.

[Steve Searles]

I believe I have solved this issue.   It appears that ./package.sh is not 
honoring the —pack noredist but will honor –p noredist.  I noticed that the 
vmware module was being built during the initial compile but not when using the 
packager even though it echoes the proper build env back during the process.

 --pack:simulator:distribution:release noredist —

I changed

./package.sh —pack noredist —distribution centos63

TO

./package.sh –p noredist –d centos63

The package script seems to be having issues with the — format as —distribution 
was not recognized either but –d is.


Hope this helps anyone else who may run into this issue.


Steven Searles | ssearles@zimcom<mailto:ssear...@zimcom.net>
Zimcom Internet Solutions  | www.zimcom.net<http://www.zimcom.net/>
O: 513.231.9500  |  D: 513.233.4130


[cid:0F891D66-A4B8-45B9-A82B-750605BCEAB9]

From: Steve Searles <ssear...@zimcom.net<mailto:ssear...@zimcom.net>>
Reply-To: "users@cloudstack.apache.org<mailto:users@cloudstack.apache.org>" 
<users@cloudstack.apache.org<mailto:users@cloudstack.apache.org>>
Date: Monday, February 29, 2016 at 12:19 PM
To: "users@cloudstack.apache.org<mailto:users@cloudstack.apache.org>" 
<users@cloudstack.apache.org<mailto:users@cloudstack.apache.org>>
Subject: Re: UPDATE: Upgrade from 4.5.2 to 4.7 Vmware broken for me and now 4.8 
as well.

Rohit, thank you for the reply. I will grab your binaries and give it a shot. 
Not sure if others are seeing this build problem so I would like to get it 
resolved for their benefit as well.  I have built CS from source for the last 
several versions and am aware of how to perform a non-oss build.  It seems that 
I must be missing something in the process with 4.6-4.8.  In my lab I still 
have 4.5.2 running with vmware support without issue.  I will walk through the 
process and perhaps someone on list can point out my error.


  1.  Downloaded the newest version apache-cloudstack-4.8.0-src.tar.bz2.
  2.  I noticed that the release notes contain the following caveat “Build 
Dependency when building Apache CloudStack with noredist, it now require the 
file deps/vim25_55.jar available from vSphere SDK 5.5, 
VMware-vSphere-SDK-5.5.0-1284541.zip”  I can assume we should use 
./vsphere-ws/java/JAXWS/lib/vim25.jar from the SDK and rename it? The 
documentation has no mention of this???   Since the MD5 sums match on this file 
and the file that has been renamed via the mail archive as indicated below I 
assume this is true and the documentation needs to be updated? (See Below)
  3.  Downloaded VMware-vSphere-SDK-5.5.0-1284541.zip
  4.  Noticed that VMware-vSphere-SDK-5.5.0-1284541.zip does not contain 
vim25_55.jar as indicated.  (See Below)

[root@csmgmt01 ss]# ls -asl
total 50100
4 drwxr-xr-x.  3 root root 4096 Feb 26 12:42 .
4 dr-xr-x---. 16 root root 4096 Feb 29 11:14 ..
4 drwxr-xr-x.  7 root root 4096 Feb 26 12:44 SDK
50088 -rw-r-.  1 root root 51284262 Feb 26 12:42 
VMware-vSphere-SDK-5.5.0-1284541.zip
[root@csmgmt01 ss]# cd SDK/
[root@csmgmt01 SDK]# ls
eam  open_source_licenses.txt  sms-sdk  spbm  ssoclient  VMware vSphere Mgmt 
SDK-License Agmt.docx  vSphereManagementSDKReadme.html  vsphere-ws
[root@csmgmt01 SDK]# find ./ -name vim25_55.jar
[root@csmgmt01 SDK]#

 5.  Found a version of vim25_55.jar online from this article on the list  
http://mail-archives.apache.org/mod_mbox/cloudstack-dev/201512.mbox/%3ccc7785ef-95ae-4bba-b76c-49f206f78...@shapeblue.com%3E
   so I grabbed it and attempted to use it.
 6. Copied the vim25_55.jar from the site above to the ./deps folder in my 
source tree and ran install-non-oss.sh which completed without error, I also 
checked this file against the file in the official SDK with the wrong name and 
they seem to match.

[root@csmgmt01 deps]# pwd
/usr/local/src/cs-4.8.0/deps
[root@csmgmt01 deps]#
[root@csmgmt01 deps]#
[root@csmgmt01 deps]# ls -lash vim25_55.jar
3.4M -rw-r--r--. 1 root root 3.4M Feb 26 12:57 vim25_55.jar
[root@csmgmt01 deps]# ls -las vim25_55.jar
3480 -rw-r--r--. 1 root root 3561084 Feb 26 12:57 vim25_55.jar
[root@csmgmt01 deps]# md5sum vim25_55.jar
db14abf8347c6244c5143299d97a9abb  vim25_55.jar
[root@csmgmt01 deps]#


[root@csmgmt01 SDK]# md5sum ./vsphere-ws/java/JAXWS/lib/vim25.jar
db14abf8347c6244c5143299d97a9abb  ./vsphere-ws/java/JAXWS/lib/vim25.jar
[root@csmgmt01 SDK]# md5sum /usr/local/src/cs-4.8.0/deps/vim25_55.jar
db14abf8347c6244c5143299d97a9abb  /usr/local/src/cs-4.8.0/deps/vim25_55.jar
[root@csmgmt01 SDK]#


./install-non-oss.sh

[INFO] 
[INFO] Building Maven Stub Project (No POM) 1
[INFO] 
[INFO]
[INFO] --- maven-install-plugin:2.4:install-file (default-cli) @ standalone-pom 
---
[INFO] Installing /usr/local/src/cs-4.8.0/deps/vim25_55.jar to 
/root

Re: UPDATE: Upgrade from 4.5.2 to 4.7 Vmware broken for me and now 4.8 as well.

[Steve Searles]

pany incorporated in 
Brasil and is operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd 
is a company registered by The Republic of South Africa and is traded under 
license from Shape Blue Ltd. ShapeBlue is a registered trademark.
This email and any attachments to it may be confidential and are intended 
solely for the use of the individual to whom it is addressed. Any views or 
opinions expressed are solely those of the author and do not necessarily 
represent those of Shape Blue Ltd or related companies. If you are not the 
intended recipient of this email, you must neither take any action based upon 
its contents, nor copy or show it to anyone. Please contact the sender if you 
believe you have received this email in error.




On 08-Jan-2016, at 10:52 AM, Steve Searles 
<ssear...@zimcom.net<mailto:ssear...@zimcom.net>> wrote:

Can someone please help point me in the right direction on this one.  I 
upgraded 4.5.2 to 4.7 in my lab. Built with all deps and noredist options as 
done for all previous releases.  I saw in the release notes the build 
dependency and resolved that. Build was successful (Confirmed vmware support 
built) and rpm generation was successful.  After loading CS and updating the 
vcenter passwords in the DB and launching CS all of my vmware hosts show 
disconnected and as such any commands are failing to those hosts.  Below is the 
exception I am seeing.

2016-01-08 00:12:29,113 INFO  [c.c.a.m.AgentManagerImpl] (ClusteredAgentManager 
Timer:ctx-4957e5b4) (logid:c087a0ff) Could not to find a Discoverer to load the 
resource: 1 for hypervisor type: VMware
2016-01-08 00:12:29,114 WARN  [c.c.a.m.AgentManagerImpl] (ClusteredAgentManager 
Timer:ctx-4957e5b4) (logid:c087a0ff) Unable to find class 
com.cloud.hypervisor.vmware.resource.VmwareResource
java.lang.ClassNotFoundException: 
com.cloud.hypervisor.vmware.resource.VmwareResource
   at 
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1483)
   at 
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1329)
   at java.lang.Class.forName0(Native Method)
   at java.lang.Class.forName(Class.java:190)
   at 
com.cloud.agent.manager.AgentManagerImpl.loadResourcesWithoutHypervisor(AgentManagerImpl.java:623)
   at 
com.cloud.agent.manager.AgentManagerImpl.loadDirectlyConnectedHost(AgentManagerImpl.java:697)
   at 
com.cloud.agent.manager.ClusteredAgentManagerImpl.scanDirectAgentToLoad(ClusteredAgentManagerImpl.java:217)
   at 
com.cloud.agent.manager.ClusteredAgentManagerImpl.runDirectAgentScanTimerTask(ClusteredAgentManagerImpl.java:182)
   at 
com.cloud.agent.manager.ClusteredAgentManagerImpl.access$100(ClusteredAgentManagerImpl.java:96)
   at 
com.cloud.agent.manager.ClusteredAgentManagerImpl$DirectAgentScanTimerTask.runInContext(ClusteredAgentManagerImpl.java:233)
   at 
org.apache.cloudstack.managed.context.ManagedContextTimerTask$1.runInContext(ManagedContextTimerTask.java:30)
   at 
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
   at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
   at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
   at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
   at 
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
   at 
org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(ManagedContextTimerTask.java:27)
   at java.util.TimerThread.mainLoop(Timer.java:555)
   at java.util.TimerThread.run(Timer.java:505)
2016-01-08 00:12:29,115 WARN  [c.c.a.m.AgentManagerImpl] (ClusteredAgentManager 
Timer:ctx-4957e5b4) (logid:c087a0ff) Unable to load the resource: 1
2016-01-08 00:12:29,116 DEBUG [c.c.h.Status] (ClusteredAgentManager 
Timer:ctx-4957e5b4) (logid:c087a0ff) Transition:[Resource state = Enabled, 
Agent event = AgentDisconnected, Host id = 1, name = 
esx02.z01p1.us.east.cvg.cs.zimcom.net]
2016-01-08 00:12:29,128 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] 
(ClusteredAgentManager Timer:ctx-4957e5b4) (logid:c087a0ff) Loading directly 
connected host 109(esx03.z01p1.us.east.cvg.cs.zimcom.net)
2016-01-08 00:12:29,128 INFO  [c.c.a.m.AgentManagerImpl] (ClusteredAgentManager 
Timer:ctx-4957e5b4) (logid:c087a0ff) Could not to find a Discoverer to load the 
resource: 109 for hypervisor type: VMware
2016-01-08 00:12:29,129 WARN  [c.c.a.m.AgentManagerImpl] (ClusteredAgentManager 
Timer:ctx-4957e5b4) (logid:c087a0ff) Unable to find class 
com.cloud.hypervisor.vmware.resource.VmwareResource
java.lang.ClassNotFoundException: 
com.cloud.hypervisor.vmware.resource.VmwareResource
   at 
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1483)
  

Upgrade from 4.5.2 to 4.7 Vmware broken for me

[Steve Searles]

Can someone please help point me in the right direction on this one.  I 
upgraded 4.5.2 to 4.7 in my lab. Built with all deps and noredist options as 
done for all previous releases.  I saw in the release notes the build 
dependency and resolved that. Build was successful (Confirmed vmware support 
built) and rpm generation was successful.  After loading CS and updating the 
vcenter passwords in the DB and launching CS all of my vmware hosts show 
disconnected and as such any commands are failing to those hosts.  Below is the 
exception I am seeing.

2016-01-08 00:12:29,113 INFO  [c.c.a.m.AgentManagerImpl] (ClusteredAgentManager 
Timer:ctx-4957e5b4) (logid:c087a0ff) Could not to find a Discoverer to load the 
resource: 1 for hypervisor type: VMware
2016-01-08 00:12:29,114 WARN  [c.c.a.m.AgentManagerImpl] (ClusteredAgentManager 
Timer:ctx-4957e5b4) (logid:c087a0ff) Unable to find class 
com.cloud.hypervisor.vmware.resource.VmwareResource
java.lang.ClassNotFoundException: 
com.cloud.hypervisor.vmware.resource.VmwareResource
at 
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1483)
at 
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1329)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:190)
at 
com.cloud.agent.manager.AgentManagerImpl.loadResourcesWithoutHypervisor(AgentManagerImpl.java:623)
at 
com.cloud.agent.manager.AgentManagerImpl.loadDirectlyConnectedHost(AgentManagerImpl.java:697)
at 
com.cloud.agent.manager.ClusteredAgentManagerImpl.scanDirectAgentToLoad(ClusteredAgentManagerImpl.java:217)
at 
com.cloud.agent.manager.ClusteredAgentManagerImpl.runDirectAgentScanTimerTask(ClusteredAgentManagerImpl.java:182)
at 
com.cloud.agent.manager.ClusteredAgentManagerImpl.access$100(ClusteredAgentManagerImpl.java:96)
at 
com.cloud.agent.manager.ClusteredAgentManagerImpl$DirectAgentScanTimerTask.runInContext(ClusteredAgentManagerImpl.java:233)
at 
org.apache.cloudstack.managed.context.ManagedContextTimerTask$1.runInContext(ManagedContextTimerTask.java:30)
at 
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at 
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at 
org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(ManagedContextTimerTask.java:27)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
2016-01-08 00:12:29,115 WARN  [c.c.a.m.AgentManagerImpl] (ClusteredAgentManager 
Timer:ctx-4957e5b4) (logid:c087a0ff) Unable to load the resource: 1
2016-01-08 00:12:29,116 DEBUG [c.c.h.Status] (ClusteredAgentManager 
Timer:ctx-4957e5b4) (logid:c087a0ff) Transition:[Resource state = Enabled, 
Agent event = AgentDisconnected, Host id = 1, name = 
esx02.z01p1.us.east.cvg.cs.zimcom.net]
2016-01-08 00:12:29,128 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] 
(ClusteredAgentManager Timer:ctx-4957e5b4) (logid:c087a0ff) Loading directly 
connected host 109(esx03.z01p1.us.east.cvg.cs.zimcom.net)
2016-01-08 00:12:29,128 INFO  [c.c.a.m.AgentManagerImpl] (ClusteredAgentManager 
Timer:ctx-4957e5b4) (logid:c087a0ff) Could not to find a Discoverer to load the 
resource: 109 for hypervisor type: VMware
2016-01-08 00:12:29,129 WARN  [c.c.a.m.AgentManagerImpl] (ClusteredAgentManager 
Timer:ctx-4957e5b4) (logid:c087a0ff) Unable to find class 
com.cloud.hypervisor.vmware.resource.VmwareResource
java.lang.ClassNotFoundException: 
com.cloud.hypervisor.vmware.resource.VmwareResource
at 
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1483)
at 
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1329)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:190)
at 
com.cloud.agent.manager.AgentManagerImpl.loadResourcesWithoutHypervisor(AgentManagerImpl.java:623)
at 
com.cloud.agent.manager.AgentManagerImpl.loadDirectlyConnectedHost(AgentManagerImpl.java:697)
at 
com.cloud.agent.manager.ClusteredAgentManagerImpl.scanDirectAgentToLoad(ClusteredAgentManagerImpl.java:217)
at 
com.cloud.agent.manager.ClusteredAgentManagerImpl.runDirectAgentScanTimerTask(ClusteredAgentManagerImpl.java:182)
at 
com.cloud.agent.manager.ClusteredAgentManagerImpl.access$100(ClusteredAgentManagerImpl.java:96)
at 

HAProxy Max Sessions

[Steve Searles]

Does anyone know how to change the max backend sessions on the HA proxy load 
balancer? Currently with 2 servers I am seeing a backend session limit of 200. 
Is this something that needs to be changed on the VR by hand? If so how can the 
change be made persistent?   This is what I am seeing from the HAProxy status 
page.

XXX-443

Queue   Session rateSessionsBytes   Denied  Errors  
WarningsServer
Cur Max Limit   Cur Max Limit   Cur Max Limit   Total   
LbTot   LastIn  Out Req RespReq ConnRespRetr
Redis   Status  LastChk WghtAct Bck Chk Dwn Dwntme  Thrtle
Frontend  
1


33


-   161 223 20009554


116492541   403138671
0   0   0   OPEN
XXX-443_0
0   0   -   1   26  33  110 -   3382





209 1s  5289161282984   0   
0   183
68  34  8m13s UPL4OK in 0ms
1   Y   -   3
1   1m4s-
XXX-443_1
0   0   -   1   28  128 158 -   6274





333 1s  64159152241855687   0   
0   261
0   0   22m14s UP   L4OK in 0ms
1   Y   -   0
0   0s  -
Backend
0   0   1   33  161 223 200 9554




542 1s  116492541   403138671
0   0   0   444



- Steve

Re: Trouble Attaching Volume

[Steve Searles]

That was it.  Thanks!.  What is the best practice for DRS and HA with vmware 
for use with Cloudstack? Disabled?

Steven Searles



On Apr 2, 2015, at 2:17 AM, Sanjeev N 
sanj...@apache.orgmailto:sanj...@apache.org wrote:

Also make sure that vm is actually running on the host

On Thu, Apr 2, 2015 at 6:11 AM, Yitao Jiang 
willier...@gmail.commailto:willier...@gmail.com wrote:

I met the same issue before. Cause under vmware cluster ha enabled, so the
vm mirgrated among hosts. You can check the host which vm lives in cloud
databases, and compare to the actual host, if mismatchrd, changing the db
table, thrn all works fine.
On Apr 1, 2015 9:52 PM, Steve Searles 
ssear...@zimcom.netmailto:ssear...@zimcom.net wrote:

I am having trouble attaching a volume to an existing VM.  Adding an
extra
volume during the instance creation works fine.  It does not seem to
matter
what state the VM is in eg. stopped or running I receive the same error.
Anyone know what this means? I can not find any further information on
Message: Unable to find the VM to execute AttachVolumeCommand”, ‘


Cloudstack 4.4.2
Vmware 5.5



Any help would be appreciated.


INFO  [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-1:ctx-4e1f0aca
job-484/job-485) Add job-485 into job monitoring
INFO  [c.c.s.VolumeApiServiceImpl] (Work-Job-Executor-1:ctx-4e1f0aca
job-484/job-485 ctx-3e014d23) Trying to attaching volume 175 to vm
instance:154, update async job-485 progress status
ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-1:ctx-4e1f0aca
job-484/job-485 ctx-3e014d23) Invocation exception, caused by:
com.cloud.utils.exception.CloudRuntimeException: Failed to attach volume
test-cvg to VM RESIZE-TEST-2012; AttachVolumeCommand failed due to
Exception: java.lang.Exception
Message: Unable to find the VM to execute AttachVolumeCommand, vmName:
i-2-154-VM

INFO  [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-1:ctx-4e1f0aca
job-484/job-485 ctx-3e014d23) Rethrow exception
com.cloud.utils.exception.CloudRuntimeException: Failed to attach volume
test-cvg to VM RESIZE-TEST-2012; AttachVolumeCommand failed due to
Exception: java.lang.Exception
Message: Unable to find the VM to execute AttachVolumeCommand, vmName:
i-2-154-VM

ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-1:ctx-4e1f0aca
job-484/job-485) Unable to complete AsyncJobVO {id:485, userId: 2,
accountId: 2, instanceType: null, instanceId: null, cmd:
com.cloud.storage.VmWorkAttachVolume, cmdInfo:

rO0ABXNyACRjb20uY2xvdWQuc3RvcmFnZS5WbVdvcmtBdHRhY2hWb2x1bWUHra_5YYfiHAIAAkwACGRldmljZUlkdAAQTGphdmEvbGFuZy9Mb25nO0wACHZvbHVtZUlkcQB-AAF4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1ldAASTGphdmEvbGFuZy9TdHJpbmc7eHAAAgACAJp0ABRWb2x1bWVBcGlTZXJ2aWNlSW1wbHBzcgAOamF2YS5sYW5nLkxvbmc7i-SQzI8j3wIAAUoABXZhbHVleHIAEGphdmEubGFuZy5OdW1iZXKGrJUdC5TgiwIAAHhwAK8,
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
result: null, initMsid: 345049815164, completeMsid: null, lastUpdated:
null, lastPolled: null, created: Wed Apr 01 09:47:24 EDT 2015}, job
origin:484
com.cloud.utils.exception.CloudRuntimeException: Failed to attach volume
test-cvg to VM RESIZE-TEST-2012; AttachVolumeCommand failed due to
Exception: java.lang.Exception
Message: Unable to find the VM to execute AttachVolumeCommand, vmName:
i-2-154-VM

at

com.cloud.storage.VolumeApiServiceImpl.sendAttachVolumeCommand(VolumeApiServiceImpl.java:2110)
at

com.cloud.storage.VolumeApiServiceImpl.attachVolumeToVM(VolumeApiServiceImpl.java:1247)
at

com.cloud.storage.VolumeApiServiceImpl.orchestrateAttachVolumeToVM(VolumeApiServiceImpl.java:1091)
at

com.cloud.storage.VolumeApiServiceImpl.orchestrateAttachVolumeToVM(VolumeApiServiceImpl.java:2476)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at

sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at

com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at

com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:2511)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at

sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at

org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at

org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at

org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at

org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91

Trouble Attaching Volume

[Steve Searles]

I am having trouble attaching a volume to an existing VM.  Adding an extra 
volume during the instance creation works fine.  It does not seem to matter 
what state the VM is in eg. stopped or running I receive the same error.  
Anyone know what this means? I can not find any further information on 
Message: Unable to find the VM to execute AttachVolumeCommand”, ‘


Cloudstack 4.4.2
Vmware 5.5



Any help would be appreciated.


INFO  [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-1:ctx-4e1f0aca 
job-484/job-485) Add job-485 into job monitoring
INFO  [c.c.s.VolumeApiServiceImpl] (Work-Job-Executor-1:ctx-4e1f0aca 
job-484/job-485 ctx-3e014d23) Trying to attaching volume 175 to vm 
instance:154, update async job-485 progress status
ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-1:ctx-4e1f0aca 
job-484/job-485 ctx-3e014d23) Invocation exception, caused by: 
com.cloud.utils.exception.CloudRuntimeException: Failed to attach volume 
test-cvg to VM RESIZE-TEST-2012; AttachVolumeCommand failed due to Exception: 
java.lang.Exception
Message: Unable to find the VM to execute AttachVolumeCommand, vmName: 
i-2-154-VM

INFO  [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-1:ctx-4e1f0aca 
job-484/job-485 ctx-3e014d23) Rethrow exception 
com.cloud.utils.exception.CloudRuntimeException: Failed to attach volume 
test-cvg to VM RESIZE-TEST-2012; AttachVolumeCommand failed due to Exception: 
java.lang.Exception
Message: Unable to find the VM to execute AttachVolumeCommand, vmName: 
i-2-154-VM

ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-1:ctx-4e1f0aca 
job-484/job-485) Unable to complete AsyncJobVO {id:485, userId: 2, accountId: 
2, instanceType: null, instanceId: null, cmd: 
com.cloud.storage.VmWorkAttachVolume, cmdInfo: 
rO0ABXNyACRjb20uY2xvdWQuc3RvcmFnZS5WbVdvcmtBdHRhY2hWb2x1bWUHra_5YYfiHAIAAkwACGRldmljZUlkdAAQTGphdmEvbGFuZy9Mb25nO0wACHZvbHVtZUlkcQB-AAF4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1ldAASTGphdmEvbGFuZy9TdHJpbmc7eHAAAgACAJp0ABRWb2x1bWVBcGlTZXJ2aWNlSW1wbHBzcgAOamF2YS5sYW5nLkxvbmc7i-SQzI8j3wIAAUoABXZhbHVleHIAEGphdmEubGFuZy5OdW1iZXKGrJUdC5TgiwIAAHhwAK8,
 cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: 
null, initMsid: 345049815164, completeMsid: null, lastUpdated: null, 
lastPolled: null, created: Wed Apr 01 09:47:24 EDT 2015}, job origin:484
com.cloud.utils.exception.CloudRuntimeException: Failed to attach volume 
test-cvg to VM RESIZE-TEST-2012; AttachVolumeCommand failed due to Exception: 
java.lang.Exception
Message: Unable to find the VM to execute AttachVolumeCommand, vmName: 
i-2-154-VM

at 
com.cloud.storage.VolumeApiServiceImpl.sendAttachVolumeCommand(VolumeApiServiceImpl.java:2110)
at 
com.cloud.storage.VolumeApiServiceImpl.attachVolumeToVM(VolumeApiServiceImpl.java:1247)
at 
com.cloud.storage.VolumeApiServiceImpl.orchestrateAttachVolumeToVM(VolumeApiServiceImpl.java:1091)
at 
com.cloud.storage.VolumeApiServiceImpl.orchestrateAttachVolumeToVM(VolumeApiServiceImpl.java:2476)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at 
com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:2511)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at 
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at 
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy183.handleVmWorkJob(Unknown Source)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at 
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
at 
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at 

VMware dvSwitch issues on 4.4.2.

[Steve Searles]

Is anyone running 4.4.2 with a vmware dvSwitch for guest and public traffic 
successfully.  I have enabled vmware.use.dvswitch=true in the global config but 
still am getting the java.lang.NumberFormatException when trying to provision 
an instance.  I have opened a new JIRA issue for this as it seems that the same 
issue was reported in CLOUDSTACK-5046 and has returned.  Below is the relevant 
portion of the management server log.  Does anyone have any idea that might 
help? I have defined my network in the following format for traffic labels 
dvSwitch0,1901,vmwaredvs” Any help would be appreciated.




2015-01-12 10:14:09,535 INFO  [c.c.h.v.r.VmwareResource] 
(DirectAgent-5:ctx-8212ae15 
esx01.p1c1.ushttp://esx01.p1c1.us-east-cvg.cs.zimcom.net) Prepare NIC device 
based on NicTO: 
{deviceId:0,networkRateMbps:200,defaultNic:false,uuid:0a9287b8-656b-4f3a-904f-47ee13a2fe70,ip:172.31.255.1,netmask:255.255.255.0,mac:02:00:16:4a:00:02,dns1:209.212.245.10,dns2:209.212.246.10,broadcastType:Vlan,type:Guest,broadcastUri:vlan://2016,isolationUri:vlan://2016,isSecurityGroupEnabled:false,name:dvSwitch0,1901,vmwaredvs}
2015-01-12 10:14:09,542 INFO  [c.c.h.v.r.VmwareResource] 
(DirectAgent-5:ctx-8212ae15 
esx01.p1c1.ushttp://esx01.p1c1.us-east-cvg.cs.zimcom.net) Prepare network on 
vmwaredvs dvSwitch0 with name prefix: cloud.guest
2015-01-12 10:14:09,550 WARN  [c.c.h.v.r.VmwareResource] 
(DirectAgent-5:ctx-8212ae15 
esx01.p1c1.ushttp://esx01.p1c1.us-east-cvg.cs.zimcom.net) dvsw
Message: null

java.lang.NumberFormatException: null
at java.lang.Integer.parseInt(Integer.java:454)
at java.lang.Integer.parseInt(Integer.java:527)
at 
com.cloud.hypervisor.vmware.mo.HypervisorHostHelper.prepareNetwork(HypervisorHostHelper.java:484)
at 
com.cloud.hypervisor.vmware.resource.VmwareResource.prepareNetworkFromNicInfo(VmwareResource.java:2350)
at 
com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:1606)
at 
com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:448)
at 
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:307)
at 
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at 
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2015-01-12 10:14:09,555 DEBUG [c.c.a.m.DirectAgentAttache] 
(DirectAgent-5:ctx-8212ae15) Seq 1-6221722885212340234: Cancelling because one 
of the answers is false and it is stop on error.






Steven Searles, CTO | ssear...@zimcom.netmailto:ssear...@zimcom.net
Zimcom Internet Solutions  | www.zimcom.nethttp://www.zimcom.net
O: 513.231.9500  |  D: 513.233.4130

[cid:image001.jpg@01CF54D3.D25E1ED0]

Re: ALL Hosts Stuck in Maintenance

[Steve Searles]

Thanks for the reply. Yes that did seem to be the issue.  I did a manual 
cleanup of all of the vm’s that were assigned to that host, cleared out the 
async job queue and restarted the management server.  This seems to have 
resolved the issue system wide.


Steven Searles, CTO | ssear...@zimcom.netmailto:ssear...@zimcom.net
Zimcom Internet Solutions  | www.zimcom.nethttp://www.zimcom.net
O: 513.231.9500  |  D: 513.233.4130

[cid:image001.jpg@01CF54D3.D25E1ED0]

On Nov 22, 2014, at 2:54 PM, Daan Hoogland 
daan.hoogl...@gmail.commailto:daan.hoogl...@gmail.com wrote:

Steven,

I had a look at your stacktrace. It seems a vm is marked to be migrated
back during maintenance mode cancel and it doesn't exist. Did you destroy
any of the vms on the host while it was in maintenance?

There is a number of jobs for migration each of which pertains to a vm. For
one of them the vm does not exist. Doesn't exist in this case means that it
is not in the database. This is a bug, obviously. I can't be sure what
happened during the maintenance mode and thus not what the nature of the
bug exactly is. It would be nice if you could to investigate this.

Daan


On Fri, Nov 21, 2014 at 6:35 PM, Steve Searles 
ssear...@zimcom.netmailto:ssear...@zimcom.net wrote:

Below is the full NPE from the catalina.out if that would help.  This is
the same NPE for all hypervisor types.

INFO  [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-10:ctx-208629c7
job-12679) Add job-12679 into job monitoring
ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-10:ctx-208629c7
job-12679) Unexpected exception while executing
org.apache.cloudstack.api.command.admin.host.CancelMaintenanceCmd
java.lang.NullPointerException
at
com.cloud.resource.ResourceManagerImpl.doCancelMaintenance(ResourceManagerImpl.java:2083)
at
com.cloud.resource.ResourceManagerImpl.cancelMaintenance(ResourceManagerImpl.java:2140)
at
com.cloud.resource.ResourceManagerImpl.cancelMaintenance(ResourceManagerImpl.java:1127)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy148.cancelMaintenance(Unknown Source)
at
org.apache.cloudstack.api.command.admin.host.CancelMaintenanceCmd.execute(CancelMaintenanceCmd.java:102)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141)
at
com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
INFO  [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-10:ctx-208629c7
job-12679) Remove job-12679 from job monitoring


  Steven Searles, CTO | ssear...@zimcom.netmailto:ssear...@zimcom.net
Zimcom Internet Solutions  | www.zimcom.nethttp://www.zimcom.net
O: 513.231.9500  |  D: 513.233.4130


On Nov 21, 2014, at 12:31 PM, Steve Searles 
ssear...@zimcom.netmailto:ssear...@zimcom.net wrote:

In my case the agent is connected fine.  I have tried even restarting
the hosts.  I have the agent log in debug for the KVM server and am not
seeing the mgmt server even trying to execute anything when coming out of
maintenance.  I just get the NPE immediately in the management server
logs.

 Steven Searles

Re: Cloudstack 4.4.1 VPN Routing issues

[Steve Searles]

I was using systemvm64template-4.4.1-vmware.ova.  I will update it and test 
again.


Steven Searles, CTO | ssear...@zimcom.netmailto:ssear...@zimcom.net
Zimcom Internet Solutions  | www.zimcom.nethttp://www.zimcom.net
O: 513.231.9500  |  D: 513.233.4130

[cid:image001.jpg@01CF54D3.D25E1ED0]

On Oct 15, 2014, at 8:51 PM, Pierre-Luc Dion 
pdion...@apache.orgmailto:pdion...@apache.org wrote:

Did test this on 4.4.1-RC3, did work well with OSX client.

I've used
http://cloudstack.apt-get.eu/systemvm/4.4/systemvm64template-4.4.1-7-xen.vhd.bz2
from 4.4.1 release-notes on xenserver.

does the issue still persist?

regards,

On Thu, Oct 9, 2014 at 5:27 PM, Steve Searles ssear...@zimcom.net wrote:

Hello everyone, I am having issues with the remote network route not being
created on the client side when connected to the VPN (Virtual Router) on a
guest network (not site to site).  If I add the route manually on the
client it works fine. It is happening on both windows7 and OSX.  I saw a
few older posts about VPN issues specifically with the OSX client but they
seem to be resolved at this point.  I have the latest 4.4.1 from git and I
am using the new 4.4.1 systemvm’s.  Did I miss a setting on the CS side
somewhere? Any ideas?

Steve Searles

Xenserver Thin Provisioning

[Steve Searles]

I am unable to provision instances including systemvm-templates on xenserver 
SR’s that are configured with the thin attribute specified.  I created 2 SR’s 
as seen below.  The thin SR produces “Invalid Session Cookie”. If I re-create 
the SR not thin provisioned there are no issues.  This error is produced during 
the copy of the systemvm template from secondary storage. Is this configuration 
not supported? These LUN’s are on the same SAN, the only difference I can see 
is the sm-config:allocation=thin attribute set at the time of SR creation.  Any 
help would be appreciated.  (Cloudstack 4.4.1/Xenserver 6.2SP1)




[root@xen01 log]# xe sr-param-list uuid=5e728b55-690c-1208-d42b-3c2b5b80cc22
uuid ( RO): 5e728b55-690c-1208-d42b-3c2b5b80cc22
  name-label ( RW): VNX5400CVG_ZC_CS_XEN_1
name-description ( RW):
host ( RO): xen01.z01p2.us.east.cvg.cs.zimcom.net
  allowed-operations (SRO): VDI.create; VDI.snapshot; PBD.create; 
PBD.destroy; plug; update; VDI.destroy; scan; VDI.clone; VDI.resize; unplug
  current-operations (SRO):
VDIs (SRO):
PBDs (SRO): b55ea29c-21e0-f9fa-64f5-a0422f8afa02
  virtual-allocation ( RO): 0
physical-utilisation ( RO): 8388608
   physical-size ( RO): 1099499044864
type ( RO): lvmohba
content-type ( RO): user
  shared ( RW): true
   introduced-by ( RO): not in database
other-config (MRW):
   sm-config (MRO): allocation: thin; use_vhd: true; multipathable: 
true; devserial: scsi-36006016057613600e17993d17f75e411
   blobs ( RO):
 local-cache-enabled ( RO): false
tags (SRW):


[root@xen01 log]# xe sr-param-list uuid=76824440-4d85-227c-0f1a-3917d0a7ea36
uuid ( RO): 76824440-4d85-227c-0f1a-3917d0a7ea36
  name-label ( RW): VNX5400CVG_ZC_CS_XEN_2
name-description ( RW): Hardware HBA SR [DGC - APM00135175725]
host ( RO): xen01.z01p2.us.east.cvg.cs.zimcom.net
  allowed-operations (SRO): VDI.create; VDI.snapshot; PBD.create; 
PBD.destroy; plug; update; VDI.destroy; scan; VDI.clone; VDI.resize; unplug
  current-operations (SRO):
VDIs (SRO):
PBDs (SRO): 9cab320a-fc4f-c046-7c36-0ba42eea52b1
  virtual-allocation ( RO): 0
physical-utilisation ( RO): 4194304
   physical-size ( RO): 1099499044864
type ( RO): lvmohba
content-type ( RO):
  shared ( RW): true
   introduced-by ( RO): not in database
other-config (MRW): auto-scan: false
   sm-config (MRO): allocation: thick; use_vhd: true; 
multipathable: true; devserial: scsi-36006016057613600ce87f64b1a77e411
   blobs ( RO):
 local-cache-enabled ( RO): false
tags (SRW):

(Xenserver /var/log/messages)

Nov 28 10:39:30 xen01 xapi: [ 
info|xen01.lab.zimcom.nethttp://xen01.lab.zimcom.net|219547 INET 
0.0.0.0:80|sm_exec D:6eb88541a744|xapi] Session.create 
trackid=56714e29817e260dfe98bb3964ffae56 pool=false uname= 
is_local_superuser=true auth_user_sid= 
parent=trackid=9834f5af41c964e225f24279aefe4e49
Nov 28 10:39:30 xen01 vhd-util: libvhd::vhd_validate_footer: invalid footer 
cookie: 
Nov 28 10:39:30 xen01 vhd-util: libvhd::vhd_read_footer_at: 
/dev/VG_XenStorage-5e728b55-690c-1208-d42b-3c2b5b80cc22/VHD-4c0d0407-326a-4a5a-bac6-04ea9ba6ea9e:
 reading footer at 0x007ffe00 failed: -22
Nov 28 10:39:30 xen01 vhd-util: libvhd::vhd_read: 
/dev/VG_XenStorage-5e728b55-690c-1208-d42b-3c2b5b80cc22/VHD-4c0d0407-326a-4a5a-bac6-04ea9ba6ea9e:
 read of 512 returned 0, errno: -22
Nov 28 10:39:30 xen01 vhd-util: libvhd::vhd_validate_footer: invalid footer 
cookie:


(Management Server Log)

2014-11-28 10:39:32,648 DEBUG [c.c.a.ApiServlet] 
(catalina-exec-12:ctx-11bf6733) ===START===  172.23.0.1 -- GET  
command=queryAsyncJobResultjobId=8271b914-7ef4-4e2d-a20e-7cc261abed16response=jsonsessionkey=JcnsuIv5wJpH5JwQrzE7MTw7nCo%3D_=1417189173488
2014-11-28 10:39:32,653 DEBUG [o.a.c.s.m.AncientDataMotionStrategy] 
(Work-Job-Executor-44:ctx-02700c7f job-13494/job-13496 ctx-77b3b664) copyAsync 
inspecting src type TEMPLATE copyAsync inspecting dest type TEMPLATE
2014-11-28 10:39:32,664 DEBUG [c.c.a.t.Request] 
(Work-Job-Executor-44:ctx-02700c7f job-13494/job-13496 ctx-77b3b664) Seq 
132-2874703937145945308: Sending  { Cmd , MgmtId: 345049793560, via: 
132(xen01.z01p2.us.east.cvg.cs.zimcom.net), Ver: v1, Flags: 100011, 
[{org.apache.cloudstack.storage.command.CopyCommand:{srcTO:{org.apache.cloudstack.storage.to.TemplateObjectTO:{path:template/tmpl/2/343/9772bd4a-5a0c-4e41-a050-2daf612ad78c.vhd,uuid:468baa4a-3de3-47d2-877e-f97d40b2b272,id:343,format:VHD,accountId:2,hvm:true,displayText:Windows2012R2-Datacenter
 

Re: Xenserver Hosts Stuck in Maintenance

[Steve Searles]

Yea, tried all that.  Now its affecting KVM as well. Thanks for the reply, I 
will dig a bit deeper.


Steven Searles

[cid:image001.jpg@01CF54D3.D25E1ED0]

On Nov 20, 2014, at 4:00 PM, Motty Cruz 
motty.c...@gmail.commailto:motty.c...@gmail.com wrote:

Hi Steve,
have you try stopping and restarting ACS? also I would do the following in 
xenserver
xe-toolstack-restart
it won't affect your VMs.

To restart Cloudstack
service cloudstack-management restart (in CentOs)

Thanks,
Motty
On 11/20/2014 12:55 PM, Steve Searles wrote:
Found this in the catalina.out log on the management server.

INFO  [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-4:ctx-8faa1563 job-12695) 
Add job-12695 into job monitoring
WARN  [c.c.a.d.ParamGenericValidationWorker] (API-Job-Executor-4:ctx-8faa1563 
job-12695 ctx-81dfab11) Received unknown parameters for command 
cancelHostMaintenance. Unknown parameters : signatureversion expires
ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-4:ctx-8faa1563 job-12695) 
Unexpected exception while executing 
org.apache.cloudstack.api.command.admin.host.CancelMaintenanceCmd


Anyone know what the unknown parameters are all about?

—Steve



On Nov 20, 2014, at 3:14 PM, Steve Searles 
ssear...@zimcom.netmailto:ssear...@zimcom.netmailto:ssear...@zimcom.net 
wrote:

CS 4.4.1 - 4.4.2
I am having a problem with my xenserver hosts getting stuck in maintenance.  
Trying to cancel the maintenance produces the following NPE.

2014-11-20 15:04:28,575 INFO  [o.a.c.f.j.i.AsyncJobMonitor] 
(API-Job-Executor-14:ctx-4e8a63d4 job-12626) Add job-12626 into job monitoring
2014-11-20 15:04:28,576 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] 
(API-Job-Executor-14:ctx-4e8a63d4 job-12626) Executing AsyncJobVO {id:12626, 
userId: 2, accountId: 2, instanceType: Host, instanceId: 114, cmd: 
org.apache.cloudstack.api.command.admin.host.CancelMaintenanceCmd, cmdInfo: 
{id:189c3843-8d92-419b-a8b2-e343ea02c8fd,response:json,sessionkey:OEXANRcg2kzKJrfGXpvCK3E6k28\u003d,ctxDetails:{\com.cloud.host.Host\:\189c3843-8d92-419b-a8b2-e343ea02c8fd\},cmdEventType:MAINT.CANCEL,ctxUserId:2,httpmethod:GET,_:1416513869627,uuid:189c3843-8d92-419b-a8b2-e343ea02c8fd,ctxAccountId:2,ctxStartEventId:140662},
 cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: 
null, initMsid: 345049793560, completeMsid: null, lastUpdated: null, 
lastPolled: null, created: null}
2014-11-20 15:04:28,577 DEBUG [c.c.a.ApiServlet] (catalina-exec-12:ctx-78ec5c48 
ctx-e7500b2b) ===END===  172.23.0.1 -- GET  
command=cancelHostMaintenanceid=189c3843-8d92-419b-a8b2-e343ea02c8fdresponse=jsonsessionkey=OEXANRcg2kzKJrfGXpvCK3E6k28%3D_=1416513869627
2014-11-20 15:04:28,601 ERROR [c.c.a.ApiAsyncJobDispatcher] 
(API-Job-Executor-14:ctx-4e8a63d4 job-12626) Unexpected exception while 
executing org.apache.cloudstack.api.command.admin.host.CancelMaintenanceCmd
java.lang.NullPointerException
at 
com.cloud.resource.ResourceManagerImpl.doCancelMaintenance(ResourceManagerImpl.java:2083)
at 
com.cloud.resource.ResourceManagerImpl.cancelMaintenance(ResourceManagerImpl.java:2140)
at 
com.cloud.resource.ResourceManagerImpl.cancelMaintenance(ResourceManagerImpl.java:1127)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)




I have tried upgrading to the latest git build 4.4.2 and the problem still 
exists.   I think it started in 4.4.1 because it used to work properly in 
4.4.0.  I also deleted and re-created the SSVM but that did not help either.  
Does anyone have a solution or workaround?  Is there a way to manually take a 
host out of maintenance? I think there is more to it than setting the status in 
the DB?


— Steve

Re: ALL Hosts Stuck in Maintenance

[Steve Searles]

For some reason it is affecting every host.  VMware, KVM, and XenServer.  No 
hosts will come out of maintenance same NPE for all.  Storage will go in and 
out of maintenance fine.  Weird. Any ideas?  The only way to get the host back 
online is to remove it and re-add it.


Steven Searles

[cid:image001.jpg@01CF54D3.D25E1ED0]

On Nov 21, 2014, at 9:04 AM, Steve Searles 
ssear...@zimcom.netmailto:ssear...@zimcom.net wrote:

Yea, tried all that.  Now its affecting KVM as well. Thanks for the reply, I 
will dig a bit deeper.


Steven Searles

[cid:image001.jpg@01CF54D3.D25E1ED0]

On Nov 20, 2014, at 4:00 PM, Motty Cruz 
motty.c...@gmail.commailto:motty.c...@gmail.com wrote:

Hi Steve,
have you try stopping and restarting ACS? also I would do the following in 
xenserver
xe-toolstack-restart
it won't affect your VMs.

To restart Cloudstack
service cloudstack-management restart (in CentOs)

Thanks,
Motty
On 11/20/2014 12:55 PM, Steve Searles wrote:
Found this in the catalina.out log on the management server.

INFO  [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-4:ctx-8faa1563 job-12695) 
Add job-12695 into job monitoring
WARN  [c.c.a.d.ParamGenericValidationWorker] (API-Job-Executor-4:ctx-8faa1563 
job-12695 ctx-81dfab11) Received unknown parameters for command 
cancelHostMaintenance. Unknown parameters : signatureversion expires
ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-4:ctx-8faa1563 job-12695) 
Unexpected exception while executing 
org.apache.cloudstack.api.command.admin.host.CancelMaintenanceCmd


Anyone know what the unknown parameters are all about?

—Steve



On Nov 20, 2014, at 3:14 PM, Steve Searles 
ssear...@zimcom.netmailto:ssear...@zimcom.netmailto:ssear...@zimcom.net 
wrote:

CS 4.4.1 - 4.4.2
I am having a problem with my xenserver hosts getting stuck in maintenance.  
Trying to cancel the maintenance produces the following NPE.

2014-11-20 15:04:28,575 INFO  [o.a.c.f.j.i.AsyncJobMonitor] 
(API-Job-Executor-14:ctx-4e8a63d4 job-12626) Add job-12626 into job monitoring
2014-11-20 15:04:28,576 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] 
(API-Job-Executor-14:ctx-4e8a63d4 job-12626) Executing AsyncJobVO {id:12626, 
userId: 2, accountId: 2, instanceType: Host, instanceId: 114, cmd: 
org.apache.cloudstack.api.command.admin.host.CancelMaintenanceCmd, cmdInfo: 
{id:189c3843-8d92-419b-a8b2-e343ea02c8fd,response:json,sessionkey:OEXANRcg2kzKJrfGXpvCK3E6k28\u003d,ctxDetails:{\com.cloud.host.Host\:\189c3843-8d92-419b-a8b2-e343ea02c8fd\},cmdEventType:MAINT.CANCEL,ctxUserId:2,httpmethod:GET,_:1416513869627,uuid:189c3843-8d92-419b-a8b2-e343ea02c8fd,ctxAccountId:2,ctxStartEventId:140662},
 cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: 
null, initMsid: 345049793560, completeMsid: null, lastUpdated: null, 
lastPolled: null, created: null}
2014-11-20 15:04:28,577 DEBUG [c.c.a.ApiServlet] (catalina-exec-12:ctx-78ec5c48 
ctx-e7500b2b) ===END===  172.23.0.1 -- GET  
command=cancelHostMaintenanceid=189c3843-8d92-419b-a8b2-e343ea02c8fdresponse=jsonsessionkey=OEXANRcg2kzKJrfGXpvCK3E6k28%3D_=1416513869627
2014-11-20 15:04:28,601 ERROR [c.c.a.ApiAsyncJobDispatcher] 
(API-Job-Executor-14:ctx-4e8a63d4 job-12626) Unexpected exception while 
executing org.apache.cloudstack.api.command.admin.host.CancelMaintenanceCmd
java.lang.NullPointerException
at 
com.cloud.resource.ResourceManagerImpl.doCancelMaintenance(ResourceManagerImpl.java:2083)
at 
com.cloud.resource.ResourceManagerImpl.cancelMaintenance(ResourceManagerImpl.java:2140)
at 
com.cloud.resource.ResourceManagerImpl.cancelMaintenance(ResourceManagerImpl.java:1127)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)




I have tried upgrading to the latest git build 4.4.2 and the problem still 
exists.   I think it started in 4.4.1 because it used to work properly in 
4.4.0.  I also deleted and re-created the SSVM but that did not help either.  
Does anyone have a solution or workaround?  Is there a way to manually take a 
host out of maintenance? I think there is more to it than setting the status in 
the DB?


— Steve

Re: ALL Hosts Stuck in Maintenance

[Steve Searles]

In my case the agent is connected fine.  I have tried even restarting the 
hosts.  I have the agent log in debug for the KVM server and am not seeing the 
mgmt server even trying to execute anything when coming out of maintenance.  I 
just get the NPE immediately in the management server logs.

Steven Searles, CTO | ssear...@zimcom.netmailto:ssear...@zimcom.net
Zimcom Internet Solutions  | www.zimcom.nethttp://www.zimcom.net
O: 513.231.9500  |  D: 513.233.4130

[cid:image001.jpg@01CF54D3.D25E1ED0]

On Nov 21, 2014, at 10:54 AM, Andrija Panic 
andrija.pa...@gmail.commailto:andrija.pa...@gmail.com wrote:

Experienced similar behaviour, for kvm - seems like restarting libvirtd and
give it some time to settle, and than agent connects ... on its own...

Sent from Google Nexus 4
On Nov 21, 2014 4:43 PM, Steve Searles 
ssear...@zimcom.netmailto:ssear...@zimcom.net wrote:

For some reason it is affecting every host.  VMware, KVM, and XenServer.
No hosts will come out of maintenance same NPE for all.  Storage will go in
and out of maintenance fine.  Weird. Any ideas?  The only way to get the
host back online is to remove it and re-add it.


 Steven Searles


On Nov 21, 2014, at 9:04 AM, Steve Searles 
ssear...@zimcom.netmailto:ssear...@zimcom.net wrote:

Yea, tried all that.  Now its affecting KVM as well. Thanks for the
reply, I will dig a bit deeper.


 Steven Searles


On Nov 20, 2014, at 4:00 PM, Motty Cruz 
motty.c...@gmail.commailto:motty.c...@gmail.com wrote:

Hi Steve,
have you try stopping and restarting ACS? also I would do the following in
xenserver
xe-toolstack-restart
it won't affect your VMs.

To restart Cloudstack
service cloudstack-management restart (in CentOs)

Thanks,
Motty
On 11/20/2014 12:55 PM, Steve Searles wrote:

Found this in the catalina.out log on the management server.

INFO  [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-4:ctx-8faa1563
job-12695) Add job-12695 into job monitoring
WARN  [c.c.a.d.ParamGenericValidationWorker]
(API-Job-Executor-4:ctx-8faa1563 job-12695 ctx-81dfab11) Received unknown
parameters for command cancelHostMaintenance. Unknown parameters :
signatureversion expires
ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-4:ctx-8faa1563
job-12695) Unexpected exception while executing
org.apache.cloudstack.api.command.admin.host.CancelMaintenanceCmd


Anyone know what the unknown parameters are all about?

—Steve



On Nov 20, 2014, at 3:14 PM, Steve Searles 
ssear...@zimcom.netmailto:ssear...@zimcom.net
mailto:ssear...@zimcom.net ssear...@zimcom.netmailto:ssear...@zimcom.net 
wrote:

CS 4.4.1 - 4.4.2
I am having a problem with my xenserver hosts getting stuck in
maintenance.  Trying to cancel the maintenance produces the following NPE.

2014-11-20 15:04:28,575 INFO  [o.a.c.f.j.i.AsyncJobMonitor]
(API-Job-Executor-14:ctx-4e8a63d4 job-12626) Add job-12626 into job
monitoring
2014-11-20 15:04:28,576 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-14:ctx-4e8a63d4 job-12626) Executing AsyncJobVO
{id:12626, userId: 2, accountId: 2, instanceType: Host, instanceId: 114,
cmd: org.apache.cloudstack.api.command.admin.host.CancelMaintenanceCmd,
cmdInfo:
{id:189c3843-8d92-419b-a8b2-e343ea02c8fd,response:json,sessionkey:OEXANRcg2kzKJrfGXpvCK3E6k28\u003d,ctxDetails:{\com.cloud.host.Host\:\189c3843-8d92-419b-a8b2-e343ea02c8fd\},cmdEventType:MAINT.CANCEL,ctxUserId:2,httpmethod:GET,_:1416513869627,uuid:189c3843-8d92-419b-a8b2-e343ea02c8fd,ctxAccountId:2,ctxStartEventId:140662},
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
result: null, initMsid: 345049793560, completeMsid: null, lastUpdated:
null, lastPolled: null, created: null}
2014-11-20 15:04:28,577 DEBUG [c.c.a.ApiServlet]
(catalina-exec-12:ctx-78ec5c48 ctx-e7500b2b) ===END===  172.23.0.1 -- GET
command=cancelHostMaintenanceid=189c3843-8d92-419b-a8b2-e343ea02c8fdresponse=jsonsessionkey=OEXANRcg2kzKJrfGXpvCK3E6k28%3D_=1416513869627
2014-11-20 15:04:28,601 ERROR [c.c.a.ApiAsyncJobDispatcher]
(API-Job-Executor-14:ctx-4e8a63d4 job-12626) Unexpected exception while
executing org.apache.cloudstack.api.command.admin.host.CancelMaintenanceCmd
java.lang.NullPointerException
at
com.cloud.resource.ResourceManagerImpl.doCancelMaintenance(ResourceManagerImpl.java:2083)
at
com.cloud.resource.ResourceManagerImpl.cancelMaintenance(ResourceManagerImpl.java:2140)
at
com.cloud.resource.ResourceManagerImpl.cancelMaintenance(ResourceManagerImpl.java:1127)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)




I have tried upgrading to the latest git build 4.4.2 and the problem still
exists.   I think it started in 4.4.1 because it used to work properly in
4.4.0.  I also deleted and re-created the SSVM but that did not help
either.  Does anyone have a solution or workaround?  Is there a way to
manually take a host out of maintenance? I think there is more to it than
setting the status in the DB?


— Steve

Re: ALL Hosts Stuck in Maintenance

[Steve Searles]

Below is the full NPE from the catalina.out if that would help.  This is the 
same NPE for all hypervisor types.

INFO  [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-10:ctx-208629c7 
job-12679) Add job-12679 into job monitoring
ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-10:ctx-208629c7 
job-12679) Unexpected exception while executing 
org.apache.cloudstack.api.command.admin.host.CancelMaintenanceCmd
java.lang.NullPointerException
at 
com.cloud.resource.ResourceManagerImpl.doCancelMaintenance(ResourceManagerImpl.java:2083)
at 
com.cloud.resource.ResourceManagerImpl.cancelMaintenance(ResourceManagerImpl.java:2140)
at 
com.cloud.resource.ResourceManagerImpl.cancelMaintenance(ResourceManagerImpl.java:1127)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at 
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at 
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy148.cancelMaintenance(Unknown Source)
at 
org.apache.cloudstack.api.command.admin.host.CancelMaintenanceCmd.execute(CancelMaintenanceCmd.java:102)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
at 
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
at 
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at 
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at 
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
INFO  [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-10:ctx-208629c7 
job-12679) Remove job-12679 from job monitoring


Steven Searles, CTO | ssear...@zimcom.netmailto:ssear...@zimcom.net
Zimcom Internet Solutions  | www.zimcom.nethttp://www.zimcom.net
O: 513.231.9500  |  D: 513.233.4130

[cid:image001.jpg@01CF54D3.D25E1ED0]

On Nov 21, 2014, at 12:31 PM, Steve Searles 
ssear...@zimcom.netmailto:ssear...@zimcom.net wrote:

In my case the agent is connected fine.  I have tried even restarting the 
hosts.  I have the agent log in debug for the KVM server and am not seeing the 
mgmt server even trying to execute anything when coming out of maintenance.  I 
just get the NPE immediately in the management server logs.

Steven Searles, CTO | ssear...@zimcom.netmailto:ssear...@zimcom.net
Zimcom Internet Solutions  | www.zimcom.nethttp://www.zimcom.net/
O: 513.231.9500  |  D: 513.233.4130

image001.jpg

On Nov 21, 2014, at 10:54 AM, Andrija Panic 
andrija.pa...@gmail.commailto:andrija.pa...@gmail.com wrote:

Experienced similar behaviour, for kvm - seems like restarting libvirtd and
give it some time to settle, and than agent connects ... on its own...

Sent from Google Nexus 4
On Nov 21, 2014 4:43 PM, Steve Searles 
ssear...@zimcom.netmailto:ssear...@zimcom.net wrote:

For some reason it is affecting every host.  VMware, KVM, and XenServer.
No hosts will come out of maintenance same NPE for all.  Storage will go in
and out of maintenance fine.  Weird. Any ideas?  The only way to get the
host back online is to remove it and re-add it.


 Steven Searles


On Nov 21, 2014, at 9:04 AM, Steve Searles 
ssear...@zimcom.netmailto:ssear...@zimcom.net wrote:

Yea, tried all that.  Now its affecting KVM as well. Thanks for the
reply, I will dig a bit deeper.


 Steven Searles


On Nov 20, 2014, at 4:00 PM, Motty Cruz 
motty.c...@gmail.commailto:motty.c...@gmail.com wrote:

Hi Steve,
have you try stopping and restarting

Xenserver Hosts Stuck in Maintenance

[Steve Searles]

CS 4.4.1 - 4.4.2
I am having a problem with my xenserver hosts getting stuck in maintenance.  
Trying to cancel the maintenance produces the following NPE.

2014-11-20 15:04:28,575 INFO  [o.a.c.f.j.i.AsyncJobMonitor] 
(API-Job-Executor-14:ctx-4e8a63d4 job-12626) Add job-12626 into job monitoring
2014-11-20 15:04:28,576 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] 
(API-Job-Executor-14:ctx-4e8a63d4 job-12626) Executing AsyncJobVO {id:12626, 
userId: 2, accountId: 2, instanceType: Host, instanceId: 114, cmd: 
org.apache.cloudstack.api.command.admin.host.CancelMaintenanceCmd, cmdInfo: 
{id:189c3843-8d92-419b-a8b2-e343ea02c8fd,response:json,sessionkey:OEXANRcg2kzKJrfGXpvCK3E6k28\u003d,ctxDetails:{\com.cloud.host.Host\:\189c3843-8d92-419b-a8b2-e343ea02c8fd\},cmdEventType:MAINT.CANCEL,ctxUserId:2,httpmethod:GET,_:1416513869627,uuid:189c3843-8d92-419b-a8b2-e343ea02c8fd,ctxAccountId:2,ctxStartEventId:140662},
 cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: 
null, initMsid: 345049793560, completeMsid: null, lastUpdated: null, 
lastPolled: null, created: null}
2014-11-20 15:04:28,577 DEBUG [c.c.a.ApiServlet] (catalina-exec-12:ctx-78ec5c48 
ctx-e7500b2b) ===END===  172.23.0.1 -- GET  
command=cancelHostMaintenanceid=189c3843-8d92-419b-a8b2-e343ea02c8fdresponse=jsonsessionkey=OEXANRcg2kzKJrfGXpvCK3E6k28%3D_=1416513869627
2014-11-20 15:04:28,601 ERROR [c.c.a.ApiAsyncJobDispatcher] 
(API-Job-Executor-14:ctx-4e8a63d4 job-12626) Unexpected exception while 
executing org.apache.cloudstack.api.command.admin.host.CancelMaintenanceCmd
java.lang.NullPointerException
at 
com.cloud.resource.ResourceManagerImpl.doCancelMaintenance(ResourceManagerImpl.java:2083)
at 
com.cloud.resource.ResourceManagerImpl.cancelMaintenance(ResourceManagerImpl.java:2140)
at 
com.cloud.resource.ResourceManagerImpl.cancelMaintenance(ResourceManagerImpl.java:1127)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)




I have tried upgrading to the latest git build 4.4.2 and the problem still 
exists.   I think it started in 4.4.1 because it used to work properly in 
4.4.0.  I also deleted and re-created the SSVM but that did not help either.  
Does anyone have a solution or workaround?  Is there a way to manually take a 
host out of maintenance? I think there is more to it than setting the status in 
the DB?


— Steve

Re: Xenserver Hosts Stuck in Maintenance

[Steve Searles]

Found this in the catalina.out log on the management server.

INFO  [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-4:ctx-8faa1563 job-12695) 
Add job-12695 into job monitoring
WARN  [c.c.a.d.ParamGenericValidationWorker] (API-Job-Executor-4:ctx-8faa1563 
job-12695 ctx-81dfab11) Received unknown parameters for command 
cancelHostMaintenance. Unknown parameters : signatureversion expires
ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-4:ctx-8faa1563 job-12695) 
Unexpected exception while executing 
org.apache.cloudstack.api.command.admin.host.CancelMaintenanceCmd


Anyone know what the unknown parameters are all about?

—Steve



On Nov 20, 2014, at 3:14 PM, Steve Searles 
ssear...@zimcom.netmailto:ssear...@zimcom.net wrote:

CS 4.4.1 - 4.4.2
I am having a problem with my xenserver hosts getting stuck in maintenance.  
Trying to cancel the maintenance produces the following NPE.

2014-11-20 15:04:28,575 INFO  [o.a.c.f.j.i.AsyncJobMonitor] 
(API-Job-Executor-14:ctx-4e8a63d4 job-12626) Add job-12626 into job monitoring
2014-11-20 15:04:28,576 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] 
(API-Job-Executor-14:ctx-4e8a63d4 job-12626) Executing AsyncJobVO {id:12626, 
userId: 2, accountId: 2, instanceType: Host, instanceId: 114, cmd: 
org.apache.cloudstack.api.command.admin.host.CancelMaintenanceCmd, cmdInfo: 
{id:189c3843-8d92-419b-a8b2-e343ea02c8fd,response:json,sessionkey:OEXANRcg2kzKJrfGXpvCK3E6k28\u003d,ctxDetails:{\com.cloud.host.Host\:\189c3843-8d92-419b-a8b2-e343ea02c8fd\},cmdEventType:MAINT.CANCEL,ctxUserId:2,httpmethod:GET,_:1416513869627,uuid:189c3843-8d92-419b-a8b2-e343ea02c8fd,ctxAccountId:2,ctxStartEventId:140662},
 cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: 
null, initMsid: 345049793560, completeMsid: null, lastUpdated: null, 
lastPolled: null, created: null}
2014-11-20 15:04:28,577 DEBUG [c.c.a.ApiServlet] (catalina-exec-12:ctx-78ec5c48 
ctx-e7500b2b) ===END===  172.23.0.1 -- GET  
command=cancelHostMaintenanceid=189c3843-8d92-419b-a8b2-e343ea02c8fdresponse=jsonsessionkey=OEXANRcg2kzKJrfGXpvCK3E6k28%3D_=1416513869627
2014-11-20 15:04:28,601 ERROR [c.c.a.ApiAsyncJobDispatcher] 
(API-Job-Executor-14:ctx-4e8a63d4 job-12626) Unexpected exception while 
executing org.apache.cloudstack.api.command.admin.host.CancelMaintenanceCmd
java.lang.NullPointerException
at 
com.cloud.resource.ResourceManagerImpl.doCancelMaintenance(ResourceManagerImpl.java:2083)
at 
com.cloud.resource.ResourceManagerImpl.cancelMaintenance(ResourceManagerImpl.java:2140)
at 
com.cloud.resource.ResourceManagerImpl.cancelMaintenance(ResourceManagerImpl.java:1127)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)




I have tried upgrading to the latest git build 4.4.2 and the problem still 
exists.   I think it started in 4.4.1 because it used to work properly in 
4.4.0.  I also deleted and re-created the SSVM but that did not help either.  
Does anyone have a solution or workaround?  Is there a way to manually take a 
host out of maintenance? I think there is more to it than setting the status in 
the DB?


— Steve

Cloudstack 4.4.1 VPN Routing issues

[Steve Searles]

Hello everyone, I am having issues with the remote network route not being 
created on the client side when connected to the VPN (Virtual Router) on a 
guest network (not site to site).  If I add the route manually on the client it 
works fine. It is happening on both windows7 and OSX.  I saw a few older posts 
about VPN issues specifically with the OSX client but they seem to be resolved 
at this point.  I have the latest 4.4.1 from git and I am using the new 4.4.1 
systemvm’s.  Did I miss a setting on the CS side somewhere? Any ideas? 

Steve Searles 

VM Instance running pptp server inside virtual router.

[Steve Searles]

Anyone know the secret to making pptp server work through a virtual router.  
The IP PROT 47 (GRE) seems to not be handled correctly.  I have not dug into 
the VR any further to troubleshoot, thought I would ask here and see if there 
was a quick answer.  I have not tried a static nat inside a vpc yet but I 
assume that may work.  Running CS 4.3.0/Advanced Networking. (Egress rule 
0.0.0.0/0 any) ingress firewall open for ports 1-65535 and port forwarding for 
1-65535 for testing.I know we can just use the l2tp vpn on the VR and 
essentially accomplish the same thing, however this particular application 
relies on the Microsoft RAS services and therefore needs to be terminated on 
the VM instance itself.

Thanks.

Re: Usage Server Issues

[Steve Searles]

It looks like I am getting closer with the help of some of you privately.  Here 
is where we are so far.  Using the API calling the generateUsageRecords did 
nothing but create another usage job.  But calling listUsagerecords produced an 
error, when tracked down we can see that it is caused by a missing DB column.  
See below.

2014-06-13 00:12:37,597 - requester.py:41 - [DEBUG]  START
Request 
2014-06-13 00:12:37,597 - requester.py:41 - [DEBUG] Requesting
command=listUsageRecords, args={'startdate': '2014-06-01', 'enddate':
'2014-06-13'}
2014-06-13 00:12:37,598 - requester.py:41 - [DEBUG] Request sent:
http://localhost:8080/client/api?apikey=GqlaXiDzG1TWAg8smpO21gzzo_6Uxq
oQnk8wdpzCCOyEQ8oGcal9elX-y4R8Tm01sLTuuE40zMCZ7_yeqqZRcAcommand=listU
sageRecordsenddate=2014-06-13expires=2014-06-13T04%3A22%3A37%2B
response=jsonsignatureversion=3startdate=2014-06-01signature=0kk0CP
XBv%2BZrbg3J7oosTk7mYIM%3D
2014-06-13 00:12:37,616 - requester.py:41 - [DEBUG] Response received:
None
2014-06-13 00:12:37,616 - requester.py:41 - [DEBUG] Error: : DB
Exception on: com.mysql.jdbc.JDBC4PreparedStatement@d02448f: SELECT
cloud_usage.id, cloud_usage.zone_id, cloud_usage.account_id,
cloud_usage.domain_id, cloud_usage.description,
cloud_usage.usage_display, cloud_usage.usage_type,
cloud_usage.raw_usage, cloud_usage.vm_instance_id,
cloud_usage.vm_name, cloud_usage.offering_id, cloud_usage.template_id,
cloud_usage.usage_id, cloud_usage.type, cloud_usage.size,
cloud_usage.virtual_size, cloud_usage.network_id,
cloud_usage.start_date, cloud_usage.end_date FROM cloud_usage WHERE
cloud_usage.start_date BETWEEN '2014-06-01 04:00:00' AND '2014-06-14
03:59:59'  AND cloud_usage.end_date BETWEEN '2014-06-01 04:00:00' AND
'2014-06-14 03:59:59'  ORDER BY cloud_usage.start_date DESC  LIMIT 0,
500
2014-06-13 00:12:37,616 - requester.py:41 - [DEBUG]  END
Request 

Now we are getting somewhere.  When running the query in mysql we can see that 
a column is missing.


mysql SELECT cloud_usage.id, cloud_usage.zone_id,
mysql cloud_usage.account_id, cloud_usage.domain_id,
mysql cloud_usage.description, cloud_usage.usage_display,
mysql cloud_usage.usage_type, cloud_usage.raw_usage,
mysql cloud_usage.vm_instance_id, cloud_usage.vm_name,
mysql cloud_usage.offering_id, cloud_usage.template_id,
mysql cloud_usage.usage_id, cloud_usage.type, cloud_usage.size,
mysql cloud_usage.virtual_size, cloud_usage.network_id,
mysql cloud_usage.start_date, cloud_usage.end_date FROM cloud_usage
mysql WHERE cloud_usage.start_date BETWEEN '2014-06-01 04:00:00' AND
mysql '2014-06-14 03:59:59'  AND cloud_usage.end_date BETWEEN
mysql '2014-06-01 04:00:00' AND '2014-06-14 03:59:59'  ORDER BY
mysql cloud_usage.start_date DESC  LIMIT 0, 500;
ERROR 1054 (42S22): Unknown column 'cloud_usage.virtual_size' in 'field list'
mysql

I don't see the virtual_size table being created in the cloud_usage schema file.

So it looks like virtual_size and size are missing from my DB.  It is my 
understanding that the DB is created with create-database-premium.sql  
create-schema-premium.sql.  The create database is fine, but the schema is 
missing virtual_size and size as well.

CREATE TABLE  `cloud_usage`.`cloud_usage` (
  `id` bigint unsigned NOT NULL auto_increment,
  `zone_id` bigint unsigned NOT NULL,
  `account_id` bigint unsigned NOT NULL,
  `domain_id` bigint unsigned NOT NULL,
  `description` varchar(1024) NOT NULL,
  `usage_display` varchar(255) NOT NULL,
  `usage_type` int(1) unsigned,
  `raw_usage` DOUBLE UNSIGNED NOT NULL,
  `vm_instance_id` bigint unsigned,
  `vm_name` varchar(255),
  `offering_id` bigint unsigned,
  `template_id` bigint unsigned,
  `usage_id` bigint unsigned,
  `type` varchar(32),
  `size` bigint unsigned,
  `network_id` bigint unsigned,
  `start_date` DATETIME NOT NULL,
  `end_date` DATETIME NOT NULL,
  PRIMARY KEY  (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;


Does anyone have an updated copy of the usageDB schema?  I am not sure how 
anyone’s usage servers are working since this is a fresh install of 4.3 on my 
end and they should be missing the tables as well.   The usage job will 
populate account and user_statistics because they seem not to be dependent on 
the cloud_usage table.

Any help would be appreciated.

Steve Searles



On Jun 13, 2014, at 1:46 AM, Shweta Agarwal 
shweta.agar...@citrix.commailto:shweta.agar...@citrix.com wrote:

You can use generateUsageRecords api  . It Generates usage records. This will 
generate records only if there any records to be generated, i.e if the 
scheduled usage job was not run or failed
Request parameters
Parameter Name Description Required
enddate End date range for usage record query. Use -MM-dd as the date 
format, e.g. startDate=2009-06-03. true
startdate Start date range for usage record query. Use -MM-dd as the date 
format, e.g. startDate=2009-06-01. true
domainid List events for the specified domain. false
Response Tags
Response Name Description
displaytext any

Re: Usage Server Issues

[Steve Searles]

After adding the virtual_size column to the cloud_usage table the 
listusagerecords is no longer producing an error, however in the usage.log it 
appears there is still a missing table, I will create this by hand and report 
back.

2014-06-13 14:38:55,401 ERROR [cloud.usage.UsageManagerImpl] (Usage-Job-1:null) 
Usage Manager error
com.cloud.utils.exception.CloudRuntimeException: Table 
'cloud_usage.vm_disk_statistics' doesn't exist
at com.cloud.usage.dao.UsageDaoImpl.saveVmDiskStats(UsageDaoImpl.java:377)
at 
com.cloud.usage.dao.UsageDaoImpl_EnhancerByCloudStack_30c322bf.CGLIB$saveVmDiskStats$11(generated)
at 
com.cloud.usage.dao.UsageDaoImpl_EnhancerByCloudStack_30c322bf_FastClassByCloudStack_bdbde463.invoke(generated)
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at 
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at 
com.cloud.usage.dao.UsageDaoImpl_EnhancerByCloudStack_30c322bf.saveVmDiskStats(generated)
at com.cloud.usage.UsageManagerImpl.parse(UsageManagerImpl.java:543)
at 
com.cloud.usage.UsageManagerImpl.runInContextInternal(UsageManagerImpl.java:349)
at com.cloud.usage.UsageManagerImpl$1.runInContext(UsageManagerImpl.java:291)
at 
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at 
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at com.cloud.usage.UsageManagerImpl.run(UsageManagerImpl.java:288)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)
2014-06-13 14:38:55,402 INFO  [cloud.usage.UsageManagerImpl] (Usage-Job-1:null) 
usage job complete


Steven Searles, CTO | ssear...@zimcom.netmailto:ssear...@zimcom.net
Zimcom Internet Solutions  | www.zimcom.nethttp://www.zimcom.net
O: 513.231.9500  |  D: 513.233.4130

[cid:image001.jpg@01CF54D3.D25E1ED0]

On Jun 13, 2014, at 10:25 AM, Steve Searles 
ssear...@zimcom.netmailto:ssear...@zimcom.net wrote:

It looks like I am getting closer with the help of some of you privately.  Here 
is where we are so far.  Using the API calling the generateUsageRecords did 
nothing but create another usage job.  But calling listUsagerecords produced an 
error, when tracked down we can see that it is caused by a missing DB column.  
See below.

2014-06-13 00:12:37,597 - requester.py:41 - [DEBUG]  START
Request 
2014-06-13 00:12:37,597 - requester.py:41 - [DEBUG] Requesting
command=listUsageRecords, args={'startdate': '2014-06-01', 'enddate':
'2014-06-13'}
2014-06-13 00:12:37,598 - requester.py:41 - [DEBUG] Request sent:
http://localhost:8080/client/api?apikey=GqlaXiDzG1TWAg8smpO21gzzo_6Uxq
oQnk8wdpzCCOyEQ8oGcal9elX-y4R8Tm01sLTuuE40zMCZ7_yeqqZRcAcommand=listU
sageRecordsenddate=2014-06-13expires=2014-06-13T04%3A22%3A37%2B
response=jsonsignatureversion=3startdate=2014-06-01signature=0kk0CP
XBv%2BZrbg3J7oosTk7mYIM%3D
2014-06-13 00:12:37,616 - requester.py:41 - [DEBUG] Response received:
None
2014-06-13 00:12:37,616 - requester.py:41 - [DEBUG] Error: : DB
Exception on: com.mysql.jdbc.JDBC4PreparedStatement@d02448f: SELECT
cloud_usage.id, cloud_usage.zone_id, cloud_usage.account_id,
cloud_usage.domain_id, cloud_usage.description,
cloud_usage.usage_display, cloud_usage.usage_type,
cloud_usage.raw_usage, cloud_usage.vm_instance_id,
cloud_usage.vm_name, cloud_usage.offering_id, cloud_usage.template_id,
cloud_usage.usage_id, cloud_usage.type, cloud_usage.size,
cloud_usage.virtual_size, cloud_usage.network_id,
cloud_usage.start_date, cloud_usage.end_date FROM cloud_usage WHERE
cloud_usage.start_date BETWEEN '2014-06-01 04:00:00' AND '2014-06-14
03:59:59'  AND cloud_usage.end_date BETWEEN '2014-06-01 04:00:00' AND
'2014-06-14 03:59:59'  ORDER BY cloud_usage.start_date DESC  LIMIT 0,
500
2014-06-13 00:12:37,616 - requester.py:41 - [DEBUG]  END
Request 

Now we are getting somewhere.  When running the query in mysql we can see that 
a column is missing.


mysql SELECT cloud_usage.id, cloud_usage.zone_id,
mysql

Re: Usage Server Issues

[Steve Searles]

I have resolved this my finding the proper schema for the cloud_usage DB and 
creating a blank db.  Thank you everyone.

Steven Searles, CTO | ssear...@zimcom.netmailto:ssear...@zimcom.net
Zimcom Internet Solutions  | www.zimcom.nethttp://www.zimcom.net
O: 513.231.9500  |  D: 513.233.4130

[cid:image001.jpg@01CF54D3.D25E1ED0]

On Jun 13, 2014, at 10:48 AM, Steve Searles 
ssear...@zimcom.netmailto:ssear...@zimcom.net wrote:

After adding the virtual_size column to the cloud_usage table the 
listusagerecords is no longer producing an error, however in the usage.log it 
appears there is still a missing table, I will create this by hand and report 
back.

2014-06-13 14:38:55,401 ERROR [cloud.usage.UsageManagerImpl] (Usage-Job-1:null) 
Usage Manager error
com.cloud.utils.exception.CloudRuntimeException: Table 
'cloud_usage.vm_disk_statistics' doesn't exist
at com.cloud.usage.dao.UsageDaoImpl.saveVmDiskStats(UsageDaoImpl.java:377)
at 
com.cloud.usage.dao.UsageDaoImpl_EnhancerByCloudStack_30c322bf.CGLIB$saveVmDiskStats$11(generated)
at 
com.cloud.usage.dao.UsageDaoImpl_EnhancerByCloudStack_30c322bf_FastClassByCloudStack_bdbde463.invoke(generated)
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at 
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at 
com.cloud.usage.dao.UsageDaoImpl_EnhancerByCloudStack_30c322bf.saveVmDiskStats(generated)
at com.cloud.usage.UsageManagerImpl.parse(UsageManagerImpl.java:543)
at 
com.cloud.usage.UsageManagerImpl.runInContextInternal(UsageManagerImpl.java:349)
at com.cloud.usage.UsageManagerImpl$1.runInContext(UsageManagerImpl.java:291)
at 
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at 
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at com.cloud.usage.UsageManagerImpl.run(UsageManagerImpl.java:288)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)
2014-06-13 14:38:55,402 INFO  [cloud.usage.UsageManagerImpl] (Usage-Job-1:null) 
usage job complete


Steven Searles, CTO | ssear...@zimcom.netmailto:ssear...@zimcom.net
Zimcom Internet Solutions  | www.zimcom.nethttp://www.zimcom.net/
O: 513.231.9500  |  D: 513.233.4130

[cid:image001.jpg@01CF54D3.D25E1ED0]

On Jun 13, 2014, at 10:25 AM, Steve Searles 
ssear...@zimcom.netmailto:ssear...@zimcom.net wrote:

It looks like I am getting closer with the help of some of you privately.  Here 
is where we are so far.  Using the API calling the generateUsageRecords did 
nothing but create another usage job.  But calling listUsagerecords produced an 
error, when tracked down we can see that it is caused by a missing DB column.  
See below.

2014-06-13 00:12:37,597 - requester.py:41 - [DEBUG]  START
Request 
2014-06-13 00:12:37,597 - requester.py:41 - [DEBUG] Requesting
command=listUsageRecords, args={'startdate': '2014-06-01', 'enddate':
'2014-06-13'}
2014-06-13 00:12:37,598 - requester.py:41 - [DEBUG] Request sent:
http://localhost:8080/client/api?apikey=GqlaXiDzG1TWAg8smpO21gzzo_6Uxq
oQnk8wdpzCCOyEQ8oGcal9elX-y4R8Tm01sLTuuE40zMCZ7_yeqqZRcAcommand=listU
sageRecordsenddate=2014-06-13expires=2014-06-13T04%3A22%3A37%2B
response=jsonsignatureversion=3startdate=2014-06-01signature=0kk0CP
XBv%2BZrbg3J7oosTk7mYIM%3D
2014-06-13 00:12:37,616 - requester.py:41 - [DEBUG] Response received:
None
2014-06-13 00:12:37,616 - requester.py:41 - [DEBUG] Error: : DB
Exception on: com.mysql.jdbc.JDBC4PreparedStatement@d02448f: SELECT
cloud_usage.id, cloud_usage.zone_id, cloud_usage.account_id,
cloud_usage.domain_id, cloud_usage.description,
cloud_usage.usage_display, cloud_usage.usage_type,
cloud_usage.raw_usage, cloud_usage.vm_instance_id,
cloud_usage.vm_name, cloud_usage.offering_id, cloud_usage.template_id,
cloud_usage.usage_id, cloud_usage.type, cloud_usage.size,
cloud_usage.virtual_size, cloud_usage.network_id,
cloud_usage.start_date, cloud_usage.end_date FROM cloud_usage WHERE
cloud_usage.start_date BETWEEN

RE: Vmware Full Clones vs Linked Clones

[Steve Searles]

Very infomitive explination, thanks for taking the time.  We use VMAX and VNX 
storage with FAST-CACHE/FASTVP, I would almost rather burn the storage on the 
vmware VM's for full clones after reading your explination and let the lighter 
loaded vm's use KVM or XEN which seem to behave in a similar manner.  I would 
hate to loose a whole set of enterprise servers over a single mishap with the 
snapshot chain.  There is also the issue of resizing the ROOT disk which does 
not seem to be possible with linked clones, (understandably so).  It will be 
nice when root disk resizing is implemented in CS rather than changing the disk 
in vmware and manually updating the DB. 

Thanks Again, 

Steve Searles

-Original Message-
From: ilya musayev [mailto:ilya.mailing.li...@gmail.com] 
Sent: Thursday, June 12, 2014 2:18 AM
To: users@cloudstack.apache.org
Subject: Re: Vmware Full Clones vs Linked Clones

This topic comes up many times, it all depends what backend storage, number of 
spindles, workload type and SLAs during issues.

Linked Clones

Pros:

VM comes up online within 1 minute or less (2 minutes if you are running slower 
backend storage) You are saving on diskspace, if the rate of change on ROOT is 
small and data does NOT reside on ROOT volume If you storage uses FAST 
technology and moves frequently accessed data blocks to something like SSD, 
initial boot up time improves greatly

Cons:
You are leveraging VmWare Snapshot technology and changes are written in the 
form of deltas, which means if you create a 5GB file and delete it, your vmdk 
delta file will still be 5GB in size and will most likely only grow, Corruption 
to a parent vmdk on the primary datastore will impact other VMs that are 
dependent on it - hence reliable storage is needed.
Perfomance may degrade overtime if rate of change is high - also pending your 
storage backend Snapshotting (using vmware snapshot feature), will work, but if 
you have a complex snapshot three and some delta vmdk under this tree get 
corrupted, you may loose your data (until good working state) - this issue 
applies to snapshots in general


Full Clones:

Gets your independent disks with no delta complexity, at the expense of extra 
storage and some IO if you dont have FAST technology enabled.
Corruption to a vmdk file, affects only 1 VM.
If the VM has heavy read and write IO, you should consider running it as full 
clone as you will avoid delta complexity.

There are probably more reasons, just cant think of them now,

Regards
ilya



On 6/11/14, 7:42 AM, Steve Searles wrote:
 Can someone speak about using Linked Clones vs Full Clones in a production CS 
 environment?  What is the performance impact on the parent virtual machine? 
 What type of density can be expected if all the child vm's are performing 
 read operations from the same snapshot of the parent VM? What are the dangers 
 of using linked clones in this manner? What are the best practices from the 
 CS community?


 Steve Searles

Usage Server Issues

[Steve Searles]

Any experts on the usage server out there?  I have a weird problem where my 
accounts table and user_statistics table are populated but the rest of the 
tables in the DB are empty (with the exception of usage_job).  Does anyone know 
how to manually fire off the usage job from the command line rather than 
waiting for it to happen? Any pointers on how to debug this?  I have about 40 
vm's running in my lab that should be making plenty of data.

CS 4.3

Steve Searles

RE: Vmware Full Clones vs Linked Clones

[Steve Searles]

It would also be nice to define this on the cluster or at least the zone level 
rather than being an all or nothing global setting. 


Steven Searles, CTO |  ssear...@zimcom.net
Zimcom Internet Solutions  | www.zimcom.net
O: 513.231.9500  |  D: 513.233.4130



-Original Message-
From: Steve Searles [mailto:ssear...@zimcom.net] 
Sent: Thursday, June 12, 2014 11:07 PM
To: users@cloudstack.apache.org
Subject: RE: Vmware Full Clones vs Linked Clones

Very infomitive explination, thanks for taking the time.  We use VMAX and VNX 
storage with FAST-CACHE/FASTVP, I would almost rather burn the storage on the 
vmware VM's for full clones after reading your explination and let the lighter 
loaded vm's use KVM or XEN which seem to behave in a similar manner.  I would 
hate to loose a whole set of enterprise servers over a single mishap with the 
snapshot chain.  There is also the issue of resizing the ROOT disk which does 
not seem to be possible with linked clones, (understandably so).  It will be 
nice when root disk resizing is implemented in CS rather than changing the disk 
in vmware and manually updating the DB. 

Thanks Again, 

Steve Searles

-Original Message-
From: ilya musayev [mailto:ilya.mailing.li...@gmail.com] 
Sent: Thursday, June 12, 2014 2:18 AM
To: users@cloudstack.apache.org
Subject: Re: Vmware Full Clones vs Linked Clones

This topic comes up many times, it all depends what backend storage, number of 
spindles, workload type and SLAs during issues.

Linked Clones

Pros:

VM comes up online within 1 minute or less (2 minutes if you are running slower 
backend storage) You are saving on diskspace, if the rate of change on ROOT is 
small and data does NOT reside on ROOT volume If you storage uses FAST 
technology and moves frequently accessed data blocks to something like SSD, 
initial boot up time improves greatly

Cons:
You are leveraging VmWare Snapshot technology and changes are written in the 
form of deltas, which means if you create a 5GB file and delete it, your vmdk 
delta file will still be 5GB in size and will most likely only grow, Corruption 
to a parent vmdk on the primary datastore will impact other VMs that are 
dependent on it - hence reliable storage is needed.
Perfomance may degrade overtime if rate of change is high - also pending your 
storage backend Snapshotting (using vmware snapshot feature), will work, but if 
you have a complex snapshot three and some delta vmdk under this tree get 
corrupted, you may loose your data (until good working state) - this issue 
applies to snapshots in general


Full Clones:

Gets your independent disks with no delta complexity, at the expense of extra 
storage and some IO if you dont have FAST technology enabled.
Corruption to a vmdk file, affects only 1 VM.
If the VM has heavy read and write IO, you should consider running it as full 
clone as you will avoid delta complexity.

There are probably more reasons, just cant think of them now,

Regards
ilya



On 6/11/14, 7:42 AM, Steve Searles wrote:
 Can someone speak about using Linked Clones vs Full Clones in a production CS 
 environment?  What is the performance impact on the parent virtual machine? 
 What type of density can be expected if all the child vm's are performing 
 read operations from the same snapshot of the parent VM? What are the dangers 
 of using linked clones in this manner? What are the best practices from the 
 CS community?


 Steve Searles

Vmware Full Clones vs Linked Clones

[Steve Searles]

Can someone speak about using Linked Clones vs Full Clones in a production CS 
environment?  What is the performance impact on the parent virtual machine? 
What type of density can be expected if all the child vm’s are performing read 
operations from the same snapshot of the parent VM? What are the dangers of 
using linked clones in this manner? What are the best practices from the CS 
community?


Steve Searles

Re: Management Server Startup Issue

[Steve Searles]

Yes, I will see if I can get any more debugging info out of it.  Yes that is 
the URL we are using to access.  In the condition where the bin log is on I get 
a 404 from tomcat meaning that the application has not started.  I will try 
some further debugging on the mysql side as well.  This is not an upgrade but 
rather a fresh install of 4.3.  That is the problem I am seeing as well, no 
useful debug information. 

Steve Searles

On Jun 10, 2014, at 8:21 AM, Rafael Weingartner rafaelweingart...@gmail.com 
wrote:

 I thought that it could be a missing dependency problem, because with CS
 4.3 upgrade from 4.2.1 we had one. The mysql jdbc .jar was not in the
 classpath. So, we had to manually add it.
 
 I did not see any expressive tty in your log files.
 Could you look for some more meaningful java stack traces, then we might be
 able to help you.
 BTW, how are you trying to access the CS? http://serverIp:8080/client ?
 
 
 On Mon, Jun 9, 2014 at 11:31 PM, Steve Searles ssear...@zimcom.net wrote:
 
 It is also worth noting that these are the last two DB queries that happen
 before the stop. So it may have something to do with cloudbridge?
 
 
 
 140609 22:28:00 8 Connect   cl...@db01.cvg.cs.zimcom.net on
 cloudbridge
8 Query /* mysql-connector-java-5.1.17-SNAPSHOT (
 Revision: ${bzr.revision-id} ) */SHOW VARIABLES WHERE Variable_name
 ='language' OR Variable_name = 'net_write_timeout' OR Variable_name =
 'interactive_timeout' OR Variable_name = 'wait_timeout' OR Variable_name =
 'character_set_client' OR Variable_name = 'character_set_connection' OR
 Variable_name = 'character_set' OR Variable_name = 'character_set_server'
 OR Variable_name = 'tx_isolation' OR Variable_name =
 'transaction_isolation' OR Variable_name = 'character_set_results' OR
 Variable_name = 'timezone' OR Variable_name = 'time_zone' OR Variable_name
 = 'system_time_zone' OR Variable_name = 'lower_case_table_names' OR
 Variable_name = 'max_allowed_packet' OR Variable_name = 'net_buffer_length'
 OR Variable_name = 'sql_mode' OR Variable_name = 'query_cache_type' OR
 Variable_name = 'query_cache_size' OR Variable_name = 'init_connect'
8 Query /* mysql-connector-java-5.1.17-SNAPSHOT (
 Revision: ${bzr.revision-id} ) */SELECT @@session.auto_increment_increment
8 Query SHOW COLLATION
8 Query SET NAMES latin1
8 Query SET character_set_results = NULL
8 Query SET autocommit=1
8 Query SET sql_mode='STRICT_TRANS_TABLES'
8 Query SET autocommit=0
8 Query SET autocommit=0
8 Query SELECT offering_bundle.ID,
 offering_bundle.AmazonEC2Offering, offering_bundle.CloudStackOffering FROM
 offering_bundle
8 Query commit
8 Query rollback
8 Query SET autocommit=1
8 Query SET autocommit=0
8 Query SET autocommit=0
8 Query SELECT mhost.ID, mhost.MHostKey,
 mhost.Host, mhost.Version, mhost.LastHeartbeatTime FROM mhost WHERE
 mhost.MHostKey = _binary'00:50:56:92:04:18'  ORDER BY RAND() LIMIT 1
8 Query commit
8 Query rollback
8 Query SET autocommit=1
8 Query SET autocommit=0
8 Query SET autocommit=0
8 Query UPDATE mhost SET mhost.Host=_binary'
 http://localhost:7080/awsapi' WHERE mhost.ID = 1
8 Query commit
8 Query rollback
8 Query SET autocommit=1
8 Query SET autocommit=0
8 Query SET autocommit=0
8 Query SELECT shost.ID, shost.Host,
 shost.HostType, shost.MHostID, shost.ExportRoot, shost.UserOnHost,
 shost.UserPassword FROM shost WHERE shost.MHostID = 1  AND shost.ExportRoot
 = _binary'/Users/john1/S3-Mount'  ORDER BY RAND() LIMIT 1
8 Query commit
8 Query rollback
8 Query SET autocommit=1
9 Connect   cl...@db01.cvg.cs.zimcom.net on cloud
9 Query /* mysql-connector-java-5.1.17-SNAPSHOT (
 Revision: ${bzr.revision-id} ) */SHOW VARIABLES WHERE Variable_name
 ='language' OR Variable_name = 'net_write_timeout' OR Variable_name =
 'interactive_timeout' OR Variable_name = 'wait_timeout' OR Variable_name =
 'character_set_client' OR Variable_name = 'character_set_connection' OR
 Variable_name = 'character_set' OR Variable_name = 'character_set_server'
 OR Variable_name = 'tx_isolation' OR Variable_name =
 'transaction_isolation' OR Variable_name = 'character_set_results' OR
 Variable_name = 'timezone' OR Variable_name = 'time_zone' OR Variable_name
 = 'system_time_zone' OR Variable_name

Re: Management Server Startup Issue

[Steve Searles]

 overflowToOffHeap = false 
maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned =
false ]
INFO  [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = 
DiskOfferingDaoImpl status = STATUS_ALIVE eternal = false overflowToDisk = 
false maxEntriesLocalHeap = 50 maxEntriesLocalDisk = 0 
memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 600 timeToIdleSeconds = 300 
persistence = none diskExpiryThr
eadIntervalSeconds = 120 cacheEventListeners: 
net.sf.ehcache.statistics.LiveCacheStatisticsWrapper  hitCount = 0 
memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 
missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false 
maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = fal
se ]



[cid:image001.jpg@01CF54D3.D25E1ED0]

On Jun 10, 2014, at 10:30 AM, Rafael Weingartner 
rafaelweingart...@gmail.commailto:rafaelweingart...@gmail.com wrote:

you do not need to debug into the mysql, you just need to find the stack
trace in the tomcat that is causing the app to shutdown during the start up
process


On Tue, Jun 10, 2014 at 11:26 AM, Steve Searles 
ssear...@zimcom.netmailto:ssear...@zimcom.net wrote:

Yes, I will see if I can get any more debugging info out of it.  Yes that
is the URL we are using to access.  In the condition where the bin log is
on I get a 404 from tomcat meaning that the application has not started.  I
will try some further debugging on the mysql side as well.  This is not an
upgrade but rather a fresh install of 4.3.  That is the problem I am seeing
as well, no useful debug information.

Steve Searles

On Jun 10, 2014, at 8:21 AM, Rafael Weingartner 
rafaelweingart...@gmail.commailto:rafaelweingart...@gmail.com wrote:

I thought that it could be a missing dependency problem, because with CS
4.3 upgrade from 4.2.1 we had one. The mysql jdbc .jar was not in the
classpath. So, we had to manually add it.

I did not see any expressive tty in your log files.
Could you look for some more meaningful java stack traces, then we might
be
able to help you.
BTW, how are you trying to access the CS? http://serverIp:8080/client ?


On Mon, Jun 9, 2014 at 11:31 PM, Steve Searles 
ssear...@zimcom.netmailto:ssear...@zimcom.net
wrote:

It is also worth noting that these are the last two DB queries that
happen
before the stop. So it may have something to do with cloudbridge?



140609 22:28:00 8 Connect   
cl...@db01.cvg.cs.zimcom.netmailto:cl...@db01.cvg.cs.zimcom.net on
cloudbridge
  8 Query /* mysql-connector-java-5.1.17-SNAPSHOT (
Revision: ${bzr.revision-id} ) */SHOW VARIABLES WHERE Variable_name
='language' OR Variable_name = 'net_write_timeout' OR Variable_name =
'interactive_timeout' OR Variable_name = 'wait_timeout' OR
Variable_name =
'character_set_client' OR Variable_name = 'character_set_connection' OR
Variable_name = 'character_set' OR Variable_name =
'character_set_server'
OR Variable_name = 'tx_isolation' OR Variable_name =
'transaction_isolation' OR Variable_name = 'character_set_results' OR
Variable_name = 'timezone' OR Variable_name = 'time_zone' OR
Variable_name
= 'system_time_zone' OR Variable_name = 'lower_case_table_names' OR
Variable_name = 'max_allowed_packet' OR Variable_name =
'net_buffer_length'
OR Variable_name = 'sql_mode' OR Variable_name = 'query_cache_type' OR
Variable_name = 'query_cache_size' OR Variable_name = 'init_connect'
  8 Query /* mysql-connector-java-5.1.17-SNAPSHOT (
Revision: ${bzr.revision-id} ) */SELECT
@@session.auto_increment_increment
  8 Query SHOW COLLATION
  8 Query SET NAMES latin1
  8 Query SET character_set_results = NULL
  8 Query SET autocommit=1
  8 Query SET sql_mode='STRICT_TRANS_TABLES'
  8 Query SET autocommit=0
  8 Query SET autocommit=0
  8 Query SELECT offering_bundle.ID,
offering_bundle.AmazonEC2Offering, offering_bundle.CloudStackOffering
FROM
offering_bundle
  8 Query commit
  8 Query rollback
  8 Query SET autocommit=1
  8 Query SET autocommit=0
  8 Query SET autocommit=0
  8 Query SELECT mhost.ID, mhost.MHostKey,
mhost.Host, mhost.Version, mhost.LastHeartbeatTime FROM mhost WHERE
mhost.MHostKey = _binary'00:50:56:92:04:18'  ORDER BY RAND() LIMIT 1
  8 Query commit
  8 Query rollback
  8 Query SET autocommit=1
  8 Query SET autocommit=0
  8 Query SET autocommit=0
  8 Query UPDATE mhost SET mhost.Host=_binary'
http://localhost:7080/awsapi' WHERE mhost.ID = 1
  8 Query commit
  8 Query rollback
  8 Query SET autocommit=1
  8 Query SET autocommit=0
  8 Query SET autocommit=0

Re: Management Server Startup Issue

[Steve Searles]

Nothing interesting in the management server logs, you saw the complete ones I 
sent yesterday.  I have attached the complete catalina.out below.  This is 
starting to feel like a problem with the DB locking the tables during the 
startup integrity check.  I am able to lock the tables via the command line 
with the bin log enabled without issue though.

https://dl.dropboxusercontent.com/u/51895629/catalina.binlog.gz

Steve Searles

[cid:image001.jpg@01CF54D3.D25E1ED0]

On Jun 10, 2014, at 2:29 PM, Rafael Weingartner 
rafaelweingart...@gmail.commailto:rafaelweingart...@gmail.com wrote:

this is interesting:

Jun 10, 2014 2:12:48 PM org.apache.catalina.core.StandardContext start
SEVERE: Context [/client] startup failed due to previous error

does it have any extra log before that message?
Did you find anything at /var/log/cloudstack/management-server.log?


On Tue, Jun 10, 2014 at 3:24 PM, Steve Searles 
ssear...@zimcom.netmailto:ssear...@zimcom.net wrote:

I found the following differences in the catalina.out file with bin log
enabled and disabled.  Can you point me in the right direction on debugging
the actual /client application further.  Obviously the problem is

Jun 10, 2014 2:12:48 PM org.apache.catalina.core.StandardContext start
SEVERE: Error listenerStart

I could use a point in the right direction to get further debugging
info.


WITH BINLOG

INFO  [c.c.u.c.ComponentContext] (main:null) Setup Spring Application
context
INFO  [c.c.s.GsonHelper] (main:null) Default Builder inited.
Jun 10, 2014 2:12:48 PM org.apache.catalina.core.StandardContext start
SEVERE: Error listenerStart
Jun 10, 2014 2:12:48 PM org.apache.catalina.core.StandardContext start
SEVERE: Context [/client] startup failed due to previous errors
Jun 10, 2014 2:12:48 PM org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads
SEVERE: A web application appears to have started a thread named
[FileWatchdog] but has failed to stop it. This is very likely to create a
memory leak.
Jun 10, 2014 2:12:48 PM org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads
SEVERE: A web application appears to have started a thread named [Timer-0]
but has failed to stop it. This is very likely to create a memory leak.
Jun 10, 2014 2:12:48 PM org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads
SEVERE: A web application appears to have started a thread named
[ConnectionConcierge-1] but has failed to stop it. This is very likely to
create a memory leak.
Jun 10, 2014 2:12:48 PM org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads
SEVERE: A web application appears to have started a thread named
[ClusteredAgentManager Timer] but has failed to stop it. This is very
likely to create a memory leak.
Jun 10, 2014 2:12:48 PM org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads
SEVERE: A web application appears to have started a thread named [Timer-1]
but has failed to stop it. This is very likely to create a memory leak.
Jun 10, 2014 2:12:48 PM org.apache.catalina.loader.WebappClassLoader
clearThreadLocalMap
SEVERE: A web application created a ThreadLocal with key of type
[org.springframework.core.NamedThreadLocal] (value [Prototype beans
currently in creation]) and a value of type [null] (value [null]) but
failed to remove it when the web application was stopped. To prevent a
memory leak, the ThreadLocal has be
en forcibly removed.


WITHOUT BINLOG


INFO  [c.c.u.c.ComponentContext] (main:null) Setup Spring Application
context
INFO  [c.c.s.GsonHelper] (main:null) Default Builder inited.
INFO  [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:null) Running system
integrity checker com.cloud.upgrade.DatabaseIntegrityChecker@7f2af4a3
INFO  [c.c.u.DatabaseIntegrityChecker] (main:null) Grabbing lock to check
for database integrity.
INFO  [c.c.u.DatabaseIntegrityChecker] (main:null) Performing database
integrity check
INFO  [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:null) Running system
integrity checker
org.apache.cloudstack.utils.identity.ManagementServerNode@7c73b9ad
INFO  [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:null) Configuring
CloudStack Components
INFO  [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name =
UserDaoImpl status = STATUS_ALIVE eternal = false overflowToDisk = false
maxEntriesLocalHeap = 5000 maxEntriesLocalDisk = 0
memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 300 timeToIdleSeconds =
300 persistence = none diskExpiryThreadInt
ervalSeconds = 120 cacheEventListeners:
net.sf.ehcache.statistics.LiveCacheStatisticsWrapper  hitCount = 0
memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0
missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false
maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
INFO  [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name =
VMTemplateDaoImpl status = STATUS_ALIVE eternal = false overflowToDisk =
false maxEntriesLocalHeap = 100 maxEntriesLocalDisk = 0
memoryStoreEvictionPolicy = LRU

Re: Management Server Startup Issue

[Steve Searles]

I agree with what you are saying.  Yes. addressing-1.5.4.mar errors I have seen 
before.  I have no webapps7080 directory so that is why that error is being 
thrown.  I still am wondering even if a dependency is missing why is it only a 
dependency when sql replication(bin log) is on? Can you tell me how to get a 
stack trace from CS startup?

That file is present in these locations on the management node.

[root@csmgmt01 /]# find / -name addressing-1.5.4.mar
/root/.m2/repository/org/apache/axis2/addressing/1.5.4/addressing-1.5.4.mar
/root/apache-cloudstack-4.3.0-src/dist/rpmbuild/BUILD/cloudstack-4.3.0/awsapi/target/cloud-awsapi-4.3.0/WEB-INF/modules/addressing-1.5.4.mar
/usr/share/cloudstack-bridge/webapps/awsapi/WEB-INF/modules/addressing-1.5.4.mar
[root@csmgmt01 /]#



is it just a symlink that is missing?



[root@csmgmt01 webapps]# ls -las
total 12
4 drwxr-xr-x.  3 root root 4096 Mar 31 18:21 .
4 drwxr-xr-x.  4 root root 4096 Mar 31 18:21 ..
4 drwxr-xr-x. 11 root root 4096 Mar 31 18:21 client
[root@csmgmt01 webapps]# cd client/
[root@csmgmt01 client]# ls
css  dictionary.jsp  images  index.jsp  lib  META-INF  modules  plugins  
scripts  tests  WEB-INF
[root@csmgmt01 client]# ls -lash
total 264K
4.0K drwxr-xr-x. 11 root root 4.0K Mar 31 18:21 .
4.0K drwxr-xr-x.  3 root root 4.0K Mar 31 18:21 ..
4.0K drwxr-xr-x.  2 root root 4.0K Mar 31 18:21 css
116K -rw-r--r--.  1 root root 115K Mar 27 01:06 dictionary.jsp
4.0K drwxr-xr-x.  2 root root 4.0K Mar 31 18:21 images
104K -rw-r--r--.  1 root root 103K Mar 27 01:06 index.jsp
4.0K drwxr-xr-x.  5 root root 4.0K Mar 31 18:21 lib
4.0K drwxr-xr-x.  2 root root 4.0K Mar 27 01:06 META-INF
4.0K drwxr-xr-x.  6 root root 4.0K Mar 31 18:21 modules
4.0K drwxr-xr-x.  3 root root 4.0K Mar 31 18:21 plugins
4.0K drwxr-xr-x.  4 root root 4.0K Mar 31 18:21 scripts
4.0K drwxr-xr-x.  2 root root 4.0K Mar 31 18:21 tests
4.0K drwxr-xr-x.  4 root root 4.0K Mar 31 18:21 WEB-INF
[root@csmgmt01 client]# pwd
/usr/share/cloudstack-management/webapps/client
[root@csmgmt01 client]#





Steve Searles


On Jun 10, 2014, at 3:04 PM, Rafael Weingartner 
rafaelweingart...@gmail.commailto:rafaelweingart...@gmail.com wrote:

Look at this point:
SEVERE: Error listenerStart
Jun 10, 2014 2:12:48 PM org.apache.catalina.core.StandardContext start
SEVERE: Context [/client] startup failed due to previous errors
I could not find the stacktrace into the logs you provided. But, it has to
have some stack which would help to understand why the CS is not getting up.

This is a good point to look at:
SEVERE: The addressing-1.5.4.mar module, which is not valid, caused The
module.xml file cannot be found for the module:
/usr/share/cloudstack-management/webapps7080/awsapi/WEB-INF/modules/addressing-1.5.4.mar
org.apache.axis2.deployment.DeploymentException: The module.xml file cannot
be found for the module:
/usr/share/cloudstack-management/webapps7080/awsapi/WEB-INF/modules/addressing-1.5.4.mar
   at
org.apache.axis2.deployment.repository.util.ArchiveReader.readModuleArchive(ArchiveReader.java:529)
   at
org.apache.axis2.deployment.ModuleDeployer.deploy(ModuleDeployer.java:75)
   at
org.apache.axis2.deployment.repository.util.DeploymentFileData.deploy(DeploymentFileData.java:136)
   at
org.apache.axis2.deployment.DeploymentEngine.doDeploy(DeploymentEngine.java:661)



It seems there is a .xml file being missing.
But, still even though the awsapi is not working, the CS app should run
just fine, unless it depends on that service.

This is also something interesting:
Jun 10, 2014 2:12:58 PM org.apache.catalina.loader.WebappClassLoader
loadClass
INFO: Illegal access: this web application instance has been stopped
already.  Could not load
org.apache.cloudstack.managed.context.ManagedContextTimerTask$1.  The
eventual following stack trace is caused by an error thrown for debugging
purposes as well as to attempt to terminate the thread which caused the
illegal access, and has no functional impact.
java.lang.IllegalStateException
   at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1369)

it seems there is a missing dependency.

BTW: out of curiosity, what do tou have in folder
/usr/share/cloudstack-management/webapps


On Tue, Jun 10, 2014 at 3:53 PM, Steve Searles 
ssear...@zimcom.netmailto:ssear...@zimcom.net wrote:

 Nothing interesting in the management server logs, you saw the complete
ones I sent yesterday.  I have attached the complete catalina.out below.
This is starting to feel like a problem with the DB locking the tables
during the startup integrity check.  I am able to lock the tables via the
command line with the bin log enabled without issue though.

https://dl.dropboxusercontent.com/u/51895629/catalina.binlog.gz

Steve Searles


On Jun 10, 2014, at 2:29 PM, Rafael Weingartner 
rafaelweingart...@gmail.commailto:rafaelweingart...@gmail.com wrote:

this is interesting:

Jun 10, 2014 2:12:48 PM org.apache.catalina.core.StandardContext start
SEVERE: Context [/client] startup

Re: Management Server Startup Issue

[Steve Searles]

Ok I was wrong, the addressing-1.5.4.mar is present.

[root@csmgmt01 modules]# pwd
/usr/share/cloudstack-management/webapps7080/awsapi/WEB-INF/modules
[root@csmgmt01 modules]# ls -lash
total 344K
4.0K drwxr-xr-x. 2 cloud cloud 4.0K Mar 31 18:21 .
4.0K drwxr-xr-x. 7 cloud cloud 4.0K Mar 31 18:21 ..
124K -rw-r--r--. 1 cloud cloud 124K Mar 27 01:06 addressing-1.5.4.mar
 36K -rw-r--r--. 1 cloud cloud  34K Mar 27 01:06 addressing-1.5.6.mar
 16K -rw-r--r--. 1 cloud cloud  15K Mar 27 01:06 axis2-jaxws-mar-1.5.6.mar
 40K -rw-r--r--. 1 cloud cloud  38K Mar 27 01:06 mex-1.5.6.mar
4.0K -rw-r--r--. 1 cloud cloud  139 Mar 27 01:06 modules.list
 20K -rw-r--r--. 1 cloud cloud  19K Mar 27 01:06 mtompolicy-1.5.6.mar
 20K -rw-r--r--. 1 cloud cloud  18K Mar 27 01:06 ping-1.5.6.mar
 12K -rw-r--r--. 1 cloud cloud 9.5K Mar 27 01:06 rahas-1.5.1.mar
 12K -rw-r--r--. 1 cloud cloud 9.4K Mar 27 01:06 rampart-1.5.1.mar
 36K -rw-r--r--. 1 cloud cloud  33K Mar 27 01:06 scripting-1.5.6.mar
 16K -rw-r--r--. 1 cloud cloud  16K Mar 27 01:06 soapmonitor-1.5.6.mar
[root@csmgmt01 modules]#



The error is complaining about the module.xml file missing.

Steve Searles

On Jun 10, 2014, at 5:21 PM, Steve Searles 
ssear...@zimcom.netmailto:ssear...@zimcom.net wrote:

I agree with what you are saying.  Yes. addressing-1.5.4.mar errors I have seen 
before.  I have no webapps7080 directory so that is why that error is being 
thrown.  I still am wondering even if a dependency is missing why is it only a 
dependency when sql replication(bin log) is on? Can you tell me how to get a 
stack trace from CS startup?

That file is present in these locations on the management node.

[root@csmgmt01 /]# find / -name addressing-1.5.4.mar
/root/.m2/repository/org/apache/axis2/addressing/1.5.4/addressing-1.5.4.mar
/root/apache-cloudstack-4.3.0-src/dist/rpmbuild/BUILD/cloudstack-4.3.0/awsapi/target/cloud-awsapi-4.3.0/WEB-INF/modules/addressing-1.5.4.mar
/usr/share/cloudstack-bridge/webapps/awsapi/WEB-INF/modules/addressing-1.5.4.mar
[root@csmgmt01 /]#



is it just a symlink that is missing?



[root@csmgmt01 webapps]# ls -las
total 12
4 drwxr-xr-x.  3 root root 4096 Mar 31 18:21 .
4 drwxr-xr-x.  4 root root 4096 Mar 31 18:21 ..
4 drwxr-xr-x. 11 root root 4096 Mar 31 18:21 client
[root@csmgmt01 webapps]# cd client/
[root@csmgmt01 client]# ls
css  dictionary.jsp  images  index.jsp  lib  META-INF  modules  plugins  
scripts  tests  WEB-INF
[root@csmgmt01 client]# ls -lash
total 264K
4.0K drwxr-xr-x. 11 root root 4.0K Mar 31 18:21 .
4.0K drwxr-xr-x.  3 root root 4.0K Mar 31 18:21 ..
4.0K drwxr-xr-x.  2 root root 4.0K Mar 31 18:21 css
116K -rw-r--r--.  1 root root 115K Mar 27 01:06 dictionary.jsp
4.0K drwxr-xr-x.  2 root root 4.0K Mar 31 18:21 images
104K -rw-r--r--.  1 root root 103K Mar 27 01:06 index.jsp
4.0K drwxr-xr-x.  5 root root 4.0K Mar 31 18:21 lib
4.0K drwxr-xr-x.  2 root root 4.0K Mar 27 01:06 META-INF
4.0K drwxr-xr-x.  6 root root 4.0K Mar 31 18:21 modules
4.0K drwxr-xr-x.  3 root root 4.0K Mar 31 18:21 plugins
4.0K drwxr-xr-x.  4 root root 4.0K Mar 31 18:21 scripts
4.0K drwxr-xr-x.  2 root root 4.0K Mar 31 18:21 tests
4.0K drwxr-xr-x.  4 root root 4.0K Mar 31 18:21 WEB-INF
[root@csmgmt01 client]# pwd
/usr/share/cloudstack-management/webapps/client
[root@csmgmt01 client]#





Steve Searles


On Jun 10, 2014, at 3:04 PM, Rafael Weingartner 
rafaelweingart...@gmail.commailto:rafaelweingart...@gmail.commailto:rafaelweingart...@gmail.com
 wrote:

Look at this point:
SEVERE: Error listenerStart
Jun 10, 2014 2:12:48 PM org.apache.catalina.core.StandardContext start
SEVERE: Context [/client] startup failed due to previous errors
I could not find the stacktrace into the logs you provided. But, it has to
have some stack which would help to understand why the CS is not getting up.

This is a good point to look at:
SEVERE: The addressing-1.5.4.mar module, which is not valid, caused The
module.xml file cannot be found for the module:
/usr/share/cloudstack-management/webapps7080/awsapi/WEB-INF/modules/addressing-1.5.4.mar
org.apache.axis2.deployment.DeploymentException: The module.xml file cannot
be found for the module:
/usr/share/cloudstack-management/webapps7080/awsapi/WEB-INF/modules/addressing-1.5.4.mar
  at
org.apache.axis2.deployment.repository.util.ArchiveReader.readModuleArchive(ArchiveReader.java:529)
  at
org.apache.axis2.deployment.ModuleDeployer.deploy(ModuleDeployer.java:75)
  at
org.apache.axis2.deployment.repository.util.DeploymentFileData.deploy(DeploymentFileData.java:136)
  at
org.apache.axis2.deployment.DeploymentEngine.doDeploy(DeploymentEngine.java:661)



It seems there is a .xml file being missing.
But, still even though the awsapi is not working, the CS app should run
just fine, unless it depends on that service.

This is also something interesting:
Jun 10, 2014 2:12:58 PM org.apache.catalina.loader.WebappClassLoader
loadClass
INFO: Illegal access: this web application instance has been stopped
already.  Could not load

Re: Management Server Startup Issue

[Steve Searles]

Is 
http://mail-archives.apache.org/mod_mbox/cloudstack-issues/201401.mbox/%3CJIRA.12687852.1389186771873.61435.1389186832715@arcas%3E
   relevant here?



On Jun 10, 2014, at 5:31 PM, Steve Searles 
ssear...@zimcom.netmailto:ssear...@zimcom.net wrote:

Ok I was wrong, the addressing-1.5.4.mar is present.

[root@csmgmt01 modules]# pwd
/usr/share/cloudstack-management/webapps7080/awsapi/WEB-INF/modules
[root@csmgmt01 modules]# ls -lash
total 344K
4.0K drwxr-xr-x. 2 cloud cloud 4.0K Mar 31 18:21 .
4.0K drwxr-xr-x. 7 cloud cloud 4.0K Mar 31 18:21 ..
124K -rw-r--r--. 1 cloud cloud 124K Mar 27 01:06 addressing-1.5.4.mar
36K -rw-r--r--. 1 cloud cloud  34K Mar 27 01:06 addressing-1.5.6.mar
16K -rw-r--r--. 1 cloud cloud  15K Mar 27 01:06 axis2-jaxws-mar-1.5.6.mar
40K -rw-r--r--. 1 cloud cloud  38K Mar 27 01:06 mex-1.5.6.mar
4.0K -rw-r--r--. 1 cloud cloud  139 Mar 27 01:06 modules.list
20K -rw-r--r--. 1 cloud cloud  19K Mar 27 01:06 mtompolicy-1.5.6.mar
20K -rw-r--r--. 1 cloud cloud  18K Mar 27 01:06 ping-1.5.6.mar
12K -rw-r--r--. 1 cloud cloud 9.5K Mar 27 01:06 rahas-1.5.1.mar
12K -rw-r--r--. 1 cloud cloud 9.4K Mar 27 01:06 rampart-1.5.1.mar
36K -rw-r--r--. 1 cloud cloud  33K Mar 27 01:06 scripting-1.5.6.mar
16K -rw-r--r--. 1 cloud cloud  16K Mar 27 01:06 soapmonitor-1.5.6.mar
[root@csmgmt01 modules]#



The error is complaining about the module.xml file missing.

Steve Searles

On Jun 10, 2014, at 5:21 PM, Steve Searles 
ssear...@zimcom.netmailto:ssear...@zimcom.netmailto:ssear...@zimcom.net 
wrote:

I agree with what you are saying.  Yes. addressing-1.5.4.mar errors I have seen 
before.  I have no webapps7080 directory so that is why that error is being 
thrown.  I still am wondering even if a dependency is missing why is it only a 
dependency when sql replication(bin log) is on? Can you tell me how to get a 
stack trace from CS startup?

That file is present in these locations on the management node.

[root@csmgmt01 /]# find / -name addressing-1.5.4.mar
/root/.m2/repository/org/apache/axis2/addressing/1.5.4/addressing-1.5.4.mar
/root/apache-cloudstack-4.3.0-src/dist/rpmbuild/BUILD/cloudstack-4.3.0/awsapi/target/cloud-awsapi-4.3.0/WEB-INF/modules/addressing-1.5.4.mar
/usr/share/cloudstack-bridge/webapps/awsapi/WEB-INF/modules/addressing-1.5.4.mar
[root@csmgmt01 /]#



is it just a symlink that is missing?



[root@csmgmt01 webapps]# ls -las
total 12
4 drwxr-xr-x.  3 root root 4096 Mar 31 18:21 .
4 drwxr-xr-x.  4 root root 4096 Mar 31 18:21 ..
4 drwxr-xr-x. 11 root root 4096 Mar 31 18:21 client
[root@csmgmt01 webapps]# cd client/
[root@csmgmt01 client]# ls
css  dictionary.jsp  images  index.jsp  lib  META-INF  modules  plugins  
scripts  tests  WEB-INF
[root@csmgmt01 client]# ls -lash
total 264K
4.0K drwxr-xr-x. 11 root root 4.0K Mar 31 18:21 .
4.0K drwxr-xr-x.  3 root root 4.0K Mar 31 18:21 ..
4.0K drwxr-xr-x.  2 root root 4.0K Mar 31 18:21 css
116K -rw-r--r--.  1 root root 115K Mar 27 01:06 dictionary.jsp
4.0K drwxr-xr-x.  2 root root 4.0K Mar 31 18:21 images
104K -rw-r--r--.  1 root root 103K Mar 27 01:06 index.jsp
4.0K drwxr-xr-x.  5 root root 4.0K Mar 31 18:21 lib
4.0K drwxr-xr-x.  2 root root 4.0K Mar 27 01:06 META-INF
4.0K drwxr-xr-x.  6 root root 4.0K Mar 31 18:21 modules
4.0K drwxr-xr-x.  3 root root 4.0K Mar 31 18:21 plugins
4.0K drwxr-xr-x.  4 root root 4.0K Mar 31 18:21 scripts
4.0K drwxr-xr-x.  2 root root 4.0K Mar 31 18:21 tests
4.0K drwxr-xr-x.  4 root root 4.0K Mar 31 18:21 WEB-INF
[root@csmgmt01 client]# pwd
/usr/share/cloudstack-management/webapps/client
[root@csmgmt01 client]#





Steve Searles


On Jun 10, 2014, at 3:04 PM, Rafael Weingartner 
rafaelweingart...@gmail.commailto:rafaelweingart...@gmail.commailto:rafaelweingart...@gmail.commailto:rafaelweingart...@gmail.com
 wrote:

Look at this point:
SEVERE: Error listenerStart
Jun 10, 2014 2:12:48 PM org.apache.catalina.core.StandardContext start
SEVERE: Context [/client] startup failed due to previous errors
I could not find the stacktrace into the logs you provided. But, it has to
have some stack which would help to understand why the CS is not getting up.

This is a good point to look at:
SEVERE: The addressing-1.5.4.mar module, which is not valid, caused The
module.xml file cannot be found for the module:
/usr/share/cloudstack-management/webapps7080/awsapi/WEB-INF/modules/addressing-1.5.4.mar
org.apache.axis2.deployment.DeploymentException: The module.xml file cannot
be found for the module:
/usr/share/cloudstack-management/webapps7080/awsapi/WEB-INF/modules/addressing-1.5.4.mar
 at
org.apache.axis2.deployment.repository.util.ArchiveReader.readModuleArchive(ArchiveReader.java:529)
 at
org.apache.axis2.deployment.ModuleDeployer.deploy(ModuleDeployer.java:75)
 at
org.apache.axis2.deployment.repository.util.DeploymentFileData.deploy(DeploymentFileData.java:136)
 at
org.apache.axis2.deployment.DeploymentEngine.doDeploy(DeploymentEngine.java:661)



It seems there is a .xml file being missing.
But, still even though the awsapi is not working

Management Server Startup Issue

[Steve Searles]

Anyone have any idea why the management service would refuse to start if the DB 
server is configured as a master for replication?



log_bin=mysql-bin
server_id=1

Adding the above the my.cnf file of my DB server, restarting mysql then 
restarting cloudstack-managment will cause cloudstack to get hung on startup 
here.

2014-06-09 18:25:33,769 INFO  [c.c.u.c.ComponentContext] (main:null) Starting 
com.cloud.bridge.persist.dao.SHostDaoImpl_EnhancerByCloudStack_9da5499b
2014-06-09 18:25:33,769 INFO  [c.c.u.c.ComponentContext] (main:null) Starting 
com.cloud.bridge.persist.dao.SObjectDaoImpl_EnhancerByCloudStack_847fb98c
2014-06-09 18:25:33,769 INFO  [c.c.u.c.ComponentContext] (main:null) Starting 
com.cloud.bridge.persist.dao.CloudStackUserDaoImpl_EnhancerByCloudStack_7fce3ee6
2014-06-09 18:25:33,769 INFO  [c.c.u.c.ComponentContext] (main:null) Starting 
com.cloud.bridge.service.core.ec2.EC2Engine_EnhancerByCloudStack_d70c9e3c
2014-06-09 18:25:33,769 INFO  [c.c.u.c.ComponentContext] (main:null) Starting 
com.cloud.bridge.service.controller.s3.ServiceProvider_EnhancerByCloudStack_23d38b1

That is the last few lines of the management-server.log and it will hang there 
indefinitely.

I noticed this on startup as well.

2014-06-09 18:25:16,841 INFO  [c.c.u.d.T.Transaction] (main:null) Is Data Base 
High Availiability enabled? Ans : false
2014-06-09 18:25:17,164 DEBUG [c.c.u.d.ConnectionConcierge] (main:null) 
Registering a database connection for LockMaster1
2014-06-09 18:25:17,164 INFO  [c.c.u.d.Merovingian2] (main:null) Cleaning up 
locks for 345049793560
2014-06-09 18:25:17,175 INFO  [c.c.u.d.Merovingian2] (main:null) Released 0 
locks for 345049793560
2014-06-09 18:25:17,206 INFO  [o.a.c.s.l.CloudStackExtendedLifeCycle] 
(main:null) Running system integrity checker 
com.cloud.upgrade.DatabaseUpgradeChecker@63c8e4ae
2014-06-09 18:25:17,207 INFO  [c.c.u.DatabaseUpgradeChecker] (main:null) 
Grabbing lock to check for database upgrade.
2014-06-09 18:25:17,254 DEBUG [c.c.u.d.VersionDaoImpl] (main:null) Checking to 
see if the database is at a version before it was the version table is created
2014-06-09 18:25:17,295 INFO  [c.c.u.DatabaseUpgradeChecker] (main:null) DB 
version = 4.3.0 Code Version = 4.3.0
2014-06-09 18:25:17,296 INFO  [c.c.u.DatabaseUpgradeChecker] (main:null) DB 
version and code version matches so no upgrade needed.

The management server talking to the DB server at least enough to check the 
version; and mysql from the command line on the management server works fine as 
well.


This line caught my attention.


2014-06-09 18:25:16,841 INFO  [c.c.u.d.T.Transaction] (main:null) Is Data Base 
High Availiability enabled? Ans : false

Is there something I need to set when using mysql replication specific to 
cloudstack? I do not see anything in the documentation on this other than 
setting up basic replication.


if I comment out the above two lines in my.cnf and restart the management 
server it fires up just fine.  I don’t get how enabling bin log would make a 
difference at all.

Any help or explication for this strange issue would be appreciated.


- Steve Searles

RE: Management Server Startup Issue

[Steve Searles]

Rafael, Thank you for responding so quickly.  Yes 4.3.0. I am using a dedicated 
DB servers and dedicated management servers.  I will provide whatever log files 
you would find useful. It would be odd if it was a dependancy since CS really 
has no visibility to the sql server other than a connect string? Or so I 
thought; and the system is running fine with the binlog off.  With the bin log 
on I can run queries from the command line and everything seems normal from the 
sql end,  but the management server will not run.  Toggle binlog off CS works 
fine.  The replication slave has not even been started yet.  Let me know what 
other info I can send over that would be of help.  There is nothing in the 
catalina logs of interest and my logging level is currently set to trace for 
the management-server.log.  I will turn on the sql query log and compare them 
between startups of the management nodes perhaps that will yeild something.  I 
have included a dropbox link with the management server log from a normal 
startup of CS and one with mysql binlog enabled.  Any help would be greatly 
appriciated. 


https://dl.dropboxusercontent.com/u/51895629/management-server.logs.tar.gz



Steve Searles


-Original Message-
From: Rafael Weingartner [mailto:rafaelweingart...@gmail.com] 
Sent: Monday, June 9, 2014 6:57 PM
To: users@cloudstack.apache.org
Subject: Re: Management Server Startup Issue

please give more logs.
It is the 4.3.0 right? I bet there is a problem with a missing dependency.


On Mon, Jun 9, 2014 at 7:32 PM, Steve Searles ssear...@zimcom.net wrote:

 Anyone have any idea why the management service would refuse to start 
 if the DB server is configured as a master for replication?



 log_bin=mysql-bin
 server_id=1

 Adding the above the my.cnf file of my DB server, restarting mysql 
 then restarting cloudstack-managment will cause cloudstack to get hung 
 on startup here.

 2014-06-09 18:25:33,769 INFO  [c.c.u.c.ComponentContext] (main:null) 
 Starting 
 com.cloud.bridge.persist.dao.SHostDaoImpl_EnhancerByCloudStack_9da5499
 b
 2014-06-09 18:25:33,769 INFO  [c.c.u.c.ComponentContext] (main:null) 
 Starting 
 com.cloud.bridge.persist.dao.SObjectDaoImpl_EnhancerByCloudStack_847fb
 98c
 2014-06-09 18:25:33,769 INFO  [c.c.u.c.ComponentContext] (main:null) 
 Starting
 com.cloud.bridge.persist.dao.CloudStackUserDaoImpl_EnhancerByCloudStac
 k_7fce3ee6
 2014-06-09 18:25:33,769 INFO  [c.c.u.c.ComponentContext] (main:null) 
 Starting 
 com.cloud.bridge.service.core.ec2.EC2Engine_EnhancerByCloudStack_d70c9
 e3c
 2014-06-09 18:25:33,769 INFO  [c.c.u.c.ComponentContext] (main:null) 
 Starting
 com.cloud.bridge.service.controller.s3.ServiceProvider_EnhancerByCloud
 Stack_23d38b1

 That is the last few lines of the management-server.log and it will 
 hang there indefinitely.

 I noticed this on startup as well.

 2014-06-09 18:25:16,841 INFO  [c.c.u.d.T.Transaction] (main:null) Is 
 Data Base High Availiability enabled? Ans : false
 2014-06-09 18:25:17,164 DEBUG [c.c.u.d.ConnectionConcierge] 
 (main:null) Registering a database connection for LockMaster1
 2014-06-09 18:25:17,164 INFO  [c.c.u.d.Merovingian2] (main:null) 
 Cleaning up locks for 345049793560
 2014-06-09 18:25:17,175 INFO  [c.c.u.d.Merovingian2] (main:null) 
 Released
 0 locks for 345049793560
 2014-06-09 18:25:17,206 INFO  [o.a.c.s.l.CloudStackExtendedLifeCycle]
 (main:null) Running system integrity checker 
 com.cloud.upgrade.DatabaseUpgradeChecker@63c8e4ae
 2014-06-09 18:25:17,207 INFO  [c.c.u.DatabaseUpgradeChecker] 
 (main:null) Grabbing lock to check for database upgrade.
 2014-06-09 18:25:17,254 DEBUG [c.c.u.d.VersionDaoImpl] (main:null) 
 Checking to see if the database is at a version before it was the 
 version table is created
 2014-06-09 18:25:17,295 INFO  [c.c.u.DatabaseUpgradeChecker] 
 (main:null) DB version = 4.3.0 Code Version = 4.3.0
 2014-06-09 18:25:17,296 INFO  [c.c.u.DatabaseUpgradeChecker] 
 (main:null) DB version and code version matches so no upgrade needed.

 The management server talking to the DB server at least enough to 
 check the version; and mysql from the command line on the management 
 server works fine as well.


 This line caught my attention.


 2014-06-09 18:25:16,841 INFO  [c.c.u.d.T.Transaction] (main:null) Is 
 Data Base High Availiability enabled? Ans : false

 Is there something I need to set when using mysql replication specific 
 to cloudstack? I do not see anything in the documentation on this 
 other than setting up basic replication.


 if I comment out the above two lines in my.cnf and restart the 
 management server it fires up just fine.  I don’t get how enabling bin 
 log would make a difference at all.

 Any help or explication for this strange issue would be appreciated.


 - Steve Searles




--
Rafael Weingärtner

RE: Management Server Startup Issue

[Steve Searles]

, configuration.value 
FROM configuration WHERE configuration.name = _binary'enable.ec2.api'  ORDER BY 
RAND() LIMIT 1
9 Query commit
9 Query rollback
9 Query SET autocommit=1



Steve Searles



-Original Message-
From: Steve Searles [mailto:ssear...@zimcom.net] 
Sent: Monday, June 9, 2014 10:19 PM
To: users@cloudstack.apache.org
Subject: RE: Management Server Startup Issue

Rafael, Thank you for responding so quickly.  Yes 4.3.0. I am using a dedicated 
DB servers and dedicated management servers.  I will provide whatever log files 
you would find useful. It would be odd if it was a dependancy since CS really 
has no visibility to the sql server other than a connect string? Or so I 
thought; and the system is running fine with the binlog off.  With the bin log 
on I can run queries from the command line and everything seems normal from the 
sql end,  but the management server will not run.  Toggle binlog off CS works 
fine.  The replication slave has not even been started yet.  Let me know what 
other info I can send over that would be of help.  There is nothing in the 
catalina logs of interest and my logging level is currently set to trace for 
the management-server.log.  I will turn on the sql query log and compare them 
between startups of the management nodes perhaps that will yeild something.  I 
have included a dropbox link with the management server log from a normal 
startup of CS and one with mysql binlog enabled.  Any help would be greatly 
appriciated. 


https://dl.dropboxusercontent.com/u/51895629/management-server.logs.tar.gz



Steve Searles


-Original Message-
From: Rafael Weingartner [mailto:rafaelweingart...@gmail.com]
Sent: Monday, June 9, 2014 6:57 PM
To: users@cloudstack.apache.org
Subject: Re: Management Server Startup Issue

please give more logs.
It is the 4.3.0 right? I bet there is a problem with a missing dependency.


On Mon, Jun 9, 2014 at 7:32 PM, Steve Searles ssear...@zimcom.net wrote:

 Anyone have any idea why the management service would refuse to start 
 if the DB server is configured as a master for replication?



 log_bin=mysql-bin
 server_id=1

 Adding the above the my.cnf file of my DB server, restarting mysql 
 then restarting cloudstack-managment will cause cloudstack to get hung 
 on startup here.

 2014-06-09 18:25:33,769 INFO  [c.c.u.c.ComponentContext] (main:null) 
 Starting
 com.cloud.bridge.persist.dao.SHostDaoImpl_EnhancerByCloudStack_9da5499
 b
 2014-06-09 18:25:33,769 INFO  [c.c.u.c.ComponentContext] (main:null) 
 Starting 
 com.cloud.bridge.persist.dao.SObjectDaoImpl_EnhancerByCloudStack_847fb
 98c
 2014-06-09 18:25:33,769 INFO  [c.c.u.c.ComponentContext] (main:null) 
 Starting 
 com.cloud.bridge.persist.dao.CloudStackUserDaoImpl_EnhancerByCloudStac
 k_7fce3ee6
 2014-06-09 18:25:33,769 INFO  [c.c.u.c.ComponentContext] (main:null) 
 Starting
 com.cloud.bridge.service.core.ec2.EC2Engine_EnhancerByCloudStack_d70c9
 e3c
 2014-06-09 18:25:33,769 INFO  [c.c.u.c.ComponentContext] (main:null) 
 Starting 
 com.cloud.bridge.service.controller.s3.ServiceProvider_EnhancerByCloud
 Stack_23d38b1

 That is the last few lines of the management-server.log and it will 
 hang there indefinitely.

 I noticed this on startup as well.

 2014-06-09 18:25:16,841 INFO  [c.c.u.d.T.Transaction] (main:null) Is 
 Data Base High Availiability enabled? Ans : false
 2014-06-09 18:25:17,164 DEBUG [c.c.u.d.ConnectionConcierge]
 (main:null) Registering a database connection for LockMaster1
 2014-06-09 18:25:17,164 INFO  [c.c.u.d.Merovingian2] (main:null) 
 Cleaning up locks for 345049793560
 2014-06-09 18:25:17,175 INFO  [c.c.u.d.Merovingian2] (main:null) 
 Released
 0 locks for 345049793560
 2014-06-09 18:25:17,206 INFO  [o.a.c.s.l.CloudStackExtendedLifeCycle]
 (main:null) Running system integrity checker 
 com.cloud.upgrade.DatabaseUpgradeChecker@63c8e4ae
 2014-06-09 18:25:17,207 INFO  [c.c.u.DatabaseUpgradeChecker]
 (main:null) Grabbing lock to check for database upgrade.
 2014-06-09 18:25:17,254 DEBUG [c.c.u.d.VersionDaoImpl] (main:null) 
 Checking to see if the database is at a version before it was the 
 version table is created
 2014-06-09 18:25:17,295 INFO  [c.c.u.DatabaseUpgradeChecker]
 (main:null) DB version = 4.3.0 Code Version = 4.3.0
 2014-06-09 18:25:17,296 INFO  [c.c.u.DatabaseUpgradeChecker]
 (main:null) DB version and code version matches so no upgrade needed.

 The management server talking to the DB server at least enough to 
 check the version; and mysql from the command line on the management 
 server works fine as well.


 This line caught my attention.


 2014-06-09 18:25:16,841 INFO  [c.c.u.d.T.Transaction] (main:null) Is 
 Data Base High Availiability enabled? Ans : false

 Is there something I need to set when using mysql replication specific 
 to cloudstack? I do not see anything in the documentation on this 
 other than setting up basic replication.


 if I comment out

Vmware 5.5 dvSwitch support.

[Steve Searles]

Anyone know why this may be happening? vmware.use.dvswitch is set to true in 
global settings. The management server has been restarted.


The network label I am using is dvSwitch-test,602,vmwaredvs Using a standard 
vSwitch works fine. e.g. vSwitch1,602,vmwaresvs  Any ideas?

2014-04-03 12:39:45,313 DEBUG [c.c.h.v.r.VmwareResource] 
(DirectAgent-391:ctx-ed1b98f6 
esx02.cs.zimcom.nethttp://esx02.cs.zimcom.nethttp://esx02.cs.zimcom.net) VM 
i-2-40-VM will be started with NIC device type: E1000
2014-04-03 12:39:45,313 INFO  [c.c.h.v.r.VmwareResource] 
(DirectAgent-391:ctx-ed1b98f6 
esx02.cs.zimcom.nethttp://esx02.cs.zimcom.nethttp://esx02.cs.zimcom.net) 
Prepare NIC device based on NicTO: 
{deviceId:0,networkRateMbps:200,defaultNic:true,uuid:9ef9f0ff-70e4-4851-8030-edef22aa61df,ip:172.17.2.95,netmask:255.255.255.0,gateway:172.17.2.1,mac:02:00:5a:92:00:0d,dns1:209.212.245.10,dns2:209.212.246.10,broadcastType:Vlan,type:Guest,broadcastUri:vlan://610,isolationUri:vlan://610,isSecurityGroupEnabled:false,name:dvSwitch-test,602,vmwaredvs}
2014-04-03 12:39:45,318 INFO  [c.c.h.v.r.VmwareResource] 
(DirectAgent-391:ctx-ed1b98f6 
esx02.cs.zimcom.nethttp://esx02.cs.zimcom.nethttp://esx02.cs.zimcom.net) 
Prepare network on vmwaredvs dvSwitch-test with name prefix: cloud.guest
2014-04-03 12:39:45,325 WARN  [c.c.h.v.r.VmwareResource] 
(DirectAgent-391:ctx-ed1b98f6 
esx02.cs.zimcom.nethttp://esx02.cs.zimcom.nethttp://esx02.cs.zimcom.net) 
StartCommand failed due to Exception: java.lang.NumberFormatException
Message: null

java.lang.NumberFormatException: null
at java.lang.Integer.parseInt(Integer.java:443)
at java.lang.Integer.parseInt(Integer.java:514)
at 
com.cloud.hypervisor.vmware.mo.HypervisorHostHelper.prepareNetwork(HypervisorHostHelper.java:504)
at 
com.cloud.hypervisor.vmware.resource.VmwareResource.prepareNetworkFromNicInfo(VmwareResource.java:3692)
at 
com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:2934)
at 
com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:525)
at 
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216)
at 
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at 
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)



vmware 5.5
dvswitch 5.5
centos 6.4 non-oss rpm build.



Steven Searles, CTO | ssear...@zimcom.netmailto:ssear...@zimcom.net
Zimcom Internet Solutions  | www.zimcom.nethttp://www.zimcom.net
O: 513.231.9500  |  D: 513.233.4130

[cid:image001.jpg@01CF54D3.D25E1ED0]

Re: Trouble with Usage server on 4.3.

[Steve Searles]

Anyone?

Steven Searles
Zimcom Internet


On Apr 4, 2014, at 6:04 PM, Steve Searles 
ssear...@zimcom.netmailto:ssear...@zimcom.net wrote:

On 4.3 I am having a problem getting my usage server to start.   CS works fine 
and db.properties appears to be correct.  The cloudstack_usage DB has been 
created and credentials work.  I am getting the following in the 
cloudstack-usage.err log

com.cloud.utils.exception.CloudRuntimeException: DB Exception on: null
at 
com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(GenericDaoBase.java:421)
at 
org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl_EnhancerByCloudStack_964f5759.CGLIB$searchIncludingRemoved$48(generated)
at 
org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl_EnhancerByCloudStack_964f5759_FastClassByCloudStack_bb80d0f6.invoke(generated)
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at 
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at 
org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl_EnhancerByCloudStack_964f5759.searchIncludingRemoved(generated)
at 
com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(GenericDaoBase.java:356)
at 
org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl_EnhancerByCloudStack_964f5759.CGLIB$searchIncludingRemoved$49(generated)
at 
org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl_EnhancerByCloudStack_964f5759_FastClassByCloudStack_bb80d0f6.invoke(generated)
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at 
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at 
org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl_EnhancerByCloudStack_964f5759.searchIncludingRemoved(generated)
at 
com.cloud.utils.db.GenericDaoBase.listIncludingRemovedBy(GenericDaoBase.java:911)
at 
org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl_EnhancerByCloudStack_964f5759.CGLIB$listIncludingRemovedBy$78(generated)
at 
org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl_EnhancerByCloudStack_964f5759_FastClassByCloudStack_bb80d0f6.invoke(generated)
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at 
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at 
org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl_EnhancerByCloudStack_964f5759.listIncludingRemovedBy(generated)
at 
com.cloud.utils.db.GenericDaoBase.listIncludingRemovedBy(GenericDaoBase.java:916)
at 
org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl_EnhancerByCloudStack_964f5759.CGLIB$listIncludingRemovedBy$76(generated)
at 
org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl_EnhancerByCloudStack_964f5759_FastClassByCloudStack_bb80d0f6.invoke(generated)
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at 
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at 
org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl_EnhancerByCloudStack_964f5759.listIncludingRemovedBy(generated)
at 
org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl.getConfiguration(ConfigurationDaoImpl.java:82)
at 
org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl_EnhancerByCloudStack_964f5759.CGLIB$getConfiguration$8(generated)
at 
org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl_EnhancerByCloudStack_964f5759_FastClassByCloudStack_bb80d0f6.invoke(generated)
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at 
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at 
org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl_EnhancerByCloudStack_964f5759.getConfiguration(generated)
at 
org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl.getConfiguration(ConfigurationDaoImpl.java:110)
at 
org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl_EnhancerByCloudStack_964f5759.CGLIB$getConfiguration$9(generated)
at 
org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl_EnhancerByCloudStack_964f5759_FastClassByCloudStack_bb80d0f6.invoke(generated)
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at 
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at 
org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl_EnhancerByCloudStack_964f5759.getConfiguration(generated)
at com.cloud.usage.UsageManagerImpl.configure(UsageManagerImpl.java:173)
at 
com.cloud.utils.component.ComponentContext.initComponentsLifeCycle(ComponentContext.java:114

Trouble with Usage server on 4.3.

[Steve Searles]

On 4.3 I am having a problem getting my usage server to start.   CS works fine 
and db.properties appears to be correct.  The cloudstack_usage DB has been 
created and credentials work.  I am getting the following in the 
cloudstack-usage.err log

com.cloud.utils.exception.CloudRuntimeException: DB Exception on: null
at 
com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(GenericDaoBase.java:421)
at 
org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl_EnhancerByCloudStack_964f5759.CGLIB$searchIncludingRemoved$48(generated)
at 
org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl_EnhancerByCloudStack_964f5759_FastClassByCloudStack_bb80d0f6.invoke(generated)
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at 
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at 
org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl_EnhancerByCloudStack_964f5759.searchIncludingRemoved(generated)
at 
com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(GenericDaoBase.java:356)
at 
org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl_EnhancerByCloudStack_964f5759.CGLIB$searchIncludingRemoved$49(generated)
at 
org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl_EnhancerByCloudStack_964f5759_FastClassByCloudStack_bb80d0f6.invoke(generated)
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at 
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at 
org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl_EnhancerByCloudStack_964f5759.searchIncludingRemoved(generated)
at 
com.cloud.utils.db.GenericDaoBase.listIncludingRemovedBy(GenericDaoBase.java:911)
at 
org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl_EnhancerByCloudStack_964f5759.CGLIB$listIncludingRemovedBy$78(generated)
at 
org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl_EnhancerByCloudStack_964f5759_FastClassByCloudStack_bb80d0f6.invoke(generated)
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at 
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at 
org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl_EnhancerByCloudStack_964f5759.listIncludingRemovedBy(generated)
at 
com.cloud.utils.db.GenericDaoBase.listIncludingRemovedBy(GenericDaoBase.java:916)
at 
org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl_EnhancerByCloudStack_964f5759.CGLIB$listIncludingRemovedBy$76(generated)
at 
org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl_EnhancerByCloudStack_964f5759_FastClassByCloudStack_bb80d0f6.invoke(generated)
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at 
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at 
org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl_EnhancerByCloudStack_964f5759.listIncludingRemovedBy(generated)
at 
org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl.getConfiguration(ConfigurationDaoImpl.java:82)
at 
org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl_EnhancerByCloudStack_964f5759.CGLIB$getConfiguration$8(generated)
at 
org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl_EnhancerByCloudStack_964f5759_FastClassByCloudStack_bb80d0f6.invoke(generated)
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at 
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at 
org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl_EnhancerByCloudStack_964f5759.getConfiguration(generated)
at 
org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl.getConfiguration(ConfigurationDaoImpl.java:110)
at 
org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl_EnhancerByCloudStack_964f5759.CGLIB$getConfiguration$9(generated)
at 
org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl_EnhancerByCloudStack_964f5759_FastClassByCloudStack_bb80d0f6.invoke(generated)
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at 
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at 
org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl_EnhancerByCloudStack_964f5759.getConfiguration(generated)
at com.cloud.usage.UsageManagerImpl.configure(UsageManagerImpl.java:173)
at 
com.cloud.utils.component.ComponentContext.initComponentsLifeCycle(ComponentContext.java:114)
at com.cloud.usage.UsageServer.start(UsageServer.java:60)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 

Vmware dvSwitch issue in 4.3

[Steve Searles]

I am having an issue with the vmware dvSwitch support in 4.3.  The same 
configuration was working properly in 4.2.1.  I did a fresh install of 4.3 in 
the lab and am seeing the java.lang.NumberFormatException: null error when 
trying to provision an instance. The network label I am using is 
dvSwitch-test,602,vmwaredvs Using a standard vSwitch works fine.  Any ideas?

2014-04-03 12:39:45,313 DEBUG [c.c.h.v.r.VmwareResource] 
(DirectAgent-391:ctx-ed1b98f6 esx02.cs.zimcom.nethttp://esx02.cs.zimcom.net) 
VM i-2-40-VM will be started with NIC device type: E1000
2014-04-03 12:39:45,313 INFO  [c.c.h.v.r.VmwareResource] 
(DirectAgent-391:ctx-ed1b98f6 esx02.cs.zimcom.nethttp://esx02.cs.zimcom.net) 
Prepare NIC device based on NicTO: 
{deviceId:0,networkRateMbps:200,defaultNic:true,uuid:9ef9f0ff-70e4-4851-8030-edef22aa61df,ip:172.17.2.95,netmask:255.255.255.0,gateway:172.17.2.1,mac:02:00:5a:92:00:0d,dns1:209.212.245.10,dns2:209.212.246.10,broadcastType:Vlan,type:Guest,broadcastUri:vlan://610,isolationUri:vlan://610,isSecurityGroupEnabled:false,name:dvSwitch-test,602,vmwaredvs}
2014-04-03 12:39:45,318 INFO  [c.c.h.v.r.VmwareResource] 
(DirectAgent-391:ctx-ed1b98f6 esx02.cs.zimcom.nethttp://esx02.cs.zimcom.net) 
Prepare network on vmwaredvs dvSwitch-test with name prefix: cloud.guest
2014-04-03 12:39:45,325 WARN  [c.c.h.v.r.VmwareResource] 
(DirectAgent-391:ctx-ed1b98f6 esx02.cs.zimcom.nethttp://esx02.cs.zimcom.net) 
StartCommand failed due to Exception: java.lang.NumberFormatException
Message: null

java.lang.NumberFormatException: null
at java.lang.Integer.parseInt(Integer.java:443)
at java.lang.Integer.parseInt(Integer.java:514)
at 
com.cloud.hypervisor.vmware.mo.HypervisorHostHelper.prepareNetwork(HypervisorHostHelper.java:504)
at 
com.cloud.hypervisor.vmware.resource.VmwareResource.prepareNetworkFromNicInfo(VmwareResource.java:3692)
at 
com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:2934)
at 
com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:525)
at 
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216)
at 
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at 
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)




Steven Searles
Zimcom Internet
http://www.zimcom.nethttp://www.zimcom.net/
Phone. (513)231-9500
Fax. (513)624-3909
Toll Free. (888)624-3910

Re: Deploy Template Error READ TIMEOUT

[Steve Searles]

Any help would be greatly appreciated. I'm kinda dead in the water here guys 
trying to do anything other than deploy a small template. Should I ask this on 
the dev list?

Sent from my iPhone

 On Nov 22, 2013, at 2:48 PM, Steve Searles ssear...@zimcom.net wrote:
 
 Any ideas anyone?
 
 Steve Searles
 
 
 On Nov 21, 2013, at 6:02 PM, Steve Searles 
 ssear...@zimcom.netmailto:ssear...@zimcom.net wrote:
 
 I found this open issue for 4.2.1.  It seems to be effecting my 4.2 install 
 as well.
 
 https://issues.apache.org/jira/browse/CLOUDSTACK-5014
 
 I am seeing this issue as well.  It seems that the timeout occurs on any copy 
 from secondary storage that exceeds 10 Minutes on vmware 5.1. This seems to 
 affect only templates and images that are large enough not to be copied from 
 secondary storage in that window.  Small templates like the included CentOS 
 and some home-brew micro ubuntu templates work fine.  I was unable to locate 
 any global setting that is set to 10 minutes, the closest thing I could find 
 was secstorage.cmd.execution.time.max which I set to 240 minutes in my 
 environment.  I was able to mitigate some of this by using Fibrechannel as 
 the target, it seemed like my template size was then limited what my 1GBE on 
 secondary storage could transfer in 10 minutes. The deploy fails in CS but 
 the Clone VM operation continues to run on the vmware side and does 
 eventually complete leaving me with an orphaned ROOT-XXX vm on the vmware 
 side.  It seems that the solution would be to set that hard timeout to a 
 value greater than 10 minutes but I can not locate it anywhere. The 
 timestamps seem to be the smoking gun.  Any suggestions would be appreciated.
 
 
 Begin Template Deploy
 
 172.23.12.79
 userdata,user-data
 none
 metadata,service-offering
 Micro Instance
 metadata,availability-zone
 US_EAST_LEB_Z01
 metadata,local-ipv4
 172.23.12.79
 metadata,local-hostname
 MSWOAP200106
 metadata,public-ipv4
 209.212.253.138
 metadata,public-hostname
 209.212.253.138
 metadata,instance-id
 bcb8fd42-5fe9-46be-b77f-13daebfd9c35
 metadata,vm-id
 bcb8fd42-5fe9-46be-b77f-13daebfd9c35
 metadata,public-keys
 none
 metadata,cloud-identifier
 CloudStack-{bc067003-923b-453e-ad37-5e06c2670966}
 
 2013-11-21 15:41:14,957 INFO  [vmware.resource.VmwareResource] 
 (DirectAgent-8:esx01.lab.zimcom.nethttp://esx01.lab.zimcom.net) vm_data 
 command on domain router 172.23.5.119 completed
 2013-11-21 15:41:14,958 DEBUG [agent.manager.DirectAgentAttache] 
 (DirectAgent-8:null) Seq 15-1363607559: Response Received:
 2013-11-21 15:41:14,958 DEBUG [agent.transport.Request] (DirectAgent-8:null) 
 Seq 15-1363607559: Processing:  { Ans: , MgmtId: 345049822399, via: 15, Ver: 
 v1, Flags: 110, 
 [{com.cloud.agent.api.Answer:{result:true,wait:0}},{com.cloud.agent.api.Answer:{result:true,wait:0}}]
  }
 2013-11-21 15:41:14,958 DEBUG [agent.manager.AgentAttache] 
 (DirectAgent-8:null) Seq 15-1363607559: No more commands found
 2013-11-21 15:41:14,958 DEBUG [agent.transport.Request] 
 (Job-Executor-1:job-652 = [ 9bad7d26-5de2-4d97-85de-1241cc28fc16 ]) Seq 
 15-1363607559: Received:  { Ans: , MgmtId: 345049822399, via: 15, Ver: v1, 
 Flags: 110, { Answer, Answer } }
 2013-11-21 15:41:14,960 DEBUG [cloud.network.NetworkModelImpl] 
 (Job-Executor-1:job-652 = [ 9bad7d26-5de2-4d97-85de-1241cc28fc16 ]) Service 
 SecurityGroup is not supported in the network id=215
 2013-11-21 15:41:14,962 DEBUG [cloud.storage.VolumeManagerImpl] 
 (Job-Executor-1:job-652 = [ 9bad7d26-5de2-4d97-85de-1241cc28fc16 ]) Checking 
 if we need to prepare 1 volumes for VM[User|MSWOAP200106]
 2013-11-21 15:41:14,988 DEBUG [storage.image.TemplateDataFactoryImpl] 
 (Job-Executor-1:job-652 = [ 9bad7d26-5de2-4d97-85de-1241cc28fc16 ]) template 
 255 is already in store:1, type:Image
 2013-11-21 15:41:15,001 DEBUG [storage.image.TemplateDataFactoryImpl] 
 (Job-Executor-1:job-652 = [ 9bad7d26-5de2-4d97-85de-1241cc28fc16 ]) template 
 255 is already in store:2, type:Primary
 2013-11-21 15:41:15,038 DEBUG [storage.motion.AncientDataMotionStrategy] 
 (Job-Executor-1:job-652 = [ 9bad7d26-5de2-4d97-85de-1241cc28fc16 ]) copyAsync 
 inspecting src type TEMPLATE copyAsync inspecting dest type VOLUME
 2013-11-21 15:41:15,056 DEBUG [agent.transport.Request] 
 (Job-Executor-1:job-652 = [ 9bad7d26-5de2-4d97-85de-1241cc28fc16 ]) Seq 
 15-1363607560: Sending  { Cmd , MgmtId: 345049822399, via: 15, Ver: v1, 
 Flags: 100111, 
 [{org.apache.cloudstack.storage.command.CopyCommand:{srcTO:{org.apache.cloudstack.storage.to.TemplateObjectTO:{path:9c1b905e33d59385333fed5c973d
 
 
 --- End Exception---
 
 2013-11-21 15:51:15,228 ERROR [storage.resource.VmwareStorageProcessor] 
 (DirectAgent-11:esx01.lab.zimcom.nethttp://esx01.lab.zimcom.net) clone 
 volume from base image failed due to Exception: 
 javax.xml.ws.WebServiceException
 Message: java.net.SocketTimeoutException: Read timed out
 
 javax.xml.ws.WebServiceException: java.net.SocketTimeoutException: Read timed 
 out

Re: Deploy Template Error READ TIMEOUT

[Steve Searles]

Any ideas anyone?

Steve Searles


On Nov 21, 2013, at 6:02 PM, Steve Searles 
ssear...@zimcom.netmailto:ssear...@zimcom.net wrote:

I found this open issue for 4.2.1.  It seems to be effecting my 4.2 install as 
well.

https://issues.apache.org/jira/browse/CLOUDSTACK-5014

I am seeing this issue as well.  It seems that the timeout occurs on any copy 
from secondary storage that exceeds 10 Minutes on vmware 5.1. This seems to 
affect only templates and images that are large enough not to be copied from 
secondary storage in that window.  Small templates like the included CentOS and 
some home-brew micro ubuntu templates work fine.  I was unable to locate any 
global setting that is set to 10 minutes, the closest thing I could find was 
secstorage.cmd.execution.time.max which I set to 240 minutes in my environment. 
 I was able to mitigate some of this by using Fibrechannel as the target, it 
seemed like my template size was then limited what my 1GBE on secondary storage 
could transfer in 10 minutes. The deploy fails in CS but the Clone VM operation 
continues to run on the vmware side and does eventually complete leaving me 
with an orphaned ROOT-XXX vm on the vmware side.  It seems that the solution 
would be to set that hard timeout to a value greater than 10 minutes but I can 
not locate it anywhere. The timestamps seem to be the smoking gun.  Any 
suggestions would be appreciated.


Begin Template Deploy

172.23.12.79
userdata,user-data
none
metadata,service-offering
Micro Instance
metadata,availability-zone
US_EAST_LEB_Z01
metadata,local-ipv4
172.23.12.79
metadata,local-hostname
MSWOAP200106
metadata,public-ipv4
209.212.253.138
metadata,public-hostname
209.212.253.138
metadata,instance-id
bcb8fd42-5fe9-46be-b77f-13daebfd9c35
metadata,vm-id
bcb8fd42-5fe9-46be-b77f-13daebfd9c35
metadata,public-keys
none
metadata,cloud-identifier
CloudStack-{bc067003-923b-453e-ad37-5e06c2670966}

2013-11-21 15:41:14,957 INFO  [vmware.resource.VmwareResource] 
(DirectAgent-8:esx01.lab.zimcom.nethttp://esx01.lab.zimcom.net) vm_data 
command on domain router 172.23.5.119 completed
2013-11-21 15:41:14,958 DEBUG [agent.manager.DirectAgentAttache] 
(DirectAgent-8:null) Seq 15-1363607559: Response Received:
2013-11-21 15:41:14,958 DEBUG [agent.transport.Request] (DirectAgent-8:null) 
Seq 15-1363607559: Processing:  { Ans: , MgmtId: 345049822399, via: 15, Ver: 
v1, Flags: 110, 
[{com.cloud.agent.api.Answer:{result:true,wait:0}},{com.cloud.agent.api.Answer:{result:true,wait:0}}]
 }
2013-11-21 15:41:14,958 DEBUG [agent.manager.AgentAttache] (DirectAgent-8:null) 
Seq 15-1363607559: No more commands found
2013-11-21 15:41:14,958 DEBUG [agent.transport.Request] (Job-Executor-1:job-652 
= [ 9bad7d26-5de2-4d97-85de-1241cc28fc16 ]) Seq 15-1363607559: Received:  { 
Ans: , MgmtId: 345049822399, via: 15, Ver: v1, Flags: 110, { Answer, Answer } }
2013-11-21 15:41:14,960 DEBUG [cloud.network.NetworkModelImpl] 
(Job-Executor-1:job-652 = [ 9bad7d26-5de2-4d97-85de-1241cc28fc16 ]) Service 
SecurityGroup is not supported in the network id=215
2013-11-21 15:41:14,962 DEBUG [cloud.storage.VolumeManagerImpl] 
(Job-Executor-1:job-652 = [ 9bad7d26-5de2-4d97-85de-1241cc28fc16 ]) Checking if 
we need to prepare 1 volumes for VM[User|MSWOAP200106]
2013-11-21 15:41:14,988 DEBUG [storage.image.TemplateDataFactoryImpl] 
(Job-Executor-1:job-652 = [ 9bad7d26-5de2-4d97-85de-1241cc28fc16 ]) template 
255 is already in store:1, type:Image
2013-11-21 15:41:15,001 DEBUG [storage.image.TemplateDataFactoryImpl] 
(Job-Executor-1:job-652 = [ 9bad7d26-5de2-4d97-85de-1241cc28fc16 ]) template 
255 is already in store:2, type:Primary
2013-11-21 15:41:15,038 DEBUG [storage.motion.AncientDataMotionStrategy] 
(Job-Executor-1:job-652 = [ 9bad7d26-5de2-4d97-85de-1241cc28fc16 ]) copyAsync 
inspecting src type TEMPLATE copyAsync inspecting dest type VOLUME
2013-11-21 15:41:15,056 DEBUG [agent.transport.Request] (Job-Executor-1:job-652 
= [ 9bad7d26-5de2-4d97-85de-1241cc28fc16 ]) Seq 15-1363607560: Sending  { Cmd , 
MgmtId: 345049822399, via: 15, Ver: v1, Flags: 100111, 
[{org.apache.cloudstack.storage.command.CopyCommand:{srcTO:{org.apache.cloudstack.storage.to.TemplateObjectTO:{path:9c1b905e33d59385333fed5c973d


--- End Exception---

2013-11-21 15:51:15,228 ERROR [storage.resource.VmwareStorageProcessor] 
(DirectAgent-11:esx01.lab.zimcom.nethttp://esx01.lab.zimcom.net) clone volume 
from base image failed due to Exception: javax.xml.ws.WebServiceException
Message: java.net.SocketTimeoutException: Read timed out

javax.xml.ws.WebServiceException: java.net.SocketTimeoutException: Read timed 
out
at 
com.sun.xml.internal.ws.transport.http.client.HttpClientTransport.readResponseCodeAndMessage(HttpClientTransport.java:201)
at 
com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.process(HttpTransportPipe.java:151)
at 
com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.processRequest

Deploy Template Error READ TIMEOUT

[Steve Searles]

:78)

at 
com.sun.xml.internal.ws.api.pipe.Fiber.__doRun(Fiber.java:587)

at com.sun.xml.internal.ws.api.pipe.Fiber._doRun(Fiber.java:546)

at com.sun.xml.internal.ws.api.pipe.Fiber.doRun(Fiber.java:531)

at 
com.sun.xml.internal.ws.api.pipe.Fiber.runSync(Fiber.java:428)

at com.sun.xml.internal.ws.client.Stub.process(Stub.java:211)

at 
com.sun.xml.internal.ws.client.sei.SEIStub.doProcess(SEIStub.java:124)









Steve Searles

Compute offerings and Templates

[Steve Searles]

I am looking for a little guidance in tying templates to compute offerings or 
filtering them in the instance deployment wizard. For example as we step 
through the wizard we can choose Zone - Template - Compute Offering, in our 
case it is possible to create an undeployable situation by choosing a compute 
offering where a host tag is defined but the template selected does not support 
the hypervisor in the host tag.

For example,  I have both xenserver and vmware.  I have defined a compute 
offering with a storage tag for Fibrechannel storage that is not available on 
xenserver but you can still choose that compute offering after selecting a 
xenserver template which will make an undeployable situation.  Perhaps I am not 
using host/storage tags properly but they seem to be functioning as intended.  
I am trying to make the deployment process as simple as possible for the end 
user and not allow them to choose incompatible offerings.  I fully admit that I 
may be looking at this wrong and am looking for some guidance on setting up the 
offerings.  I have the following defined and it seems kind of a mess.


Vmware-
Storage Offering #1 - EMC VMAX (High I/O)
Storage Offering #2 - EMX VNX (Standard I/O)
Storage Offering #3 -  EMC ISOLON (NFS/Replicated)

XenServer Storage Offering #1 - EMC AX (Standard I/O)
  Storage Offering #2 -  EMC ISOLON (NFS/Replicated)



Steve Searles

RE: Site to Site and RA VPN?

[Steve Searles]

Anyone? 

Steve Searles
Director, Network Operations
Zimcom Internet Solutions, Inc.
http://www.zimcom.net
Phone. (513)231-9500
Fax. (513)624-3909
Toll Free. (888)624-3910 

-Original Message-
From: Steve Searles [mailto:ssear...@zimcom.net] 
Sent: Friday, November 8, 2013 11:24 PM
To: users@cloudstack.apache.org
Subject: Site to Site and RA VPN?

In CS 4.2 remote access VPN's are not supported inside a VPC; likewise site to 
site is not supported in a non VPC isolated network. (Advanced Zone).  What is 
the best practice/suggestion if both are required? A small Linux instance 
inside the VPC to serve as a vpn endpoint for the remote users?  I am looking 
for what has worked well for others in this situation.



Steve Searles
Director, Network Operations
Zimcom Internet Solutions, Inc.
http://www.zimcom.net
Phone. (513)231-9500
Fax. (513)624-3909
Toll Free. (888)624-3910

Error Creating Template From Instance

[Steve Searles]

I have a problem that developed when creating some new templates for vSphere.  
When using the create template function it kicks off the Export OVF and hangs 
In Progress in vcenter.  The vpxa.log on the host has the following, anyone 
know what may be causing this? I have tried restarting the management servers, 
the host and the ssvm. 

2013-11-13T00:16:04.570Z [FFB7B6D0 info 'Libs' opID=SWI-a3d84e8d] OBJLIB-FILEBE 
: FileBEIoctl: ioctl operation failed on 
'/vmfs/volumes/ec1e6239-8a1a0a26/i-2-206-VM/5d862114e7b144b79473f32c75043362-flat.vmdk'
 : Inappropriate ioctl for device (1638402)


Steve Searles

Re: Error Creating Template From Instance

[Steve Searles]

Additional Log Line 

2013-11-13T00:20:29.202Z [FFB7B6D0 info 'Libs' opID=SWI-a3d84e8d] OBJLIB-FILEBE 
: FileBEIoctl: ioctl operation failed on 
'/vmfs/volumes/ec1e6239-8a1a0a26/i-2-206-VM/5d862114e7b144b79473f32c75043362-flat.vmdk'
 : Inappropriate ioctl for device (1638402)
2013-11-13T00:20:38.316Z [FFB7B6D0 info 'Libs' opID=SWI-a3d84e8d] OBJLIB-FILEBE 
: FileBEIoctl: ioctl operation failed on 
'/vmfs/volumes/ec1e6239-8a1a0a26/i-2-206-VM/5d862114e7b144b79473f32c75043362-flat.vmdk'
 : Inappropriate ioctl for device (1638402)
2013-11-13T00:20:38.316Z [FFB7B6D0 info 'Default' opID=SWI-a3d84e8d] 
[STREAMVMDK] Emitting progress grain. Time elapsed: 19435439, progress made: 1%

Steve Searles
Director, Network Operations
Zimcom Internet Solutions, Inc.
http://www.zimcom.net
Phone. (513)231-9500
Fax. (513)624-3909
Toll Free. (888)624-3910

On Nov 12, 2013, at 7:20 PM, Steve Searles ssear...@zimcom.net
 wrote:

 I have a problem that developed when creating some new templates for vSphere. 
  When using the create template function it kicks off the Export OVF and 
 hangs In Progress in vcenter.  The vpxa.log on the host has the following, 
 anyone know what may be causing this? I have tried restarting the management 
 servers, the host and the ssvm. 
 
 2013-11-13T00:16:04.570Z [FFB7B6D0 info 'Libs' opID=SWI-a3d84e8d] 
 OBJLIB-FILEBE : FileBEIoctl: ioctl operation failed on 
 '/vmfs/volumes/ec1e6239-8a1a0a26/i-2-206-VM/5d862114e7b144b79473f32c75043362-flat.vmdk'
  : Inappropriate ioctl for device (1638402)
 
 
 Steve Searles
 
 

Site to Site and RA VPN?

[Steve Searles]

In CS 4.2 remote access VPN's are not supported inside a VPC; likewise site to 
site is not supported in a non VPC isolated network. (Advanced Zone).  What is 
the best practice/suggestion if both are required? A small Linux instance 
inside the VPC to serve as a vpn endpoint for the remote users?  I am looking 
for what has worked well for others in this situation.



Steve Searles
Director, Network Operations
Zimcom Internet Solutions, Inc.
http://www.zimcom.net
Phone. (513)231-9500
Fax. (513)624-3909
Toll Free. (888)624-3910

RE: cloudstack 4.2 xenserver 6.2 template not downloading

[Steve Searles]

I was having a similar issue last night. I have been running 4.1 without issue 
with both Xen and Vmware.  I installed 4.2 in the lab to do some testing before 
a full upgrade.  Below are the issues I have found so far and the workarounds I 
found. I am using Ubuntu 12.04 for the MS and building the nonoss code from 
source. 

1. System VM's will not provison. (NFS FAILURE) apparently after building the 
deb packages and installing them the correct permissions are not being set on 
/var/lib/cloud and the nfs mounts are failing due to to permissions for the 
cloud user.  (chown -R cloud.cloud /var/lib/cloud). This has corrected the 
issue. 

--Log Snip--
2013-10-03 23:22:38,931 INFO  [storage.resource.VmwareStorageProcessor] 
(DirectAgent-72:esx02.lab.zimcom.net) Executing 
copyTemplateFromSecondaryToPrimary. secondaryStorage: 
nfs://nfs01.us.east.leb.lab.zimcom.net/US_EAST_Z1_SEC, 
templatePathAtSecondaryStorage: template/tmpl/1/8/, templateName: routing-8
2013-10-03 23:22:38,931 ERROR [vmware.manager.VmwareManagerImpl] 
(DirectAgent-72:esx02.lab.zimcom.net) Unable to create mount: 
/var/lib/cloud/mnt/VM/345049822399.60d4f8d1
2013-10-03 23:22:38,932 ERROR [vmware.manager.VmwareManagerImpl] 
(DirectAgent-72:esx02.lab.zimcom.net) Unable to create mount: 
/var/lib/cloud/mnt/VM/345049822399.fe168a8
2013-10-03 23:22:38,932 ERROR
--Log Snip--

2. System VM's will provision and pass connectivity tests but templates will 
never begin downloading from cloud.com.  To make a long story short I 
pre-populated my secondary storage with all of the system images for 
KVM,VmWare, Xen, and LXC. The problem seems to be an issue with the SSVM 
enumerating the value of the LXC template from the secondary storage and 
returning Number Format Exception.  Below is a log I pulled from the SSVM 
running on vmware that caused the issue.  I removed the LXC template and the 
KVM template from the secondary storage and restarted the SSVM, the template 
download then succeeded. Below is a log snip from the SSVM that was causing the 
error.  Hope this helps. 

-- Log Snip-- 

2013-10-04 04:41:04,465 INFO  [storage.template.DownloadManagerImpl] 
(agentRequest-Handler-5:null) found 4 
templates[/mnt/SecStorage/4ebc642c-eb94-3a8a-81b8-d5971132beb3/template/tmpl/1/1/b3ac9ed6-cba1-41f3-
8965-f7c89f0772d8.vhd, 
/mnt/SecStorage/4ebc642c-eb94-3a8a-81b8-d5971132beb3/template/tmpl/1/8/79f0d6b5-22c8-4561-86d5-150260f4e2ce.ova,
 /mnt/SecStorage/4ebc642c-eb94-3a8a-81b8-d5971132beb3/template/tmpl/1/NU
LL/475e0e89-5bff-4e23-8f2c-ba9b56fc6472.qcow2, 
/mnt/SecStorage/4ebc642c-eb94-3a8a-81b8-d5971132beb3/template/tmpl/1/3/1220decc-ecdd-4794-8408-fc8836806f72.qcow2]
2013-10-04 04:41:04,477 WARN  [cloud.agent.Agent] (agentRequest-Handler-5:null) 
Caught: 
java.lang.NumberFormatException: For input string: NULL
at 
java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
at java.lang.Long.parseLong(Long.java:410)
at java.lang.Long.parseLong(Long.java:468)
at 
com.cloud.storage.template.TemplateLocation.getTemplateInfo(TemplateLocation.java:161)
at 
org.apache.cloudstack.storage.template.DownloadManagerImpl.gatherTemplateInfo(DownloadManagerImpl.java:840)
at 
org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.execute(NfsSecondaryStorageResource.java:1467)
at 
org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.executeRequest(NfsSecondaryStorageResource.java:194)
at 
com.cloud.storage.resource.PremiumSecondaryStorageResource.defaultAction(PremiumSecondaryStorageResource.java:63)
at 
com.cloud.storage.resource.PremiumSecondaryStorageResource.executeRequest(PremiumSecondaryStorageResource.java:59)
at com.cloud.agent.Agent.processRequest(Agent.java:525)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:852)
at com.cloud.utils.nio.Task.run(Task.java:83)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
2013-10-04 04:41:04,480 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:null) 
Seq 8-1237778437:  { Ans: , MgmtId: 345049822399, via: 8, Ver: v1, Flags: 110,

--Log Snip-- 


Steve Searles
Director, Network Operations
Zimcom Internet Solutions, Inc.
http://www.zimcom.net
Phone. (513)231-9500
Fax. (513)624-3909
Toll Free. (888)624-3910

-Original Message-
From: Shanker Balan [mailto:shanker.ba...@shapeblue.com] 
Sent: Friday, October 04, 2013 1:20 PM
To: users@cloudstack.apache.org
Subject: Re: cloudstack 4.2 xenserver 6.2 template not downloading

On 04-Oct-2013, at 10:05 PM, Dean Kamali dean.kam...@gmail.com wrote:

 Hi Shanker

 Are you using Ubuntu or CentOS to run CS Management station?

 It is very strange that system vms boots up correctly and check 
 scripts pass all tests.



Am on CentOS 6.4


 Please

Re: vSphere compatibility

[Steve Searles]

I have 5.1 running with CS in our lab with no issue. 

Steve Searles
Director, Network Operations
Zimcom Internet Solutions, Inc.
http://www.zimcom.net
Phone. (513)231-9500
Fax. (513)624-3909
Toll Free. (888)624-3910

On Aug 14, 2013, at 2:25 PM, John Skinner john.skin...@appcore.com
 wrote:

 Quick question regarding vSphere and vCenter. I have a customer that is 
 running version 5.1, however the documentation says 5.0 at the newest. Is 5.1 
 do-able? Has any one tested with 5.1 with any success? 
 
 Thanks! 
 

Re: Shared Network Issue HELP?!

[Steve Searles]

After posting this I thought about it more and answered my own question.  I see 
this type of setup being a management nightmare, e.g. rouge dhcp servers or 
anything else a user would attempt without the proper vlan/pvlan segmentation 
on the guest network in place it would be a hot mess.   Thanks Guys! 

Steve Searles
Zimcom Internet Solutions, Inc.
http://www.zimcom.net
Phone. (513)231-9500
Fax. (513)624-3909
Toll Free. (888)624-3910

On Aug 1, 2013, at 2:43 PM, Steve Searles st...@zimcom.net
 wrote:

 
 Hello everyone,  I am new to Cloudstack and have it deployed in our lab 
 currently so go easy :)  I have everything working as expected with the 
 advanced network zone, vmware support,  and multiple vlan isolation working 
 properly.  I have a few questions I would like to ask here about networking 
 options inside Cloudstack that I was unable to acertain from the 
 documentation. Hopefully one of you guys can steer me in the right direction.
 
 MY LAB SETUP
 Cloudstack 4.1.0 (Compiled from Source w/vmware support)
 Zone1- Advanced –
MGMT(untagged): 172.29.16.0/21
STORAGE(untagged): 172.29.16.0/21
GUEST(VLAN 601): 172.29.24.0/21
PUBLIC(VLAN 602): x.x.x.x/24
 
 Currently I create an account and can add a guest network, the virtual router 
 deploys properly assigns the necessary vlan for the isolated network  the 
 guest IP is assigned and a public ip is assigned from the public IP pool as 
 expected this works properly (Awesome).  Deploying a VPC under a user account 
 functions properly as well (Very Cool).  What I am looking for is a 
 configuration that I guess be best described as a shared isolated network? 
 Where an instance is provisioned and assigned an rfc1918 address from a large 
 guest pool much like I have currently setup with the “Default Shared 
 Network”. And a user can request an IP be assigned from the public pool and 
 create pat/nat translations and firewall rules just as on the isolated 
 network I tested.  This is for a multi-tennent setup where each user does not 
 need to create their own guest network and vlan isolation between accounts is 
 not necessary thus no need to burn a vlan and a vrouter for every customer 
 account.  Based on what I am seeing this functionality seems to already be 
 present.  If I provision and instance on the Default Shared Network (VLAN601) 
 in my setup, the machine is assigned the proper RFC1918 address from the pool 
 but when I try to allocate a public ip from the network tab I receive the 
 allocation error below.  Should this functionality even work? The problem 
 seems obvious but I don’t see where I can make the owners match even using 
 cloudmokey.
 
 2013-08-01 13:12:32,000 DEBUG [cloud.user.AccountManagerImpl] 
 (Job-Executor-21:job-269) Access to Acct[4-zimcom] granted to Acct[4-zimcom] 
 by DomainChecker_EnhancerByCloudStack_dd56169d
 2013-08-01 13:12:32,020 DEBUG [cloud.user.AccountManagerImpl] 
 (Job-Executor-21:job-269) Access to Ip[209.212.252.6-1] granted to 
 Acct[4-zimcom] by DomainChecker_EnhancerByCloudStack_dd56169d
 2013-08-01 13:12:32,031 DEBUG [cloud.user.AccountManagerImpl] 
 (Job-Executor-21:job-269) Access to Ntwk[204|Guest|7] granted to 
 Acct[4-zimcom] by DomainChecker_EnhancerByCloudStack_dd56169d
 2013-08-01 13:12:32,042 ERROR [cloud.async.AsyncJobManagerImpl] 
 (Job-Executor-21:job-269) Unexpected exception while executing 
 org.apache.cloudstack.api.command.user.address.AssociateIPAddrCmd
 com.cloud.exception.InvalidParameterValueException: The owner of the network 
 is not the same as owner of the IP
at 
 com.cloud.network.NetworkManagerImpl.associateIPToGuestNetwork(NetworkManagerImpl.java:744)
at 
 com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at 
 com.cloud.network.NetworkServiceImpl.associateIPToNetwork(NetworkServiceImpl.java:2852)
at 
 com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at 
 org.apache.cloudstack.api.command.user.address.AssociateIPAddrCmd.execute(AssociateIPAddrC
 
 
 
 
 
 I also created a second zone with basic networking but I did not see a way to 
 accomplish this with that setup either as it looks like the machine is 
 directly assigned a public address and access control is handled via SG’s 
 with ingress and egress filtering.
 
 Can anyone help me out?
 Thanks in Advance.
 
 Steve Searles
 http://www.zimcom.net
 Phone. (513)231-9500
 Fax. (513)624-3909
 Toll Free. (888)624-3910