I'm trying to figure out how CPU over-provisioning works in
cloudstack. It seems that no matter how much I change the
over-provisioning factor value in global settings, I cannot
over-provision CPU allocation on the hosts. I suspect that I'm just
having a lack of understanding on my part, but it seems like I should
be able to increase the number of VMs I can deploy by increasing the
CPU over-provisioning value (given that I'm not close to any other
resource limits). But this doesn't seem to be the case in practice.
https://screencloud.net/v/l1X7
https://screencloud.net/v/9ifx
Error message when trying to create additional machine via API.
grayson@gworkstation:~$ cloudmonkey deploy virtualmachine
zoneid=a4c20770-f17e-425e-bb21-413dd610b975
serviceofferingid=4b391121-49c7-4434-883d-ffe569f63a3b
templateid=e5701f73-b0c3-4dd7-8b2d-b8bf78875a85 keypair=key1
networkids=86e6ded0-6155-449b-a4bd-03411bf0a9cd name=tvmsman1
displayname &
[1] 3907
Async job 09148c21-3c42-4334-93b2-428b48fb1fa2 failed
Error 533, Unable to create a deployment for VM[User|i-15-1151-VM]
accountid = a4e6eaed-9b06-4806-98f9-4fcc781b773f
cmd = org.apache.cloudstack.api.command.user.vm.DeployVMCmd
created = 2015-04-17T13:46:22-0700
jobid = 09148c21-3c42-4334-93b2-428b48fb1fa2
jobprocstatus = 0
jobresult:
errorcode = 533
errortext = Unable to create a deployment for VM[User|i-15-1151-VM]
jobresultcode = 530
jobresulttype = object
jobstatus = 2
userid = c7a0889e-fcec-4095-9d07-d29ea3c1e5d9
(local) mycloudmonkey> list hosts
filter=name,clustername,cpuallocated,cpunumber,cpusockets,cpuspeed,cpuused,cpuwithoverprovisioning,
name = v-78-VM
================================================================================
name = s-871-VM
================================================================================
name = cl2hyper2
clustername = CL2
cpuallocated = 100.3%
cpunumber = 8
cpusockets = 2
cpuspeed = 1994
cpuused = 1.2%
cpuwithoverprovisioning = 63808.0
================================================================================
name = v-1-VM
================================================================================
name = cl2hyper1
clustername = CL2
cpuallocated = 81.45%
cpunumber = 8
cpusockets = 2
cpuspeed = 1995
cpuused = 0.6%
cpuwithoverprovisioning = 63840.0
================================================================================
2015-04-17 13:46:22,517 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(catalina-exec-25:ctx-7b0eb43a ctx-751b2eca ctx-5c8a75fc) submit async
job-7312, details: AsyncJobVO {id:7312, userId: 15, accountId: 15,
instanceType: VirtualMachine, instanceId: 1151, cmd:
org.apache.cloudstack.api.command.user.vm.DeployVMCmd, cmdInfo:
{"serviceofferingid":"4b391121-49c7-4434-883d-ffe569f63a3b","cmdEventType":"VM.CREATE","ctxUserId":"15","zoneid":"a4c20770-f17e-425e-bb21-413dd610b975","signatureversion":"3","httpmethod":"GET","templateid":"e5701f73-b0c3-4dd7-8b2d-b8bf78875a85","networkids":"86e6ded0-6155-449b-a4bd-03411bf0a9cd","apiKey":"wrxDdS8yi2kJPe8UrNdOdLtp55mCSKskk3MyTJ1ucIApdMoYnOWsd-dmigPywD1KC-6BVfVmKiJVenEM_kovpw","id":"1151","response":"json","ctxDetails":"{\"com.cloud.template.VirtualMachineTemplate\":\"e5701f73-b0c3-4dd7-8b2d-b8bf78875a85\",\"com.cloud.vm.VirtualMachine\":\"6b3ace85-8e4c-4467-afdf-b511136ca6b1\",\"com.cloud.network.Network\":\"86e6ded0-6155-449b-a4bd-03411bf0a9cd\",\"com.cloud.dc.DataCenter\":\"a4c20770-f17e-425e-bb21-413dd610b975\",\"com.cloud.offering.ServiceOffering\":\"4b391121-49c7-4434-883d-ffe569f63a3b\"}","expires":"2015-04-17T20:56:22+0000","keypair":"key1","name":"tvmsman1","uuid":"6b3ace85-8e4c-4467-afdf-b511136ca6b1","ctxAccountId":"15","ctxStartEventId":"11922","signature":"8Ebg1+qjfnk5nv7xz1GtdSju2ws\u003d"},
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
result: null, initMsid: 345050015385, completeMsid: null, lastUpdated:
null, lastPolled: null, created: null}
2015-04-17 13:46:22,518 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(API-Job-Executor-16:ctx-e039eccb job-7312) Add job-7312 into job
monitoring
2015-04-17 13:46:22,518 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-16:ctx-e039eccb job-7312) Executing AsyncJobVO
{id:7312, userId: 15, accountId: 15, instanceType: VirtualMachine,
instanceId: 1151, cmd:
org.apache.cloudstack.api.command.user.vm.DeployVMCmd, cmdInfo:
{"serviceofferingid":"4b391121-49c7-4434-883d-ffe569f63a3b","cmdEventType":"VM.CREATE","ctxUserId":"15","zoneid":"a4c20770-f17e-425e-bb21-413dd610b975","signatureversion":"3","httpmethod":"GET","templateid":"e5701f73-b0c3-4dd7-8b2d-b8bf78875a85","networkids":"86e6ded0-6155-449b-a4bd-03411bf0a9cd","apiKey":"wrxDdS8yi2kJPe8UrNdOdLtp55mCSKskk3MyTJ1ucIApdMoYnOWsd-dmigPywD1KC-6BVfVmKiJVenEM_kovpw","id":"1151","response":"json","ctxDetails":"{\"com.cloud.template.VirtualMachineTemplate\":\"e5701f73-b0c3-4dd7-8b2d-b8bf78875a85\",\"com.cloud.vm.VirtualMachine\":\"6b3ace85-8e4c-4467-afdf-b511136ca6b1\",\"com.cloud.network.Network\":\"86e6ded0-6155-449b-a4bd-03411bf0a9cd\",\"com.cloud.dc.DataCenter\":\"a4c20770-f17e-425e-bb21-413dd610b975\",\"com.cloud.offering.ServiceOffering\":\"4b391121-49c7-4434-883d-ffe569f63a3b\"}","expires":"2015-04-17T20:56:22+0000","keypair":"key1","name":"tvmsman1","uuid":"6b3ace85-8e4c-4467-afdf-b511136ca6b1","ctxAccountId":"15","ctxStartEventId":"11922","signature":"8Ebg1+qjfnk5nv7xz1GtdSju2ws\u003d"},
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
result: null, initMsid: 345050015385, completeMsid: null, lastUpdated:
null, lastPolled: null, created: null}
2015-04-17 13:46:22,519 DEBUG [c.c.a.ApiServlet]
(catalina-exec-25:ctx-7b0eb43a ctx-751b2eca ctx-5c8a75fc) ===END===
10.101.1.28 -- GET
networkids=86e6ded0-6155-449b-a4bd-03411bf0a9cd&apiKey=wrxDdS8yi2kJPe8UrNdOdLtp55mCSKskk3MyTJ1ucIApdMoYnOWsd-dmigPywD1KC-6BVfVmKiJVenEM_kovpw&name=tvmsman1&serviceofferingid=4b391121-49c7-4434-883d-ffe569f63a3b&expires=2015-04-17T20%3A56%3A22%2B0000&signatureversion=3&zoneid=a4c20770-f17e-425e-bb21-413dd610b975&keypair=key1&command=deployVirtualMachine&templateid=e5701f73-b0c3-4dd7-8b2d-b8bf78875a85&signature=8Ebg1%2Bqjfnk5nv7xz1GtdSju2ws%3D&response=json
2015-04-17 13:46:22,522 DEBUG [o.a.c.a.BaseCmd]
(API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Ignoring
paremeter displayvm as the caller is not authorized to pass it in
2015-04-17 13:46:22,522 DEBUG [o.a.c.a.BaseCmd]
(API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Ignoring
paremeter deploymentplanner as the caller is not authorized to pass it
in
2015-04-17 13:46:22,539 DEBUG [c.c.u.AccountManagerImpl]
(API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Access to
Acct[a4e6eaed-9b06-4806-98f9-4fcc781b773f-graysonp] granted to
Acct[a4e6eaed-9b06-4806-98f9-4fcc781b773f-graysonp] by DomainChecker
2015-04-17 13:46:22,552 DEBUG [c.c.u.AccountManagerImpl]
(API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Access to
Acct[a4e6eaed-9b06-4806-98f9-4fcc781b773f-graysonp] granted to
Acct[a4e6eaed-9b06-4806-98f9-4fcc781b773f-graysonp] by DomainChecker
2015-04-17 13:46:22,565 DEBUG [c.c.u.AccountManagerImpl]
(API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Access to
Tmpl[222-QCOW2-222-3-98769ef0-8ddf-300a-874f-0e9320ad02c9 granted to
Acct[a4e6eaed-9b06-4806-98f9-4fcc781b773f-graysonp] by DomainChecker
2015-04-17 13:46:22,565 DEBUG [o.a.c.a.BaseCmd]
(API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Ignoring
paremeter displayvm as the caller is not authorized to pass it in
2015-04-17 13:46:22,565 DEBUG [o.a.c.a.BaseCmd]
(API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Ignoring
paremeter deploymentplanner as the caller is not authorized to pass it
in
2015-04-17 13:46:22,566 WARN [c.c.a.d.ParamGenericValidationWorker]
(API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Received
unknown parameters for command deployVirtualMachine. Unknown
parameters : signatureversion expires
2015-04-17 13:46:22,588 DEBUG [c.c.u.AccountManagerImpl]
(API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Access to
VM[User|i-15-1151-VM] granted to
Acct[a4e6eaed-9b06-4806-98f9-4fcc781b773f-graysonp] by DomainChecker
2015-04-17 13:46:22,614 DEBUG [c.c.n.NetworkModelImpl]
(API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Service
SecurityGroup is not supported in the network id=243
2015-04-17 13:46:22,621 DEBUG [c.c.n.NetworkModelImpl]
(API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Service
SecurityGroup is not supported in the network id=243
2015-04-17 13:46:22,658 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Deploy avoids
pods: [], clusters: [], hosts: []
2015-04-17 13:46:22,660 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb)
DeploymentPlanner allocation algorithm:
com.cloud.deploy.FirstFitPlanner@6ba48686
2015-04-17 13:46:22,660 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Trying to
allocate a host and storage pools from dc:1, pod:null,cluster:null,
requested cpu: 500, requested ram: 536870912
2015-04-17 13:46:22,660 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Is ROOT
volume READY (pool already allocated)?: No
2015-04-17 13:46:22,660 DEBUG [c.c.d.FirstFitPlanner]
(API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Searching all
possible resources under this Zone: 1
2015-04-17 13:46:22,662 DEBUG [c.c.d.FirstFitPlanner]
(API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Listing
clusters in order of aggregate capacity, that have (atleast one host
with) enough CPU and RAM capacity under this Zone: 1
2015-04-17 13:46:22,668 DEBUG [c.c.d.FirstFitPlanner]
(API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Removing from
the clusterId list these clusters from avoid set: []
2015-04-17 13:46:22,678 DEBUG [c.c.d.FirstFitPlanner]
(API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Cannot
allocate cluster list [2] for vm creation since their allocated
percentage crosses the disable capacity threshold defined at each
cluster/ at global value for capacity Type : 1, skipping these
clusters
2015-04-17 13:46:22,678 DEBUG [c.c.d.FirstFitPlanner]
(API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) No clusters
found after removing disabled clusters and clusters in avoid list,
returning.
2015-04-17 13:46:22,683 DEBUG [c.c.v.UserVmManagerImpl]
(API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Destroying vm
VM[User|i-15-1151-VM] as it failed to create on Host with Id:null
2015-04-17 13:46:22,706 DEBUG [c.c.c.CapacityManagerImpl]
(API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) VM state
transitted from :Stopped to Error with event:
OperationFailedToErrorvm's original host id: null new host id: null
host id before state transition: null
2015-04-17 13:46:22,724 WARN [o.a.c.alerts]
(API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) alertType::
8 // dataCenterId:: 1 // podId:: null // clusterId:: null // message::
Failed to deploy Vm with Id: 1151, on Host with Id: null
2015-04-17 13:46:22,780 INFO [o.a.c.a.c.u.v.DeployVMCmd]
(API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb)
com.cloud.exception.InsufficientServerCapacityException: Unable to
create a deployment for VM[User|i-15-1151-VM]Scope=interface
com.cloud.dc.DataCenter; id=1
2015-04-17 13:46:22,781 INFO [o.a.c.a.c.u.v.DeployVMCmd]
(API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Unable to
create a deployment for VM[User|i-15-1151-VM]
com.cloud.exception.InsufficientServerCapacityException: Unable to
create a deployment for VM[User|i-15-1151-VM]Scope=interface
com.cloud.dc.DataCenter; id=1
at
org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.reserveVirtualMachine(VMEntityManagerImpl.java:214)
at
org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.reserve(VirtualMachineEntityImpl.java:200)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3468)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3123)
at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3111)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy207.startVirtualMachine(Unknown Source)
at
org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:449)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2015-04-17 13:46:22,783 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-16:ctx-e039eccb job-7312) Complete async job-7312,
jobStatus: FAILED, resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":533,"errortext":"Unable
to create a deployment for VM[User|i-15-1151-VM]"}
2015-04-17 13:46:22,794 DEBUG [o.a.c.f.j.i.AsyncJobManage
