I'm trying to figure out how CPU over-provisioning works in cloudstack. It seems that no matter how much I change the over-provisioning factor value in global settings, I cannot over-provision CPU allocation on the hosts. I suspect that I'm just having a lack of understanding on my part, but it seems like I should be able to increase the number of VMs I can deploy by increasing the CPU over-provisioning value (given that I'm not close to any other resource limits). But this doesn't seem to be the case in practice.
https://screencloud.net/v/l1X7 https://screencloud.net/v/9ifx Error message when trying to create additional machine via API. grayson@gworkstation:~$ cloudmonkey deploy virtualmachine zoneid=a4c20770-f17e-425e-bb21-413dd610b975 serviceofferingid=4b391121-49c7-4434-883d-ffe569f63a3b templateid=e5701f73-b0c3-4dd7-8b2d-b8bf78875a85 keypair=key1 networkids=86e6ded0-6155-449b-a4bd-03411bf0a9cd name=tvmsman1 displayname & [1] 3907 Async job 09148c21-3c42-4334-93b2-428b48fb1fa2 failed Error 533, Unable to create a deployment for VM[User|i-15-1151-VM] accountid = a4e6eaed-9b06-4806-98f9-4fcc781b773f cmd = org.apache.cloudstack.api.command.user.vm.DeployVMCmd created = 2015-04-17T13:46:22-0700 jobid = 09148c21-3c42-4334-93b2-428b48fb1fa2 jobprocstatus = 0 jobresult: errorcode = 533 errortext = Unable to create a deployment for VM[User|i-15-1151-VM] jobresultcode = 530 jobresulttype = object jobstatus = 2 userid = c7a0889e-fcec-4095-9d07-d29ea3c1e5d9 (local) mycloudmonkey> list hosts filter=name,clustername,cpuallocated,cpunumber,cpusockets,cpuspeed,cpuused,cpuwithoverprovisioning, name = v-78-VM ================================================================================ name = s-871-VM ================================================================================ name = cl2hyper2 clustername = CL2 cpuallocated = 100.3% cpunumber = 8 cpusockets = 2 cpuspeed = 1994 cpuused = 1.2% cpuwithoverprovisioning = 63808.0 ================================================================================ name = v-1-VM ================================================================================ name = cl2hyper1 clustername = CL2 cpuallocated = 81.45% cpunumber = 8 cpusockets = 2 cpuspeed = 1995 cpuused = 0.6% cpuwithoverprovisioning = 63840.0 ================================================================================ 2015-04-17 13:46:22,517 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (catalina-exec-25:ctx-7b0eb43a ctx-751b2eca ctx-5c8a75fc) submit async job-7312, details: AsyncJobVO {id:7312, userId: 15, accountId: 15, instanceType: VirtualMachine, instanceId: 1151, cmd: org.apache.cloudstack.api.command.user.vm.DeployVMCmd, cmdInfo: {"serviceofferingid":"4b391121-49c7-4434-883d-ffe569f63a3b","cmdEventType":"VM.CREATE","ctxUserId":"15","zoneid":"a4c20770-f17e-425e-bb21-413dd610b975","signatureversion":"3","httpmethod":"GET","templateid":"e5701f73-b0c3-4dd7-8b2d-b8bf78875a85","networkids":"86e6ded0-6155-449b-a4bd-03411bf0a9cd","apiKey":"wrxDdS8yi2kJPe8UrNdOdLtp55mCSKskk3MyTJ1ucIApdMoYnOWsd-dmigPywD1KC-6BVfVmKiJVenEM_kovpw","id":"1151","response":"json","ctxDetails":"{\"com.cloud.template.VirtualMachineTemplate\":\"e5701f73-b0c3-4dd7-8b2d-b8bf78875a85\",\"com.cloud.vm.VirtualMachine\":\"6b3ace85-8e4c-4467-afdf-b511136ca6b1\",\"com.cloud.network.Network\":\"86e6ded0-6155-449b-a4bd-03411bf0a9cd\",\"com.cloud.dc.DataCenter\":\"a4c20770-f17e-425e-bb21-413dd610b975\",\"com.cloud.offering.ServiceOffering\":\"4b391121-49c7-4434-883d-ffe569f63a3b\"}","expires":"2015-04-17T20:56:22+0000","keypair":"key1","name":"tvmsman1","uuid":"6b3ace85-8e4c-4467-afdf-b511136ca6b1","ctxAccountId":"15","ctxStartEventId":"11922","signature":"8Ebg1+qjfnk5nv7xz1GtdSju2ws\u003d"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 345050015385, completeMsid: null, lastUpdated: null, lastPolled: null, created: null} 2015-04-17 13:46:22,518 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-16:ctx-e039eccb job-7312) Add job-7312 into job monitoring 2015-04-17 13:46:22,518 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-16:ctx-e039eccb job-7312) Executing AsyncJobVO {id:7312, userId: 15, accountId: 15, instanceType: VirtualMachine, instanceId: 1151, cmd: org.apache.cloudstack.api.command.user.vm.DeployVMCmd, cmdInfo: {"serviceofferingid":"4b391121-49c7-4434-883d-ffe569f63a3b","cmdEventType":"VM.CREATE","ctxUserId":"15","zoneid":"a4c20770-f17e-425e-bb21-413dd610b975","signatureversion":"3","httpmethod":"GET","templateid":"e5701f73-b0c3-4dd7-8b2d-b8bf78875a85","networkids":"86e6ded0-6155-449b-a4bd-03411bf0a9cd","apiKey":"wrxDdS8yi2kJPe8UrNdOdLtp55mCSKskk3MyTJ1ucIApdMoYnOWsd-dmigPywD1KC-6BVfVmKiJVenEM_kovpw","id":"1151","response":"json","ctxDetails":"{\"com.cloud.template.VirtualMachineTemplate\":\"e5701f73-b0c3-4dd7-8b2d-b8bf78875a85\",\"com.cloud.vm.VirtualMachine\":\"6b3ace85-8e4c-4467-afdf-b511136ca6b1\",\"com.cloud.network.Network\":\"86e6ded0-6155-449b-a4bd-03411bf0a9cd\",\"com.cloud.dc.DataCenter\":\"a4c20770-f17e-425e-bb21-413dd610b975\",\"com.cloud.offering.ServiceOffering\":\"4b391121-49c7-4434-883d-ffe569f63a3b\"}","expires":"2015-04-17T20:56:22+0000","keypair":"key1","name":"tvmsman1","uuid":"6b3ace85-8e4c-4467-afdf-b511136ca6b1","ctxAccountId":"15","ctxStartEventId":"11922","signature":"8Ebg1+qjfnk5nv7xz1GtdSju2ws\u003d"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 345050015385, completeMsid: null, lastUpdated: null, lastPolled: null, created: null} 2015-04-17 13:46:22,519 DEBUG [c.c.a.ApiServlet] (catalina-exec-25:ctx-7b0eb43a ctx-751b2eca ctx-5c8a75fc) ===END=== 10.101.1.28 -- GET networkids=86e6ded0-6155-449b-a4bd-03411bf0a9cd&apiKey=wrxDdS8yi2kJPe8UrNdOdLtp55mCSKskk3MyTJ1ucIApdMoYnOWsd-dmigPywD1KC-6BVfVmKiJVenEM_kovpw&name=tvmsman1&serviceofferingid=4b391121-49c7-4434-883d-ffe569f63a3b&expires=2015-04-17T20%3A56%3A22%2B0000&signatureversion=3&zoneid=a4c20770-f17e-425e-bb21-413dd610b975&keypair=key1&command=deployVirtualMachine&templateid=e5701f73-b0c3-4dd7-8b2d-b8bf78875a85&signature=8Ebg1%2Bqjfnk5nv7xz1GtdSju2ws%3D&response=json 2015-04-17 13:46:22,522 DEBUG [o.a.c.a.BaseCmd] (API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Ignoring paremeter displayvm as the caller is not authorized to pass it in 2015-04-17 13:46:22,522 DEBUG [o.a.c.a.BaseCmd] (API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Ignoring paremeter deploymentplanner as the caller is not authorized to pass it in 2015-04-17 13:46:22,539 DEBUG [c.c.u.AccountManagerImpl] (API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Access to Acct[a4e6eaed-9b06-4806-98f9-4fcc781b773f-graysonp] granted to Acct[a4e6eaed-9b06-4806-98f9-4fcc781b773f-graysonp] by DomainChecker 2015-04-17 13:46:22,552 DEBUG [c.c.u.AccountManagerImpl] (API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Access to Acct[a4e6eaed-9b06-4806-98f9-4fcc781b773f-graysonp] granted to Acct[a4e6eaed-9b06-4806-98f9-4fcc781b773f-graysonp] by DomainChecker 2015-04-17 13:46:22,565 DEBUG [c.c.u.AccountManagerImpl] (API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Access to Tmpl[222-QCOW2-222-3-98769ef0-8ddf-300a-874f-0e9320ad02c9 granted to Acct[a4e6eaed-9b06-4806-98f9-4fcc781b773f-graysonp] by DomainChecker 2015-04-17 13:46:22,565 DEBUG [o.a.c.a.BaseCmd] (API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Ignoring paremeter displayvm as the caller is not authorized to pass it in 2015-04-17 13:46:22,565 DEBUG [o.a.c.a.BaseCmd] (API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Ignoring paremeter deploymentplanner as the caller is not authorized to pass it in 2015-04-17 13:46:22,566 WARN [c.c.a.d.ParamGenericValidationWorker] (API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Received unknown parameters for command deployVirtualMachine. Unknown parameters : signatureversion expires 2015-04-17 13:46:22,588 DEBUG [c.c.u.AccountManagerImpl] (API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Access to VM[User|i-15-1151-VM] granted to Acct[a4e6eaed-9b06-4806-98f9-4fcc781b773f-graysonp] by DomainChecker 2015-04-17 13:46:22,614 DEBUG [c.c.n.NetworkModelImpl] (API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Service SecurityGroup is not supported in the network id=243 2015-04-17 13:46:22,621 DEBUG [c.c.n.NetworkModelImpl] (API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Service SecurityGroup is not supported in the network id=243 2015-04-17 13:46:22,658 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Deploy avoids pods: [], clusters: [], hosts: [] 2015-04-17 13:46:22,660 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) DeploymentPlanner allocation algorithm: com.cloud.deploy.FirstFitPlanner@6ba48686 2015-04-17 13:46:22,660 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Trying to allocate a host and storage pools from dc:1, pod:null,cluster:null, requested cpu: 500, requested ram: 536870912 2015-04-17 13:46:22,660 DEBUG [c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Is ROOT volume READY (pool already allocated)?: No 2015-04-17 13:46:22,660 DEBUG [c.c.d.FirstFitPlanner] (API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Searching all possible resources under this Zone: 1 2015-04-17 13:46:22,662 DEBUG [c.c.d.FirstFitPlanner] (API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Listing clusters in order of aggregate capacity, that have (atleast one host with) enough CPU and RAM capacity under this Zone: 1 2015-04-17 13:46:22,668 DEBUG [c.c.d.FirstFitPlanner] (API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Removing from the clusterId list these clusters from avoid set: [] 2015-04-17 13:46:22,678 DEBUG [c.c.d.FirstFitPlanner] (API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Cannot allocate cluster list [2] for vm creation since their allocated percentage crosses the disable capacity threshold defined at each cluster/ at global value for capacity Type : 1, skipping these clusters 2015-04-17 13:46:22,678 DEBUG [c.c.d.FirstFitPlanner] (API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) No clusters found after removing disabled clusters and clusters in avoid list, returning. 2015-04-17 13:46:22,683 DEBUG [c.c.v.UserVmManagerImpl] (API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Destroying vm VM[User|i-15-1151-VM] as it failed to create on Host with Id:null 2015-04-17 13:46:22,706 DEBUG [c.c.c.CapacityManagerImpl] (API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) VM state transitted from :Stopped to Error with event: OperationFailedToErrorvm's original host id: null new host id: null host id before state transition: null 2015-04-17 13:46:22,724 WARN [o.a.c.alerts] (API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) alertType:: 8 // dataCenterId:: 1 // podId:: null // clusterId:: null // message:: Failed to deploy Vm with Id: 1151, on Host with Id: null 2015-04-17 13:46:22,780 INFO [o.a.c.a.c.u.v.DeployVMCmd] (API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[User|i-15-1151-VM]Scope=interface com.cloud.dc.DataCenter; id=1 2015-04-17 13:46:22,781 INFO [o.a.c.a.c.u.v.DeployVMCmd] (API-Job-Executor-16:ctx-e039eccb job-7312 ctx-40bb38cb) Unable to create a deployment for VM[User|i-15-1151-VM] com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[User|i-15-1151-VM]Scope=interface com.cloud.dc.DataCenter; id=1 at org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.reserveVirtualMachine(VMEntityManagerImpl.java:214) at org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.reserve(VirtualMachineEntityImpl.java:200) at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3468) at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3123) at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3111) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) at com.sun.proxy.$Proxy207.startVirtualMachine(Unknown Source) at org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:449) at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141) at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2015-04-17 13:46:22,783 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-16:ctx-e039eccb job-7312) Complete async job-7312, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":533,"errortext":"Unable to create a deployment for VM[User|i-15-1151-VM]"} 2015-04-17 13:46:22,794 DEBUG [o.a.c.f.j.i.AsyncJobManage
