Capacity type 1 is CPU:
http://cloudstack.apache.org/api/apidocs-4.10/apis/listCapacity.html
You would look at changing cluster.cpu.allocated.capacity.disablethreshold in
global settings – or the same under your cluster settings.
You can also play with cpu.overprovisioning.factor – but keep in mind
overprovisioning is a science in itself – which has lots of gotcha’s depending
on your workload.
Regards,
Dag Sonstebo
Cloud Architect
ShapeBlue
On 20/10/2017, 11:16, "Jevgeni Zolotarjov" <j.zolotar...@gmail.com> wrote:
I changed cluster.threshold.enabled to false. But it didn't help.
I guess the root cause it something different.
My storage capacity if above 3TB, and only under 300GB used.
No threshold setting was changed ever since installation.
On Fri, Oct 20, 2017 at 12:59 PM, Ivan Kudryavtsev <kudryavtsev...@bw-sw.com
> wrote:
> Just look for threshold. Logs don't include such information.
>
> 2017-10-20 16:56 GMT+07:00 Jevgeni Zolotarjov <j.zolotar...@gmail.com>:
>
> > What is exact parameter name?
> >
> > On Fri, Oct 20, 2017 at 12:44 PM, Ivan Kudryavtsev <
> > kudryavtsev...@bw-sw.com
> > > wrote:
> >
> > > Ah, OK
> > >
> > > 2017-10-20 09:10:53,697 DEBUG [c.c.d.FirstFitPlanner]
> > > (API-Job-Executor-40:ctx-a4cff151 job-181 ctx-91a5717c)
> (logid:25e762a0)
> > > Cannot allocate cluster list [1] for vm creation since their allocated
> > > percentage crosses the disable capacity threshold defined at each
> > cluster/
> > > at global value for capacity Type : 1, skipping these clusters
> > >
> > > That's it. You have to change global variables or cluster
configuration
> > > variables, because you're close to thresholds.
> > >
> > > 2017-10-20 16:40 GMT+07:00 Jevgeni Zolotarjov <j.zolotar...@gmail.com
> >:
> > >
> > > > Hi,
> > > >
> > > > Yes I have one host. But I have exactly zero affinity groups. I
could
> > > > successfully add few instances and they are running.
> > > > The
> > > >
> > > >
> > > > On Fri, Oct 20, 2017 at 12:35 PM, Ivan Kudryavtsev <
> > > > kudryavtsev...@bw-sw.com
> > > > > wrote:
> > > >
> > > > > Hi,
> > > > > It seems you have one host and place several VMs in a single
> > > > anti-affinity
> > > > > group, don't you? This is how it should really work if you do that
> > > thing.
> > > > > Anti-affinity groups are meaningless if you don't have several
> hosts.
> > > > >
> > > > > 2017-10-20 16:19 GMT+07:00 Jevgeni Zolotarjov <
> > j.zolotar...@gmail.com
> > > >:
> > > > >
> > > > > > I could successfully create and run an instance with prepared
> > Compute
> > > > > > offering. But when I try to add one more instance of the same
> > Compute
> > > > > > offering, I am getting an error.
> > > > > > Unable to create a deployment for VM
> > > > > >
> > > > > > There are plenty of resources on host: cores, RAM, storage.
> > > > > >
> > > > > > Pleas help
> > > > > >
> > > > > > management-server.log:
> > > > > > 2017-10-20 09:10:43,713 DEBUG [c.c.a.m.AgentManagerImpl]
> > > > > > (AgentManager-Handler-3:null) (logid:) SeqA 5-377872: Processing
> > Seq
> > > > > > 5-377872: { Cmd , MgmtId: -1, via: 5, Ver: v1, Flags: 11,
> > > > > > [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand"
> > > > > > :{"_proxyVmId":1,"_loadInfo":"{\n
> > > > > > \"connections\": []\n}","wait":0}}] }
> > > > > > 2017-10-20 09:10:43,715 DEBUG [c.c.a.m.AgentManagerImpl]
> > > > > > (AgentManager-Handler-3:null) (logid:) SeqA 5-377872: Sending
Seq
> > > > > > 5-377872: { Ans: , MgmtId: 264216221068220, via: 5, Ver: v1,
> > Flags:
> > > > > > 100010,
> > > > > > [{"com.cloud.agent.api.AgentControlAnswer":{"result":
> > > true,"wait":0}}]
> > > > }
> > > > > > 2017-10-20 09:10:45,239 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > (AsyncJobMgr-Heartbeat-1:ctx-566e7818) (logid:24eee607) Begin
> > > cleanup
> > > > > > expired async-jobs
> > > > > > 2017-10-20 09:10:45,243 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > (AsyncJobMgr-Heartbeat-1:ctx-566e7818) (logid:24eee607) End
> > cleanup
> > > > > > expired
> > > > > > async-jobs
> > > > > > 2017-10-20 09:10:53,566 DEBUG [c.c.a.ApiServlet]
> > > > > > (catalina-exec-6:ctx-3911a4cc) (logid:41db964f) ===START===
> > > > > 192.168.1.194
> > > > > > -- GET
> > > > > > command=deployVirtualMachine&response=json&zoneid=b3d5b8bb-
> > > > > > 2dea-478b-8077-fc813b90d4c6&templateid=244d6029-0f9a-4479-
> > > > > > b11b-09e4fe6c40e1&hypervisor=KVM&serviceofferingid=
> > > > > > e0ef2da9-a489-4c42-ab6d-c49fd41b0fd2&diskofferingid=
> > > > > > f39fb02b-4fc0-4f2e-9e4e-dd54f7fc9508&size=50&
> > > > securitygroupids=1586621a-
> > > > > > 8e8d-11e7-8f52-f04da2002bbe&displayname=mtl-dev04-app02&
> > > > > > name=mtl-dev04-app02&keyboard=us&_=1508490653550
> > > > > > 2017-10-20 09:10:53,581 DEBUG [c.c.u.AccountManagerImpl]
> > > > > > (catalina-exec-6:ctx-3911a4cc ctx-773820d9) (logid:41db964f)
> Access
> > > > > granted
> > > > > > to Acct[15863393-8e8d-11e7-8f52-f04da2002bbe-admin] to
> > > > > > org.apache.cloudstack.quota.vo.ServiceOfferingVO$$
> > > > > > EnhancerByCGLIB$$4d79c9fa@540d8fcb
> > > > > > by AffinityGroupAccessChecker
> > > > > > 2017-10-20 09:10:53,581 DEBUG [c.c.u.AccountManagerImpl]
> > > > > > (catalina-exec-6:ctx-3911a4cc ctx-773820d9) (logid:41db964f)
> Access
> > > > > granted
> > > > > > to Acct[15863393-8e8d-11e7-8f52-f04da2002bbe-admin] to
> > > > > > com.cloud.storage.DiskOfferingVO$$EnhancerByCGLIB$$f64346ed@
> > 7d0dcb96
> > > > by
> > > > > > AffinityGroupAccessChecker
> > > > > > 2017-10-20 09:10:53,605 DEBUG [c.c.v.UserVmManagerImpl]
> > > > > > (catalina-exec-6:ctx-3911a4cc ctx-773820d9) (logid:41db964f)
> > > Allocating
> > > > > in
> > > > > > the DB for vm
> > > > > > 2017-10-20 09:10:53,613 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > > > > > (catalina-exec-6:ctx-3911a4cc ctx-773820d9) (logid:41db964f)
> > > Allocating
> > > > > > entries for VM: VM[User|i-2-23-VM]
> > > > > > 2017-10-20 09:10:53,614 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > > > > > (catalina-exec-6:ctx-3911a4cc ctx-773820d9) (logid:41db964f)
> > > Allocating
> > > > > > nics for VM[User|i-2-23-VM]
> > > > > > 2017-10-20 09:10:53,614 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > > > > > (catalina-exec-6:ctx-3911a4cc ctx-773820d9) (logid:41db964f)
> > > Allocating
> > > > > nic
> > > > > > for vm VM[User|i-2-23-VM] in network Ntwk[204|Guest|6] with
> > requested
> > > > > > profile NicProfile[0-0-null-null-null
> > > > > > 2017-10-20 09:10:53,619 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > > > > > (catalina-exec-6:ctx-3911a4cc ctx-773820d9) (logid:41db964f)
> > > Allocating
> > > > > > disks for VM[User|i-2-23-VM]
> > > > > > 2017-10-20 09:10:53,626 DEBUG [c.c.v.VirtualMachineManagerImpl]
> > > > > > (catalina-exec-6:ctx-3911a4cc ctx-773820d9) (logid:41db964f)
> > > Allocation
> > > > > > completed for VM: VM[User|i-2-23-VM]
> > > > > > 2017-10-20 09:10:53,626 DEBUG [c.c.v.UserVmManagerImpl]
> > > > > > (catalina-exec-6:ctx-3911a4cc ctx-773820d9) (logid:41db964f)
> > > > Successfully
> > > > > > allocated DB entry for VM[User|i-2-23-VM]
> > > > > > 2017-10-20 09:10:53,653 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> > > > > > (API-Job-Executor-40:ctx-a4cff151 job-181) (logid:0c9b9886) Add
> > > > job-181
> > > > > > into job monitoring
> > > > > > 2017-10-20 09:10:53,658 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > (catalina-exec-6:ctx-3911a4cc ctx-773820d9) (logid:41db964f)
> submit
> > > > async
> > > > > > job-181, details: AsyncJobVO {id:181, userId: 2, accountId: 2,
> > > > > > instanceType: VirtualMachine, instanceId: 23, cmd:
> > > > > > org.apache.cloudstack.api.command.admin.vm.DeployVMCmdByAdmin,
> > > > cmdInfo:
> > > > > > {"keyboard":"us","httpmethod":"GET","templateid":"244d6029-
> > > > > > 0f9a-4479-b11b-09e4fe6c40e1","securitygroupids":"1586621a-
> > > > > > 8e8d-11e7-8f52-f04da2002bbe","ctxAccountId":"2","uuid":"
> > > > > > df7da1f1-4f17-4e98-9950-84f30de1fcb2","cmdEventType":"
> > > > > > VM.CREATE","diskofferingid":"f39fb02b-4fc0-4f2e-9e4e-
> > > > > > dd54f7fc9508","size":"50","serviceofferingid":"e0ef2da9-
> > > > > > a489-4c42-ab6d-c49fd41b0fd2","response":"json","ctxUserId":"
> > > > > > 2","hypervisor":"KVM","displayname":"mtl-dev04-app02"
> > > > > > ,"name":"mtl-dev04-app02","zoneid":"b3d5b8bb-2dea-478b-
> > > > > > 8077-fc813b90d4c6","ctxStartEventId":"277","id":"
> > > > > > 23","ctxDetails":"{\"interface
> > > > > > com.cloud.offering.DiskOffering\":\"f39fb02b-
> > > > > > 4fc0-4f2e-9e4e-dd54f7fc9508\",\"interface
> > > > > > com.cloud.network.security.SecurityGroup\":\"1586621a-
> > > > > > 8e8d-11e7-8f52-f04da2002bbe\",\"interface
> > > > > > com.cloud.vm.VirtualMachine\":\"df7da1f1-4f17-4e98-9950-
> > > > > > 84f30de1fcb2\",\"interface
> > > > > > com.cloud.dc.DataCenter\":\"b3d5b8bb-2dea-478b-8077-
> > > > > > fc813b90d4c6\",\"interface
> > > > > > com.cloud.offering.ServiceOffering\":\"e0ef2da9-
> > > > > > a489-4c42-ab6d-c49fd41b0fd2\",\"interface
> > > > > > com.cloud.template.VirtualMachineTemplate\":\"
> > > 244d6029-0f9a-4479-b11b-
> > > > > > 09e4fe6c40e1\"}","_":"1508490653550"},
> > > > > > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0,
resultCode:
> > 0,
> > > > > > result: null, initMsid: 264216221068220, completeMsid: null,
> > > > lastUpdated:
> > > > > > null, lastPolled: null, created: null}
> > > > > > 2017-10-20 09:10:53,659 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > (API-Job-Executor-40:ctx-a4cff151 job-181) (logid:25e762a0)
> > > Executing
> > > > > > AsyncJobVO {id:181, userId: 2, accountId: 2, instanceType:
> > > > > VirtualMachine,
> > > > > > instanceId: 23, cmd:
> > > > > > org.apache.cloudstack.api.command.admin.vm.DeployVMCmdByAdmin,
> > > > cmdInfo:
> > > > > > {"keyboard":"us","httpmethod":"GET","templateid":"244d6029-
> > > > > > 0f9a-4479-b11b-09e4fe6c40e1","securitygroupids":"1586621a-
> > > > > > 8e8d-11e7-8f52-f04da2002bbe","ctxAccountId":"2","uuid":"
> > > > > > df7da1f1-4f17-4e98-9950-84f30de1fcb2","cmdEventType":"
> > > > > > VM.CREATE","diskofferingid":"f39fb02b-4fc0-4f2e-9e4e-
> > > > > > dd54f7fc9508","size":"50","serviceofferingid":"e0ef2da9-
> > > > > > a489-4c42-ab6d-c49fd41b0fd2","response":"json","ctxUserId":"
> > > > > > 2","hypervisor":"KVM","displayname":"mtl-dev04-app02"
> > > > > > ,"name":"mtl-dev04-app02","zoneid":"b3d5b8bb-2dea-478b-
> > > > > > 8077-fc813b90d4c6","ctxStartEventId":"277","id":"
> > > > > > 23","ctxDetails":"{\"interface
> > > > > > com.cloud.offering.DiskOffering\":\"f39fb02b-
> > > > > > 4fc0-4f2e-9e4e-dd54f7fc9508\",\"interface
> > > > > > com.cloud.network.security.SecurityGroup\":\"1586621a-
> > > > > > 8e8d-11e7-8f52-f04da2002bbe\",\"interface
> > > > > > com.cloud.vm.VirtualMachine\":\"df7da1f1-4f17-4e98-9950-
> > > > > > 84f30de1fcb2\",\"interface
> > > > > > com.cloud.dc.DataCenter\":\"b3d5b8bb-2dea-478b-8077-
> > > > > > fc813b90d4c6\",\"interface
> > > > > > com.cloud.offering.ServiceOffering\":\"e0ef2da9-
> > > > > > a489-4c42-ab6d-c49fd41b0fd2\",\"interface
> > > > > > com.cloud.template.VirtualMachineTemplate\":\"
> > > 244d6029-0f9a-4479-b11b-
> > > > > > 09e4fe6c40e1\"}","_":"1508490653550"},
> > > > > > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0,
resultCode:
> > 0,
> > > > > > result: null, initMsid: 264216221068220, completeMsid: null,
> > > > lastUpdated:
> > > > > > null, lastPolled: null, created: null}
> > > > > > 2017-10-20 09:10:53,659 DEBUG [c.c.a.ApiServlet]
> > > > > > (catalina-exec-6:ctx-3911a4cc ctx-773820d9) (logid:41db964f)
> > > ===END===
> > > > > > 192.168.1.194 -- GET
> > > > > > command=deployVirtualMachine&response=json&zoneid=b3d5b8bb-
> > > > > > 2dea-478b-8077-fc813b90d4c6&templateid=244d6029-0f9a-4479-
> > > > > > b11b-09e4fe6c40e1&hypervisor=KVM&serviceofferingid=
> > > > > > e0ef2da9-a489-4c42-ab6d-c49fd41b0fd2&diskofferingid=
> > > > > > f39fb02b-4fc0-4f2e-9e4e-dd54f7fc9508&size=50&
> > > > securitygroupids=1586621a-
> > > > > > 8e8d-11e7-8f52-f04da2002bbe&displayname=mtl-dev04-app02&
> > > > > > name=mtl-dev04-app02&keyboard=us&_=1508490653550
> > > > > > 2017-10-20 09:10:53,688 DEBUG [c.c.d.
> > DeploymentPlanningManagerImpl]
> > > > > > (API-Job-Executor-40:ctx-a4cff151 job-181 ctx-91a5717c)
> > > > (logid:25e762a0)
> > > > > > DeploymentPlanner allocation algorithm: null
> > > > > > 2017-10-20 09:10:53,688 DEBUG [c.c.d.
> > DeploymentPlanningManagerImpl]
> > > > > > (API-Job-Executor-40:ctx-a4cff151 job-181 ctx-91a5717c)
> > > > (logid:25e762a0)
> > > > > > Trying to allocate a host and storage pools from dc:1,
> > > > > > pod:null,cluster:null, requested cpu: 10000, requested ram:
> > > 8594128896
> > > > > > 2017-10-20 09:10:53,688 DEBUG [c.c.d.
> > DeploymentPlanningManagerImpl]
> > > > > > (API-Job-Executor-40:ctx-a4cff151 job-181 ctx-91a5717c)
> > > > (logid:25e762a0)
> > > > > > Is
> > > > > > ROOT volume READY (pool already allocated)?: No
> > > > > > 2017-10-20 09:10:53,691 DEBUG [c.c.d.
> > DeploymentPlanningManagerImpl]
> > > > > > (API-Job-Executor-40:ctx-a4cff151 job-181 ctx-91a5717c)
> > > > (logid:25e762a0)
> > > > > > Deploy avoids pods: [], clusters: [], hosts: []
> > > > > > 2017-10-20 09:10:53,691 DEBUG [c.c.d.FirstFitPlanner]
> > > > > > (API-Job-Executor-40:ctx-a4cff151 job-181 ctx-91a5717c)
> > > > (logid:25e762a0)
> > > > > > Searching all possible resources under this Zone: 1
> > > > > > 2017-10-20 09:10:53,692 DEBUG [c.c.d.FirstFitPlanner]
> > > > > > (API-Job-Executor-40:ctx-a4cff151 job-181 ctx-91a5717c)
> > > > (logid:25e762a0)
> > > > > > Listing clusters in order of aggregate capacity, that have
> (atleast
> > > one
> > > > > > host with) enough CPU and RAM capacity under this Zone: 1
> > > > > > 2017-10-20 09:10:53,693 DEBUG [c.c.d.FirstFitPlanner]
> > > > > > (API-Job-Executor-40:ctx-a4cff151 job-181 ctx-91a5717c)
> > > > (logid:25e762a0)
> > > > > > Removing from the clusterId list these clusters from avoid set:
> []
> > > > > > 2017-10-20 09:10:53,697 DEBUG [c.c.d.FirstFitPlanner]
> > > > > > (API-Job-Executor-40:ctx-a4cff151 job-181 ctx-91a5717c)
> > > > (logid:25e762a0)
> > > > > > Cannot allocate cluster list [1] for vm creation since their
> > > allocated
> > > > > > percentage crosses the disable capacity threshold defined at
each
> > > > > cluster/
> > > > > > at global value for capacity Type : 1, skipping these clusters
> > > > > > 2017-10-20 09:10:53,697 DEBUG [c.c.d.FirstFitPlanner]
> > > > > > (API-Job-Executor-40:ctx-a4cff151 job-181 ctx-91a5717c)
> > > > (logid:25e762a0)
> > > > > > No
> > > > > > clusters found after removing disabled clusters and clusters in
> > avoid
> > > > > list,
> > > > > > returning.
> > > > > > 2017-10-20 09:10:53,698 DEBUG [c.c.v.UserVmManagerImpl]
> > > > > > (API-Job-Executor-40:ctx-a4cff151 job-181 ctx-91a5717c)
> > > > (logid:25e762a0)
> > > > > > Destroying vm VM[User|i-2-23-VM] as it failed to create on Host
> > with
> > > > > > Id:null
> > > > > > 2017-10-20 09:10:53,701 DEBUG [c.c.c.CapacityManagerImpl]
> > > > > > (API-Job-Executor-40:ctx-a4cff151 job-181 ctx-91a5717c)
> > > > (logid:25e762a0)
> > > > > > VM
> > > > > > state transitted from :Stopped to Error with event:
> > > > > > OperationFailedToErrorvm's original host id: null new host id:
> null
> > > > host
> > > > > id
> > > > > > before state transition: null
> > > > > > 2017-10-20 09:10:53,712 DEBUG [c.c.u.AccountManagerImpl]
> > > > > > (API-Job-Executor-40:ctx-a4cff151 job-181 ctx-91a5717c)
> > > > (logid:25e762a0)
> > > > > > Access granted to Acct[15863393-8e8d-11e7-8f52-
> f04da2002bbe-admin]
> > > to
> > > > > > Domain:1/ by AffinityGroupAccessChecker
> > > > > > 2017-10-20 09:10:53,713 DEBUG [c.c.a.m.AgentManagerImpl]
> > > > > > (AgentManager-Handler-5:null) (logid:) SeqA 5-377873: Processing
> > Seq
> > > > > > 5-377873: { Cmd , MgmtId: -1, via: 5, Ver: v1, Flags: 11,
> > > > > > [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand"
> > > > > > :{"_proxyVmId":1,"_loadInfo":"{\n
> > > > > > \"connections\": []\n}","wait":0}}] }
> > > > > > 2017-10-20 09:10:53,714 DEBUG [c.c.a.m.AgentManagerImpl]
> > > > > > (AgentManager-Handler-5:null) (logid:) SeqA 5-377873: Sending
Seq
> > > > > > 5-377873: { Ans: , MgmtId: 264216221068220, via: 5, Ver: v1,
> > Flags:
> > > > > > 100010,
> > > > > > [{"com.cloud.agent.api.AgentControlAnswer":{"result":
> > > true,"wait":0}}]
> > > > }
> > > > > > 2017-10-20 09:10:53,715 WARN [o.a.c.alerts]
> > > > > > (API-Job-Executor-40:ctx-a4cff151 job-181 ctx-91a5717c)
> > > > (logid:25e762a0)
> > > > > > alertType:: 8 // dataCenterId:: 1 // podId:: null // clusterId::
> > null
> > > > //
> > > > > > message:: Failed to deploy Vm with Id: 23, on Host with Id: null
> > > > > > 2017-10-20 09:10:53,726 INFO [o.a.c.a.c.a.v.DeployVMCmdByAdmin]
> > > > > > (API-Job-Executor-40:ctx-a4cff151 job-181 ctx-91a5717c)
> > > > (logid:25e762a0)
> > > > > > com.cloud.exception.InsufficientServerCapacityException: Unable
> to
> > > > > create
> > > > > > a
> > > > > > deployment for VM[User|i-2-23-VM]Scope=interface
> > > > > com.cloud.dc.DataCenter;
> > > > > > id=1
> > > > > > 2017-10-20 09:10:53,726 INFO [o.a.c.a.c.a.v.DeployVMCmdByAdmin]
> > > > > > (API-Job-Executor-40:ctx-a4cff151 job-181 ctx-91a5717c)
> > > > (logid:25e762a0)
> > > > > > Unable to create a deployment for VM[User|i-2-23-VM]
> > > > > > com.cloud.exception.InsufficientServerCapacityException: Unable
> to
> > > > > create
> > > > > > a
> > > > > > deployment for VM[User|i-2-23-VM]Scope=interface
> > > > > com.cloud.dc.DataCenter;
> > > > > > id=1
> > > > > > at
> > > > > > org.apache.cloudstack.engine.cloud.entity.api.
> VMEntityManagerImpl.
> > > > > > reserveVirtualMachine(VMEntityManagerImpl.java:215)
> > > > > > at
> > > > > > org.apache.cloudstack.engine.cloud.entity.api.
> > > > VirtualMachineEntityImpl.
> > > > > > reserve(VirtualMachineEntityImpl.java:200)
> > > > > > at
> > > > > > com.cloud.vm.UserVmManagerImpl.startVirtualMachine(
> > > > > > UserVmManagerImpl.java:4158)
> > > > > > at
> > > > > > com.cloud.vm.UserVmManagerImpl.startVirtualMachine(
> > > > > > UserVmManagerImpl.java:3738)
> > > > > > at
> > > > > > com.cloud.vm.UserVmManagerImpl.startVirtualMachine(
> > > > > > UserVmManagerImpl.java:3726)
> > > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > > > > at
> > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(
> > > > > NativeMethodAccessorImpl.java:
> > > > > > 62)
> > > > > > at
> > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(
> > > > > > DelegatingMethodAccessorImpl.java:43)
> > > > > > at java.lang.reflect.Method.invoke(Method.java:498)
> > > > > > at
> > > > > > org.springframework.aop.support.AopUtils.
> > > > invokeJoinpointUsingReflection
> > > > > > (AopUtils.java:333)
> > > > > > at
> > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.
> > > > > > invokeJoinpoint(ReflectiveMethodInvocation.java:190)
> > > > > > at
> > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.
> > > proceed(
> > > > > > ReflectiveMethodInvocation.java:157)
> > > > > > at
> > > > > > org.apache.cloudstack.network.contrail.management.
> > > > > > EventUtils$EventInterceptor.invoke(EventUtils.java:107)
> > > > > > at
> > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.
> > > proceed(
> > > > > > ReflectiveMethodInvocation.java:168)
> > > > > > at
> > > > > > com.cloud.event.ActionEventInterceptor.invoke(
> > > > > ActionEventInterceptor.java:
> > > > > > 51)
> > > > > > at
> > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.
> > > proceed(
> > > > > > ReflectiveMethodInvocation.java:168)
> > > > > > at
> > > > > > org.springframework.aop.interceptor.ExposeInvocationInterceptor.
> > > > invoke(
> > > > > > ExposeInvocationInterceptor.java:92)
> > > > > > at
> > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.
> > > proceed(
> > > > > > ReflectiveMethodInvocation.java:179)
> > > > > > at
> > > > > > org.springframework.aop.framework.JdkDynamicAopProxy.
> > > > > > invoke(JdkDynamicAopProxy.java:213)
> > > > > > at com.sun.proxy.$Proxy207.startVirtualMachine(Unknown Source)
> > > > > > at
> > > > > > org.apache.cloudstack.api.command.admin.vm.
> > > DeployVMCmdByAdmin.execute(
> > > > > > DeployVMCmdByAdmin.java:50)
> > > > > > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
> > > > > > at
> > > > > > com.cloud.api.ApiAsyncJobDispatcher.runJob(
> > > ApiAsyncJobDispatcher.java:
> > > > > 108)
> > > > > > at
> > > > > > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.
> > > > > > runInContext(AsyncJobManagerImpl.java:558)
> > > > > > at
> > > > > > org.apache.cloudstack.managed.context.
> > ManagedContextRunnable$1.run(
> > > > > > ManagedContextRunnable.java:49)
> > > > > > at
> > > > > > org.apache.cloudstack.managed.context.impl.
> > > > DefaultManagedContext$1.call(
> > > > > > DefaultManagedContext.java:56)
> > > > > > at
> > > > > > org.apache.cloudstack.managed.context.impl.
> DefaultManagedContext.
> > > > > > callWithContext(DefaultManagedContext.java:103)
> > > > > > at
> > > > > > org.apache.cloudstack.managed.context.impl.
> DefaultManagedContext.
> > > > > > runWithContext(DefaultManagedContext.java:53)
> > > > > > at
> > > > > > org.apache.cloudstack.managed.context.
> ManagedContextRunnable.run(
> > > > > > ManagedContextRunnable.java:46)
> > > > > > at
> > > > > > org.apache.cloudstack.framework.jobs.impl.
> > AsyncJobManagerImpl$5.run(
> > > > > > AsyncJobManagerImpl.java:506)
> > > > > > at java.util.concurrent.Executors$RunnableAdapter.
> > > > > call(Executors.java:511)
> > > > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > > > at
> > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > > > > ThreadPoolExecutor.java:1149)
> > > > > > at
> > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > > > > ThreadPoolExecutor.java:624)
> > > > > > at java.lang.Thread.run(Thread.java:748)
> > > > > > 2017-10-20 09:10:53,726 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > (API-Job-Executor-40:ctx-a4cff151 job-181) (logid:25e762a0)
> > Complete
> > > > > async
> > > > > > job-181, jobStatus: FAILED, resultCode: 530, result:
> > > > > > org.apache.cloudstack.api.response.ExceptionResponse/
> > > > > null/{"uuidList":[],"
> > > > > > errorcode":533,"errortext":"Unable
> > > > > > to create a deployment for VM[User|i-2-23-VM]"}
> > > > > > 2017-10-20 09:10:53,727 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > (API-Job-Executor-40:ctx-a4cff151 job-181) (logid:25e762a0)
> > Publish
> > > > > async
> > > > > > job-181 complete on message bus
> > > > > > 2017-10-20 09:10:53,727 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > (API-Job-Executor-40:ctx-a4cff151 job-181) (logid:25e762a0) Wake
> > up
> > > > jobs
> > > > > > related to job-181
> > > > > > 2017-10-20 09:10:53,727 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > (API-Job-Executor-40:ctx-a4cff151 job-181) (logid:25e762a0)
> Update
> > > db
> > > > > > status for job-181
> > > > > > 2017-10-20 09:10:53,728 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > (API-Job-Executor-40:ctx-a4cff151 job-181) (logid:25e762a0) Wake
> > up
> > > > jobs
> > > > > > joined with job-181 and disjoin all subjobs created from job-
181
> > > > > > 2017-10-20 09:10:53,729 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > (API-Job-Executor-40:ctx-a4cff151 job-181) (logid:25e762a0) Done
> > > > > executing
> > > > > > org.apache.cloudstack.api.command.admin.vm.DeployVMCmdByAdmin
> for
> > > > > job-181
> > > > > > 2017-10-20 09:10:53,729 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> > > > > > (API-Job-Executor-40:ctx-a4cff151 job-181) (logid:25e762a0)
> Remove
> > > > > job-181
> > > > > > from job monitoring
> > > > > > 2017-10-20 09:10:55,239 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > (AsyncJobMgr-Heartbeat-1:ctx-befa869b) (logid:3bfc7ad9) Begin
> > > cleanup
> > > > > > expired async-jobs
> > > > > > 2017-10-20 09:10:55,242 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > (AsyncJobMgr-Heartbeat-1:ctx-befa869b) (logid:3bfc7ad9) End
> > cleanup
> > > > > > expired
> > > > > > async-jobs
> > > > > > 2017-10-20 09:10:55,339 DEBUG [c.c.n.r.
> > > VirtualNetworkApplianceManager
> > > > > Impl]
> > > > > > (RouterStatusMonitor-1:ctx-cc2ebd5d) (logid:00c6f41b) Found 0
> > > routers
> > > > to
> > > > > > update status.
> > > > > > 2017-10-20 09:10:55,340 DEBUG [c.c.n.r.
> > > VirtualNetworkApplianceManager
> > > > > Impl]
> > > > > > (RouterStatusMonitor-1:ctx-cc2ebd5d) (logid:00c6f41b) Found 0
> VPC
> > > > > networks
> > > > > > to update Redundant State.
> > > > > > 2017-10-20 09:10:55,341 DEBUG [c.c.n.r.
> > > VirtualNetworkApplianceManager
> > > > > Impl]
> > > > > > (RouterStatusMonitor-1:ctx-cc2ebd5d) (logid:00c6f41b) Found 0
> > > networks
> > > > > to
> > > > > > update RvR status.
> > > > > > 2017-10-20 09:10:56,695 DEBUG [c.c.a.ApiServlet]
> > > > > > (catalina-exec-13:ctx-8a748fc4) (logid:3489934a) ===START===
> > > > > > 192.168.1.194
> > > > > > -- GET
> > > > > > command=queryAsyncJobResult&jobId=25e762a0-615e-4cc7-8da0-
> > > > > > 66a178e0e5f9&response=json&_=1508490656682
> > > > > > 2017-10-20 09:10:56,706 DEBUG [c.c.a.ApiServlet]
> > > > > > (catalina-exec-13:ctx-8a748fc4 ctx-7b9abd96) (logid:3489934a)
> > > ===END===
> > > > > > 192.168.1.194 -- GET
> > > > > > command=queryAsyncJobResult&jobId=25e762a0-615e-4cc7-8da0-
> > > > > > 66a178e0e5f9&response=json&_=1508490656682
> > > > > > 2017-10-20 09:10:56,711 DEBUG [c.c.a.ApiServlet]
> > > > > > (catalina-exec-8:ctx-53eff2d0) (logid:f9178c9c) ===START===
> > > > > 192.168.1.194
> > > > > > -- GET
> > > > > > command=listVirtualMachines&id=df7da1f1-4f17-4e98-9950-
> > > > > > 84f30de1fcb2&response=json&_=1508490656699
> > > > > > 2017-10-20 09:10:56,733 DEBUG [c.c.a.ApiServlet]
> > > > > > (catalina-exec-8:ctx-53eff2d0 ctx-20971edb) (logid:f9178c9c)
> > > ===END===
> > > > > > 192.168.1.194 -- GET
> > > > > > command=listVirtualMachines&id=df7da1f1-4f17-4e98-9950-
> > > > > > 84f30de1fcb2&response=json&_=1508490656699
> > > > > > 2017-10-20 09:10:58,694 DEBUG [c.c.s.StatsCollector]
> > > > > > (StatsCollector-6:ctx-bcafae07) (logid:afeb8a5c)
> VmStatsCollector
> > is
> > > > > > running...
> > > > > > 2017-10-20 09:10:58,714 DEBUG [c.c.a.m.AgentManagerImpl]
> > > > > > (AgentManager-Handler-15:null) (logid:) SeqA 5-377874:
Processing
> > Seq
> > > > > > 5-377874: { Cmd , MgmtId: -1, via: 5, Ver: v1, Flags: 11,
> > > > > > [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand"
> > > > > > :{"_proxyVmId":1,"_loadInfo":"{\n
> > > > > > \"connections\": []\n}","wait":0}}] }
> > > > > > 2017-10-20 09:10:58,716 DEBUG [c.c.a.m.AgentManagerImpl]
> > > > > > (AgentManager-Handler-15:null) (logid:) SeqA 5-377874: Sending
> Seq
> > > > > > 5-377874: { Ans: , MgmtId: 264216221068220, via: 5, Ver: v1,
> > Flags:
> > > > > > 100010,
> > > > > > [{"com.cloud.agent.api.AgentControlAnswer":{"result":
> > > true,"wait":0}}]
> > > > }
> > > > > > 2017-10-20 09:10:58,761 DEBUG [c.c.s.StatsCollector]
> > > > > > (StatsCollector-1:ctx-fe08de8f) (logid:a41fef27)
> StorageCollector
> > is
> > > > > > running...
> > > > > > 2017-10-20 09:10:58,766 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
> > > > > > (StatsCollector-1:ctx-fe08de8f) (logid:a41fef27)
> > > > > getCommandHostDelegation:
> > > > > > class com.cloud.agent.api.GetStorageStatsCommand
> > > > > > 2017-10-20 09:10:58,766 DEBUG [c.c.h.XenServerGuru]
> > > > > > (StatsCollector-1:ctx-fe08de8f) (logid:a41fef27)
> > > > > getCommandHostDelegation:
> > > > > > class com.cloud.agent.api.GetStorageStatsCommand
> > > > > > 2017-10-20 09:10:58,792 DEBUG [c.c.a.t.Request]
> > > > > > (StatsCollector-6:ctx-bcafae07) (logid:afeb8a5c) Seq
> > > > > > 3-6621135877164909219:
> > > > > > Received: { Ans: , MgmtId: 264216221068220, via:
> 3(mtl1-apphst03),
> > > > Ver:
> > > > > > v1, Flags: 10, { GetVmStatsAnswer } }
> > > > > > 2017-10-20 09:10:58,813 DEBUG [c.c.a.t.Request]
> > > > > > (StatsCollector-1:ctx-fe08de8f) (logid:a41fef27) Seq
> > > > > > 4-1596244592926179066:
> > > > > > Received: { Ans: , MgmtId: 264216221068220, via: 4(s-2-VM),
Ver:
> > v1,
> > > > > > Flags: 10, { GetStorageStatsAnswer } }
> > > > > > 2017-10-20 09:10:58,814 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
> > > > > > (StatsCollector-1:ctx-fe08de8f) (logid:a41fef27)
> > > > > getCommandHostDelegation:
> > > > > > class com.cloud.agent.api.GetStorageStatsCommand
> > > > > > 2017-10-20 09:10:58,814 DEBUG [c.c.h.XenServerGuru]
> > > > > > (StatsCollector-1:ctx-fe08de8f) (logid:a41fef27)
> > > > > getCommandHostDelegation:
> > > > > > class com.cloud.agent.api.GetStorageStatsCommand
> > > > > > 2017-10-20 09:10:58,881 DEBUG [c.c.a.t.Request]
> > > > > > (StatsCollector-1:ctx-fe08de8f) (logid:a41fef27) Seq
> > > > > > 3-6621135877164909220:
> > > > > > Received: { Ans: , MgmtId: 264216221068220, via:
> 3(mtl1-apphst03),
> > > > Ver:
> > > > > > v1, Flags: 10, { GetStorageStatsAnswer } }
> > > > > > 2017-10-20 09:11:05,239 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > (AsyncJobMgr-Heartbeat-1:ctx-b43d5944) (logid:dbf81eb8) Begin
> > > cleanup
> > > > > > expired async-jobs
> > > > > > 2017-10-20 09:11:05,242 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > (AsyncJobMgr-Heartbeat-1:ctx-b43d5944) (logid:dbf81eb8) End
> > cleanup
> > > > > > expired
> > > > > > async-jobs
> > > > > > 2017-10-20 09:11:08,714 DEBUG [c.c.a.m.AgentManagerImpl]
> > > > > > (AgentManager-Handler-1:null) (logid:) SeqA 5-377875: Processing
> > Seq
> > > > > > 5-377875: { Cmd , MgmtId: -1, via: 5, Ver: v1, Flags: 11,
> > > > > > [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand"
> > > > > > :{"_proxyVmId":1,"_loadInfo":"{\n
> > > > > > \"connections\": []\n}","wait":0}}] }
> > > > > > 2017-10-20 09:11:08,716 DEBUG [c.c.a.m.AgentManagerImpl]
> > > > > > (AgentManager-Handler-1:null) (logid:) SeqA 5-377875: Sending
Seq
> > > > > > 5-377875: { Ans: , MgmtId: 264216221068220, via: 5, Ver: v1,
> > Flags:
> > > > > > 100010,
> > > > > > [{"com.cloud.agent.api.AgentControlAnswer":{"result":
> > > true,"wait":0}}]
> > > > }
> > > > > > 2017-10-20 09:11:10,978 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> > > > > > (consoleproxy-1:ctx-db1623b9) (logid:5ce90094) Zone 1 is ready
to
> > > > launch
> > > > > > console proxy
> > > > > > 2017-10-20 09:11:11,058 DEBUG [o.a.c.s.
> > SecondaryStorageManagerImpl]
> > > > > > (secstorage-1:ctx-5c3bea1b) (logid:dab66628) Zone 1 is ready to
> > > launch
> > > > > > secondary storage VM
> > > > > > 2017-10-20 09:11:12,160 DEBUG [c.c.a.m.AgentManagerImpl]
> > > > > > (AgentManager-Handler-9:null) (logid:) Ping from 5(v-1-VM)
> > > > > > 2017-10-20 09:11:15,239 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > (AsyncJobMgr-Heartbeat-1:ctx-60c073b3) (logid:cd140ec0) Begin
> > > cleanup
> > > > > > expired async-jobs
> > > > > > 2017-10-20 09:11:15,242 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > (AsyncJobMgr-Heartbeat-1:ctx-60c073b3) (logid:cd140ec0) End
> > cleanup
> > > > > > expired
> > > > > > async-jobs
> > > > > > 2017-10-20 09:11:15,766 DEBUG [c.c.s.StatsCollector]
> > > > > > (StatsCollector-2:ctx-5d5c1cb9) (logid:d0b73976)
> > HostStatsCollector
> > > is
> > > > > > running...
> > > > > > 2017-10-20 09:11:15,814 DEBUG [c.c.a.t.Request]
> > > > > > (StatsCollector-2:ctx-5d5c1cb9) (logid:d0b73976) Seq
> > > > > > 3-6621135877164909221:
> > > > > > Received: { Ans: , MgmtId: 264216221068220, via:
> 3(mtl1-apphst03),
> > > > Ver:
> > > > > > v1, Flags: 10, { GetHostStatsAnswer } }
> > > > > > 2017-10-20 09:11:16,065 DEBUG [c.c.a.m.AgentManagerImpl]
> > > > > > (AgentManager-Handler-2:null) (logid:) Ping from 4(s-2-VM)
> > > > > > 2017-10-20 09:11:18,715 DEBUG [c.c.a.m.AgentManagerImpl]
> > > > > > (AgentManager-Handler-7:null) (logid:) SeqA 5-377877: Processing
> > Seq
> > > > > > 5-377877: { Cmd , MgmtId: -1, via: 5, Ver: v1, Flags: 11,
> > > > > > [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand"
> > > > > > :{"_proxyVmId":1,"_loadInfo":"{\n
> > > > > > \"connections\": []\n}","wait":0}}] }
> > > > > > 2017-10-20 09:11:18,717 DEBUG [c.c.a.m.AgentManagerImpl]
> > > > > > (AgentManager-Handler-7:null) (logid:) SeqA 5-377877: Sending
Seq
> > > > > > 5-377877: { Ans: , MgmtId: 264216221068220, via: 5, Ver: v1,
> > Flags:
> > > > > > 100010,
> > > > > > [{"com.cloud.agent.api.AgentControlAnswer":{"result":
> > > true,"wait":0}}]
> > > > }
> > > > > > 2017-10-20 09:11:25,240 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > (AsyncJobMgr-Heartbeat-1:ctx-df0997a3) (logid:d8ea9924) Begin
> > > cleanup
> > > > > > expired async-jobs
> > > > > > 2017-10-20 09:11:25,243 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > (AsyncJobMgr-Heartbeat-1:ctx-df0997a3) (logid:d8ea9924) End
> > cleanup
> > > > > > expired
> > > > > > async-jobs
> > > > > > 2017-10-20 09:11:25,268 DEBUG [c.c.n.
> > ExternalDeviceUsageManagerImpl]
> > > > > > (ExternalNetworkMonitor-1:ctx-148fd210) (logid:3a3d51da)
> External
> > > > > devices
> > > > > > are not used. Skipping external device usage collection
> > > > > > 2017-10-20 09:11:25,310 INFO [c.c.h.
> HighAvailabilityManagerImpl]
> > > > > > (HA-5:ctx-f691d0f0) (logid:45cb53d0) checking health of usage
> > server
> > > > > > 2017-10-20 09:11:25,311 DEBUG [c.c.h.
> HighAvailabilityManagerImpl]
> > > > > > (HA-5:ctx-f691d0f0) (logid:45cb53d0) usage server running?
false,
> > > > > > heartbeat: null
> > > > > > 2017-10-20 09:11:25,312 WARN [o.a.c.alerts] (HA-5:ctx-f691d0f0)
> > > > > > (logid:45cb53d0) alertType:: 13 // dataCenterId:: 0 // podId:: 0
> > //
> > > > > > clusterId:: null // message:: No usage server process running
> > > > > > 2017-10-20 09:11:25,313 DEBUG [c.c.a.AlertManagerImpl]
> > > > > (HA-5:ctx-f691d0f0)
> > > > > > (logid:45cb53d0) Have already sent: 1 emails for alert type '13'
> --
> > > > > > skipping send email
> > > > > > 2017-10-20 09:11:25,338 DEBUG [c.c.n.r.
> > > VirtualNetworkApplianceManager
> > > > > Impl]
> > > > > > (RouterMonitor-1:ctx-e46df5fd) (logid:7cdfdeb5) Found 0 running
> > > > routers.
> > > > > > 2017-10-20 09:11:25,339 DEBUG [c.c.n.r.
> > > VirtualNetworkApplianceManager
> > > > > Impl]
> > > > > > (RouterStatusMonitor-1:ctx-04de4525) (logid:a35da4f6) Found 0
> > > routers
> > > > to
> > > > > > update status.
> > > > > > 2017-10-20 09:11:25,340 DEBUG [c.c.n.r.
> > > VirtualNetworkApplianceManager
> > > > > Impl]
> > > > > > (RouterStatusMonitor-1:ctx-04de4525) (logid:a35da4f6) Found 0
> VPC
> > > > > networks
> > > > > > to update Redundant State.
> > > > > > 2017-10-20 09:11:25,340 DEBUG [c.c.n.r.
> > > VirtualNetworkApplianceManager
> > > > > Impl]
> > > > > > (RouterStatusMonitor-1:ctx-04de4525) (logid:a35da4f6) Found 0
> > > networks
> > > > > to
> > > > > > update RvR status.
> > > > > > 2017-10-20 09:11:25,349 DEBUG [c.c.n.r.
> > > VirtualNetworkApplianceManager
> > > > > Impl]
> > > > > > (RouterStatusMonitor-1:ctx-47e38698) (logid:d39e2330) Found 1
> > > running
> > > > > > routers.
> > > > > > 2017-10-20 09:11:25,351 DEBUG [c.c.n.r.
> > > VirtualNetworkApplianceManager
> > > > > Impl]
> > > > > > (RouterStatusMonitor-1:ctx-47e38698) (logid:d39e2330) IP of
> > control
> > > > NIC
> > > > > > 169.254.0.113
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > With best regards, Ivan Kudryavtsev
> > > > > Bitworks Software, Ltd.
> > > > > Cell: +7-923-414-1515
> > > > > WWW: http://bitworks.software/ <http://bw-sw.com/>
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > With best regards, Ivan Kudryavtsev
> > > Bitworks Software, Ltd.
> > > Cell: +7-923-414-1515
> > > WWW: http://bitworks.software/ <http://bw-sw.com/>
> > >
> >
>
>
>
> --
> With best regards, Ivan Kudryavtsev
> Bitworks Software, Ltd.
> Cell: +7-923-414-1515
> WWW: http://bitworks.software/ <http://bw-sw.com/>
>
dag.sonst...@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
