Greetings Community users,
We are currently in the process of integrating our Linstor nodes with the
Cloudstack setup. Following a maintenance shutdown and subsequent restart
of all servers, we are encountering difficulties in starting existing
instances. It's worth noting that creating new instances is not
encountering any issues.
We are encountering two distinct errors during the instance start-up
process:
*Error 1 (Instance ID: i-2-286)*
Start instance
(OMS1) Unable to orchestrate start VM instance
{"id":286,"instanceName":"i-2-286-VM","type":"User","uuid":"6987bdf1-7b37-41e1-bdbd-1071a9b0fa18"}
due to [Unable to get answer that is of class
com.cloud.agent.api.StartAnswer].
*Error 2 (Instance ID: i-2-281)*
Start instance
(MM37) Unable to start a VM [622fb948-6194-4969-a4a4-b652c22c0ec6] due to
[Unable to create a deployment for VM instance
{"id":281,"instanceName":"i-2-281-VM","type":"User","uuid":"622fb948-6194-4969-a4a4-b652c22c0ec6"}].
We have also captured additional logs during the attempts to start the
instances. Your prompt attention and assistance in resolving these issues
would be highly appreciated.
Regards,
Palash Biswas
Dec 01 08:43:18 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
Dec 01 08:43:18 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
Dec 01 08:43:18 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
Dec 01 08:43:18 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
Dec 01 08:43:18 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
Dec 01 08:43:18 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
Dec 01 08:43:18 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
Dec 01 08:43:18 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
Dec 01 08:43:18 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Dec 01 08:43:18 n2ncs-mcs01 java[11044]: at
java.base/java.lang.Thread.run(Thread.java:829)
Dec 01 08:43:32 n2ncs-mcs01 java[11044]: INFO
[o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1278852808-11005:ctx-7bac6178
ctx-3cdf0c62) (logid:701251b3) Account [Account
[{"accountName":"admin","id":2,"uuid":"0e02266c-6350-11ee-a6a8-5254001f02f7"}]]
is Root Admin or Domain Admin, all APIs are allowed.
Dec 01 08:43:32 n2ncs-mcs01 java[11044]: WARN
[o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp1278852808-11005:ctx-7bac6178
ctx-3cdf0c62) (logid:701251b3) Project is null,
ProjectRoleBasedApiAccessChecker only applies to projects, returning API
[startVirtualMachine] for user [User
{"username":"admin","uuid":"0e02fe3b-6350-11ee-a6a8-5254001f02f7"}.] as allowed.
Dec 01 08:43:32 n2ncs-mcs01 java[11044]: INFO [o.a.c.f.j.i.AsyncJobMonitor]
(API-Job-Executor-58:ctx-493a5aef job-9239) (logid:27e59549) Add job-9239 into
job monitoring
Dec 01 08:43:32 n2ncs-mcs01 java[11044]: INFO
[o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1278852808-11127:ctx-b85372db
ctx-54efe125) (logid:d9b916c0) Account [Account
[{"accountName":"admin","id":2,"uuid":"0e02266c-6350-11ee-a6a8-5254001f02f7"}]]
is Root Admin or Domain Admin, all APIs are allowed.
Dec 01 08:43:32 n2ncs-mcs01 java[11044]: WARN
[o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp1278852808-11127:ctx-b85372db
ctx-54efe125) (logid:d9b916c0) Project is null,
ProjectRoleBasedApiAccessChecker only applies to projects, returning API
[queryAsyncJobResult] for user [User
{"username":"admin","uuid":"0e02fe3b-6350-11ee-a6a8-5254001f02f7"}.] as allowed.
Dec 01 08:43:32 n2ncs-mcs01 java[11044]: INFO [o.a.c.f.j.i.AsyncJobMonitor]
(Work-Job-Executor-107:ctx-9c7fb656 job-9239/job-9241) (logid:4ecad230) Add
job-9241 into job monitoring
Dec 01 08:43:33 n2ncs-mcs01 java[11044]: INFO
[c.c.n.e.VpcVirtualRouterElement] (Work-Job-Executor-107:ctx-9c7fb656
job-9239/job-9241 ctx-cd2e5e01) (logid:7afe088a) Adding VPC routers to Guest
Network: 1 to be added!
Dec 01 08:43:35 n2ncs-mcs01 java[11044]: INFO
[o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1278852808-11005:ctx-52d07839
ctx-aed1ecd0) (logid:64e14161) Account [Account
[{"accountName":"admin","id":2,"uuid":"0e02266c-6350-11ee-a6a8-5254001f02f7"}]]
is Root Admin or Domain Admin, all APIs are allowed.
Dec 01 08:43:35 n2ncs-mcs01 java[11044]: WARN
[o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp1278852808-11005:ctx-52d07839
ctx-aed1ecd0) (logid:64e14161) Project is null,
ProjectRoleBasedApiAccessChecker only applies to projects, returning API
[queryAsyncJobResult] for user [User
{"username":"admin","uuid":"0e02fe3b-6350-11ee-a6a8-5254001f02f7"}.] as allowed.
Dec 01 08:43:35 n2ncs-mcs01 java[11044]: INFO
[c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-107:ctx-9c7fb656
job-9239/job-9241 ctx-cd2e5e01) (logid:7afe088a) Unable to start VM on Host
{"id":3,"name":"n2ncs-c02","type":"Routing","uuid":"77defbe3-794f-407c-84d0-520db2e27708"}
due to missing source information for device vda
Dec 01 08:43:35 n2ncs-mcs01 java[11044]: ERROR [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-107:ctx-9c7fb656 job-9239/job-9241 ctx-cd2e5e01)
(logid:7afe088a) Invocation exception, caused by:
com.cloud.exception.InsufficientServerCapacityException: Unable to create a
deployment for VM instance
{"id":281,"instanceName":"i-2-281-VM","type":"User","uuid":"622fb948-6194-4969-a4a4-b652c22c0ec6"}Scope=interface
com.cloud.dc.DataCenter; id=1
Dec 01 08:43:35 n2ncs-mcs01 java[11044]: INFO [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-107:ctx-9c7fb656 job-9239/job-9241 ctx-cd2e5e01)
(logid:7afe088a) Rethrow exception
com.cloud.exception.InsufficientServerCapacityException: Unable to create a
deployment for VM instance
{"id":281,"instanceName":"i-2-281-VM","type":"User","uuid":"622fb948-6194-4969-a4a4-b652c22c0ec6"}Scope=interface
com.cloud.dc.DataCenter; id=1
Dec 01 08:43:35 n2ncs-mcs01 java[11044]: ERROR [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-107:ctx-9c7fb656 job-9239/job-9241) (logid:7afe088a) Unable
to complete AsyncJobVO: {id:9241, userId: 2, accountId: 2, instanceType: null,
instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo:
rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAABGXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAXBzcgAOamF2YS5sYW5nLkxvbmc7i-SQzI8j3wIAAUoABXZhbHVleHIAEGphdmEubGFuZy5OdW1iZXKGrJUdC5TgiwIAAHhwAAAAAAAAAAFzcQB-AAgAAAAAAAAAA3BwcHEAfgAKcHNyABFqYXZhLnV0aWwuSGFzaE1hcAUH2sHDFmDRAwACRgAKbG9hZEZhY3RvckkACXRocmVzaG9sZHhwP0AAAAAAAAx3CAAAABAAAAABdAAKVm1QYXNzd29yZHQAHHJPMEFCWFFBRG5OaGRtVmtYM0JoYzNOM2IzSmt4cA,
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
null, initMsid: 90520746422272, completeMsid: null, lastUpdated: null,
lastPolled: null, created: Fri Dec 01 08:43:32 MYT 2023, removed: null}, job
origin:9239
Dec 01 08:43:35 n2ncs-mcs01 java[11044]:
com.cloud.exception.InsufficientServerCapacityException: Unable to create a
deployment for VM instance
{"id":281,"instanceName":"i-2-281-VM","type":"User","uuid":"622fb948-6194-4969-a4a4-b652c22c0ec6"}Scope=interface
com.cloud.dc.DataCenter; id=1
Dec 01 08:43:35 n2ncs-mcs01 java[11044]: at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1226)
Dec 01 08:43:35 n2ncs-mcs01 java[11044]: at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5412)
Dec 01 08:43:35 n2ncs-mcs01 java[11044]: at
jdk.internal.reflect.GeneratedMethodAccessor787.invoke(Unknown Source)
Dec 01 08:43:35 n2ncs-mcs01 java[11044]: at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Dec 01 08:43:35 n2ncs-mcs01 java[11044]: at
java.base/java.lang.reflect.Method.invoke(Method.java:566)
Dec 01 08:43:35 n2ncs-mcs01 java[11044]: at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
Dec 01 08:43:35 n2ncs-mcs01 java[11044]: at
com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5536)
Dec 01 08:43:35 n2ncs-mcs01 java[11044]: at
com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
Dec 01 08:43:35 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
Dec 01 08:43:35 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
Dec 01 08:43:35 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
Dec 01 08:43:35 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
Dec 01 08:43:35 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
Dec 01 08:43:35 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
Dec 01 08:43:35 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
Dec 01 08:43:35 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
Dec 01 08:43:35 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
Dec 01 08:43:35 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
Dec 01 08:43:35 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Dec 01 08:43:35 n2ncs-mcs01 java[11044]: at
java.base/java.lang.Thread.run(Thread.java:829)
Dec 01 08:43:35 n2ncs-mcs01 java[11044]: INFO [o.a.c.f.j.i.AsyncJobMonitor]
(Work-Job-Executor-107:ctx-9c7fb656 job-9239/job-9241) (logid:7afe088a) Remove
job-9241 from job monitoring
Dec 01 08:43:36 n2ncs-mcs01 java[11044]: INFO [o.a.c.f.j.i.AsyncJobMonitor]
(Work-Job-Executor-108:ctx-f581ebda job-9239/job-9243) (logid:f2f7faaf) Add
job-9243 into job monitoring
Dec 01 08:43:37 n2ncs-mcs01 java[11044]: INFO
[c.c.n.e.VpcVirtualRouterElement] (Work-Job-Executor-108:ctx-f581ebda
job-9239/job-9243 ctx-5049c834) (logid:7afe088a) Adding VPC routers to Guest
Network: 1 to be added!
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: INFO [c.c.n.NetworkModelImpl]
(VmGroup-Monitor-21-1:ctx-dfe46a3f) (logid:693b8340) Checking permission for
account PrjAcct-CGS-3 (06859235-cd2e-4506-89e3-75c0ffe53912) on network
n2n-bs-prod-cgs-sg-private_subnet01 (be8521f0-9ffa-46b4-bad9-eb0b6160e253)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: ERROR [c.c.r.ResourceLimitManagerImpl]
(VmGroup-Monitor-21-1:ctx-dfe46a3f) (logid:693b8340) Maximum amount of
resources of Type = 'primary_storage' for Project Name = CGS in Domain Id = 3
is exceeded: Account Resource Limit = (4.0000 TB) 4398046511104, Current
Account Resource Amount = (3.8086 TB) 4187607060480, Current Account Resource
Reservation = (0 bytes) 0, Requested Resource Amount = (250.00 GB) 268436779008.
Dec 01 08:43:38 n2ncs-mcs01 java[11044]:
com.cloud.exception.ResourceAllocationException: Maximum amount of resources of
Type = 'primary_storage' for Project Name = CGS in Domain Id = 3 is exceeded:
Account Resource Limit = (4.0000 TB) 4398046511104, Current Account Resource
Amount = (3.8086 TB) 4187607060480, Current Account Resource Reservation = (0
bytes) 0, Requested Resource Amount = (250.00 GB) 268436779008.
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
com.cloud.resourcelimit.ResourceLimitManagerImpl.checkAccountResourceLimit(ResourceLimitManagerImpl.java:516)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
com.cloud.resourcelimit.ResourceLimitManagerImpl$2.doInTransactionWithoutResult(ResourceLimitManagerImpl.java:584)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:25)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:21)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
com.cloud.utils.db.Transaction.execute(Transaction.java:40)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
com.cloud.resourcelimit.ResourceLimitManagerImpl.checkResourceLimit(ResourceLimitManagerImpl.java:578)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
com.cloud.resourcelimit.CheckedReservation.<init>(CheckedReservation.java:76)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
com.cloud.vm.UserVmManagerImpl.getUncheckedUserVmResource(UserVmManagerImpl.java:3987)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
com.cloud.vm.UserVmManagerImpl.getCheckedUserVmResource(UserVmManagerImpl.java:3972)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
com.cloud.vm.UserVmManagerImpl.createVirtualMachine(UserVmManagerImpl.java:3955)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
com.cloud.vm.UserVmManagerImpl.createAdvancedVirtualMachine(UserVmManagerImpl.java:3729)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
jdk.internal.reflect.GeneratedMethodAccessor336.invoke(Unknown Source)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
java.base/java.lang.reflect.Method.invoke(Method.java:566)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
com.sun.proxy.$Proxy189.createAdvancedVirtualMachine(Unknown Source)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
com.cloud.network.as.AutoScaleManagerImpl.createNewVM(AutoScaleManagerImpl.java:1812)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
com.cloud.network.as.AutoScaleManagerImpl.doScaleUp(AutoScaleManagerImpl.java:2054)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
com.cloud.network.as.AutoScaleManagerImpl.checkAsGroupMaxAndMinMembers(AutoScaleManagerImpl.java:2444)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
com.cloud.network.as.AutoScaleManagerImpl.monitorVirtualRouterAsGroup(AutoScaleManagerImpl.java:2624)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
com.cloud.network.as.AutoScaleManagerImpl$MonitorTask.runInContext(AutoScaleManagerImpl.java:2996)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
java.base/java.lang.Thread.run(Thread.java:829)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: WARN [c.c.n.a.AutoScaleManagerImpl]
(VmGroup-Monitor-21-1:ctx-dfe46a3f) (logid:693b8340) Exception:
Dec 01 08:43:38 n2ncs-mcs01 java[11044]:
com.cloud.exception.ResourceAllocationException: Maximum amount of resources of
Type = 'primary_storage' for Project Name = CGS in Domain Id = 3 is exceeded:
Account Resource Limit = (4.0000 TB) 4398046511104, Current Account Resource
Amount = (3.8086 TB) 4187607060480, Current Account Resource Reservation = (0
bytes) 0, Requested Resource Amount = (250.00 GB) 268436779008.
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
com.cloud.resourcelimit.ResourceLimitManagerImpl.checkAccountResourceLimit(ResourceLimitManagerImpl.java:516)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
com.cloud.resourcelimit.ResourceLimitManagerImpl$2.doInTransactionWithoutResult(ResourceLimitManagerImpl.java:584)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:25)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:21)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
com.cloud.utils.db.Transaction.execute(Transaction.java:40)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
com.cloud.resourcelimit.ResourceLimitManagerImpl.checkResourceLimit(ResourceLimitManagerImpl.java:578)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
com.cloud.resourcelimit.CheckedReservation.<init>(CheckedReservation.java:76)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
com.cloud.vm.UserVmManagerImpl.getUncheckedUserVmResource(UserVmManagerImpl.java:3987)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
com.cloud.vm.UserVmManagerImpl.getCheckedUserVmResource(UserVmManagerImpl.java:3972)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
com.cloud.vm.UserVmManagerImpl.createVirtualMachine(UserVmManagerImpl.java:3955)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
com.cloud.vm.UserVmManagerImpl.createAdvancedVirtualMachine(UserVmManagerImpl.java:3729)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
jdk.internal.reflect.GeneratedMethodAccessor336.invoke(Unknown Source)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
java.base/java.lang.reflect.Method.invoke(Method.java:566)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
com.sun.proxy.$Proxy189.createAdvancedVirtualMachine(Unknown Source)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
com.cloud.network.as.AutoScaleManagerImpl.createNewVM(AutoScaleManagerImpl.java:1812)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
com.cloud.network.as.AutoScaleManagerImpl.doScaleUp(AutoScaleManagerImpl.java:2054)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
com.cloud.network.as.AutoScaleManagerImpl.checkAsGroupMaxAndMinMembers(AutoScaleManagerImpl.java:2444)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
com.cloud.network.as.AutoScaleManagerImpl.monitorVirtualRouterAsGroup(AutoScaleManagerImpl.java:2624)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
com.cloud.network.as.AutoScaleManagerImpl$MonitorTask.runInContext(AutoScaleManagerImpl.java:2996)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
java.base/java.lang.Thread.run(Thread.java:829)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: WARN [c.c.n.a.AutoScaleManagerImpl]
(VmGroup-Monitor-21-1:ctx-dfe46a3f) (logid:693b8340) Caught the following
exception on monitoring AutoScale Vm Group
Dec 01 08:43:38 n2ncs-mcs01 java[11044]:
org.apache.cloudstack.api.ServerApiException: Maximum amount of resources of
Type = 'primary_storage' for Project Name = CGS in Domain Id = 3 is exceeded:
Account Resource Limit = (4.0000 TB) 4398046511104, Current Account Resource
Amount = (3.8086 TB) 4187607060480, Current Account Resource Reservation = (0
bytes) 0, Requested Resource Amount = (250.00 GB) 268436779008.
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
com.cloud.network.as.AutoScaleManagerImpl.createNewVM(AutoScaleManagerImpl.java:1837)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
com.cloud.network.as.AutoScaleManagerImpl.doScaleUp(AutoScaleManagerImpl.java:2054)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
com.cloud.network.as.AutoScaleManagerImpl.checkAsGroupMaxAndMinMembers(AutoScaleManagerImpl.java:2444)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
com.cloud.network.as.AutoScaleManagerImpl.monitorVirtualRouterAsGroup(AutoScaleManagerImpl.java:2624)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
com.cloud.network.as.AutoScaleManagerImpl$MonitorTask.runInContext(AutoScaleManagerImpl.java:2996)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: at
java.base/java.lang.Thread.run(Thread.java:829)
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: INFO
[o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1278852808-11127:ctx-1b525749
ctx-3919e079) (logid:28b6a221) Account [Account
[{"accountName":"admin","id":2,"uuid":"0e02266c-6350-11ee-a6a8-5254001f02f7"}]]
is Root Admin or Domain Admin, all APIs are allowed.
Dec 01 08:43:38 n2ncs-mcs01 java[11044]: WARN
[o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp1278852808-11127:ctx-1b525749
ctx-3919e079) (logid:28b6a221) Project is null,
ProjectRoleBasedApiAccessChecker only applies to projects, returning API
[queryAsyncJobResult] for user [User
{"username":"admin","uuid":"0e02fe3b-6350-11ee-a6a8-5254001f02f7"}.] as allowed.
Dec 01 08:43:39 n2ncs-mcs01 java[11044]: INFO
[c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-108:ctx-f581ebda
job-9239/job-9243 ctx-5049c834) (logid:7afe088a) Unable to start VM on Host
{"id":3,"name":"n2ncs-c02","type":"Routing","uuid":"77defbe3-794f-407c-84d0-520db2e27708"}
due to missing source information for device vda
Dec 01 08:43:39 n2ncs-mcs01 java[11044]: INFO [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-108:ctx-f581ebda job-9239/job-9243 ctx-5049c834
FirstFitRoutingAllocator) (logid:7afe088a) Guest VM is requested with
Custom[UEFI] Boot Type false
Dec 01 08:43:39 n2ncs-mcs01 java[11044]: INFO
[c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-108:ctx-f581ebda
job-9239/job-9243 ctx-5049c834) (logid:7afe088a) Re-ordering hosts [Host
{"id":5,"name":"n2ncs-c03","type":"Routing","uuid":"ba19c0fd-259c-41f6-af63-8da065f0b677"}]
by priorities {}
Dec 01 08:43:39 n2ncs-mcs01 java[11044]: INFO
[c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-108:ctx-f581ebda
job-9239/job-9243 ctx-5049c834) (logid:7afe088a) Hosts after re-ordering are:
[Host
{"id":5,"name":"n2ncs-c03","type":"Routing","uuid":"ba19c0fd-259c-41f6-af63-8da065f0b677"}]
Dec 01 08:43:39 n2ncs-mcs01 java[11044]: INFO
[c.c.n.e.VpcVirtualRouterElement] (Work-Job-Executor-108:ctx-f581ebda
job-9239/job-9243 ctx-5049c834) (logid:7afe088a) Adding VPC routers to Guest
Network: 1 to be added!
Dec 01 08:43:41 n2ncs-mcs01 java[11044]: INFO
[o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1278852808-11005:ctx-4a67a274
ctx-d7083585) (logid:6e99c8c9) Account [Account
[{"accountName":"admin","id":2,"uuid":"0e02266c-6350-11ee-a6a8-5254001f02f7"}]]
is Root Admin or Domain Admin, all APIs are allowed.
Dec 01 08:43:41 n2ncs-mcs01 java[11044]: WARN
[o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp1278852808-11005:ctx-4a67a274
ctx-d7083585) (logid:6e99c8c9) Project is null,
ProjectRoleBasedApiAccessChecker only applies to projects, returning API
[queryAsyncJobResult] for user [User
{"username":"admin","uuid":"0e02fe3b-6350-11ee-a6a8-5254001f02f7"}.] as allowed.
Dec 01 08:43:41 n2ncs-mcs01 java[11044]: INFO
[c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-108:ctx-f581ebda
job-9239/job-9243 ctx-5049c834) (logid:7afe088a) Unable to start VM on Host
{"id":5,"name":"n2ncs-c03","type":"Routing","uuid":"ba19c0fd-259c-41f6-af63-8da065f0b677"}
due to missing source information for device vda
Dec 01 08:43:41 n2ncs-mcs01 java[11044]: INFO [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-108:ctx-f581ebda job-9239/job-9243 ctx-5049c834
FirstFitRoutingAllocator) (logid:7afe088a) Guest VM is requested with
Custom[UEFI] Boot Type false
Dec 01 08:43:41 n2ncs-mcs01 java[11044]: INFO
[c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-108:ctx-f581ebda
job-9239/job-9243 ctx-5049c834) (logid:7afe088a) Re-ordering hosts [] by
priorities {}
Dec 01 08:43:41 n2ncs-mcs01 java[11044]: INFO
[c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-108:ctx-f581ebda
job-9239/job-9243 ctx-5049c834) (logid:7afe088a) Hosts after re-ordering are: []
Dec 01 08:43:41 n2ncs-mcs01 java[11044]: INFO [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-108:ctx-f581ebda job-9239/job-9243 ctx-5049c834
FirstFitRoutingAllocator) (logid:7afe088a) Guest VM is requested with
Custom[UEFI] Boot Type false
Dec 01 08:43:41 n2ncs-mcs01 java[11044]: INFO
[c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-108:ctx-f581ebda
job-9239/job-9243 ctx-5049c834) (logid:7afe088a) Re-ordering hosts [Host
{"id":13,"name":"n2ncs-c05","type":"Routing","uuid":"1d1ed762-ad39-4ce7-a9ca-42057e8b1a6b"},
Host
{"id":7,"name":"n2ncs-c04","type":"Routing","uuid":"3c9bb524-fbfe-49de-9190-248251f96d4e"},
Host
{"id":15,"name":"n2ncs-c06","type":"Routing","uuid":"0f518498-d995-4d1b-a938-2d9ca8d433bd"}]
by priorities {}
Dec 01 08:43:41 n2ncs-mcs01 java[11044]: INFO
[c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-108:ctx-f581ebda
job-9239/job-9243 ctx-5049c834) (logid:7afe088a) Hosts after re-ordering are:
[Host
{"id":13,"name":"n2ncs-c05","type":"Routing","uuid":"1d1ed762-ad39-4ce7-a9ca-42057e8b1a6b"},
Host
{"id":7,"name":"n2ncs-c04","type":"Routing","uuid":"3c9bb524-fbfe-49de-9190-248251f96d4e"},
Host
{"id":15,"name":"n2ncs-c06","type":"Routing","uuid":"0f518498-d995-4d1b-a938-2d9ca8d433bd"}]
Dec 01 08:43:42 n2ncs-mcs01 java[11044]: INFO
[c.c.n.e.VpcVirtualRouterElement] (Work-Job-Executor-108:ctx-f581ebda
job-9239/job-9243 ctx-5049c834) (logid:7afe088a) Adding VPC routers to Guest
Network: 1 to be added!
Dec 01 08:43:43 n2ncs-mcs01 java[11044]: INFO
[c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-108:ctx-f581ebda
job-9239/job-9243 ctx-5049c834) (logid:7afe088a) Unable to start VM on Host
{"id":13,"name":"n2ncs-c05","type":"Routing","uuid":"1d1ed762-ad39-4ce7-a9ca-42057e8b1a6b"}
due to missing source information for device vda
Dec 01 08:43:44 n2ncs-mcs01 java[11044]: INFO [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-108:ctx-f581ebda job-9239/job-9243 ctx-5049c834
FirstFitRoutingAllocator) (logid:7afe088a) Guest VM is requested with
Custom[UEFI] Boot Type false
Dec 01 08:43:44 n2ncs-mcs01 java[11044]: INFO
[c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-108:ctx-f581ebda
job-9239/job-9243 ctx-5049c834) (logid:7afe088a) Re-ordering hosts [Host
{"id":15,"name":"n2ncs-c06","type":"Routing","uuid":"0f518498-d995-4d1b-a938-2d9ca8d433bd"},
Host
{"id":7,"name":"n2ncs-c04","type":"Routing","uuid":"3c9bb524-fbfe-49de-9190-248251f96d4e"}]
by priorities {}
Dec 01 08:43:44 n2ncs-mcs01 java[11044]: INFO
[c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-108:ctx-f581ebda
job-9239/job-9243 ctx-5049c834) (logid:7afe088a) Hosts after re-ordering are:
[Host
{"id":15,"name":"n2ncs-c06","type":"Routing","uuid":"0f518498-d995-4d1b-a938-2d9ca8d433bd"},
Host
{"id":7,"name":"n2ncs-c04","type":"Routing","uuid":"3c9bb524-fbfe-49de-9190-248251f96d4e"}]
Dec 01 08:43:44 n2ncs-mcs01 java[11044]: INFO
[c.c.n.e.VpcVirtualRouterElement] (Work-Job-Executor-108:ctx-f581ebda
job-9239/job-9243 ctx-5049c834) (logid:7afe088a) Adding VPC routers to Guest
Network: 1 to be added!
Dec 01 08:43:44 n2ncs-mcs01 java[11044]: INFO
[o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1278852808-11127:ctx-b8292844
ctx-656467d1) (logid:24224bea) Account [Account
[{"accountName":"admin","id":2,"uuid":"0e02266c-6350-11ee-a6a8-5254001f02f7"}]]
is Root Admin or Domain Admin, all APIs are allowed.
Dec 01 08:43:44 n2ncs-mcs01 java[11044]: WARN
[o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp1278852808-11127:ctx-b8292844
ctx-656467d1) (logid:24224bea) Project is null,
ProjectRoleBasedApiAccessChecker only applies to projects, returning API
[queryAsyncJobResult] for user [User
{"username":"admin","uuid":"0e02fe3b-6350-11ee-a6a8-5254001f02f7"}.] as allowed.
Dec 01 08:43:46 n2ncs-mcs01 java[11044]: INFO
[c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-108:ctx-f581ebda
job-9239/job-9243 ctx-5049c834) (logid:7afe088a) Unable to start VM on Host
{"id":15,"name":"n2ncs-c06","type":"Routing","uuid":"0f518498-d995-4d1b-a938-2d9ca8d433bd"}
due to missing source information for device vda
Dec 01 08:43:46 n2ncs-mcs01 java[11044]: INFO [c.c.s.StatsCollector]
(StatsCollector-4:ctx-e3ee062e) (logid:d2862604) system memory from /proc:
16765149184
Dec 01 08:43:46 n2ncs-mcs01 java[11044]: INFO [c.c.s.StatsCollector]
(StatsCollector-4:ctx-e3ee062e) (logid:d2862604) free memory from /proc:
12214415360
Dec 01 08:43:46 n2ncs-mcs01 java[11044]: INFO [c.c.s.StatsCollector]
(StatsCollector-4:ctx-e3ee062e) (logid:d2862604) used memory from /proc: 1202964
Dec 01 08:43:46 n2ncs-mcs01 java[11044]: ERROR
[c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:ctx-d1bdc377)
(logid:60aef995) IOException from : http://10.23.1.113:9090/clusterservice,
method : name=method, value=5
Dec 01 08:43:46 n2ncs-mcs01 java[11044]: ERROR
[c.c.c.ClusterServiceServletImpl] (Cluster-Worker-5:ctx-d1bdc377)
(logid:60aef995) IOException from : http://10.23.1.113:9090/clusterservice,
method : name=method, value=5
Dec 01 08:43:46 n2ncs-mcs01 java[11044]: INFO [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-108:ctx-f581ebda job-9239/job-9243 ctx-5049c834
FirstFitRoutingAllocator) (logid:7afe088a) Guest VM is requested with
Custom[UEFI] Boot Type false
Dec 01 08:43:46 n2ncs-mcs01 java[11044]: INFO
[c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-108:ctx-f581ebda
job-9239/job-9243 ctx-5049c834) (logid:7afe088a) Re-ordering hosts [Host
{"id":7,"name":"n2ncs-c04","type":"Routing","uuid":"3c9bb524-fbfe-49de-9190-248251f96d4e"}]
by priorities {}
Dec 01 08:43:46 n2ncs-mcs01 java[11044]: INFO
[c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-108:ctx-f581ebda
job-9239/job-9243 ctx-5049c834) (logid:7afe088a) Hosts after re-ordering are:
[Host
{"id":7,"name":"n2ncs-c04","type":"Routing","uuid":"3c9bb524-fbfe-49de-9190-248251f96d4e"}]
Dec 01 08:43:46 n2ncs-mcs01 java[11044]: INFO
[c.c.n.e.VpcVirtualRouterElement] (Work-Job-Executor-108:ctx-f581ebda
job-9239/job-9243 ctx-5049c834) (logid:7afe088a) Adding VPC routers to Guest
Network: 1 to be added!
Dec 01 08:43:47 n2ncs-mcs01 java[11044]: INFO
[o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1278852808-11005:ctx-050660c3
ctx-40d5c949) (logid:903977d5) Account [Account
[{"accountName":"admin","id":2,"uuid":"0e02266c-6350-11ee-a6a8-5254001f02f7"}]]
is Root Admin or Domain Admin, all APIs are allowed.
Dec 01 08:43:47 n2ncs-mcs01 java[11044]: WARN
[o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp1278852808-11005:ctx-050660c3
ctx-40d5c949) (logid:903977d5) Project is null,
ProjectRoleBasedApiAccessChecker only applies to projects, returning API
[queryAsyncJobResult] for user [User
{"username":"admin","uuid":"0e02fe3b-6350-11ee-a6a8-5254001f02f7"}.] as allowed.
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: INFO
[c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-108:ctx-f581ebda
job-9239/job-9243 ctx-5049c834) (logid:7afe088a) Unable to start VM on Host
{"id":7,"name":"n2ncs-c04","type":"Routing","uuid":"3c9bb524-fbfe-49de-9190-248251f96d4e"}
due to missing source information for device vda
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: INFO [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-108:ctx-f581ebda job-9239/job-9243 ctx-5049c834
FirstFitRoutingAllocator) (logid:7afe088a) Guest VM is requested with
Custom[UEFI] Boot Type false
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: INFO
[c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-108:ctx-f581ebda
job-9239/job-9243 ctx-5049c834) (logid:7afe088a) Re-ordering hosts [] by
priorities {}
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: INFO
[c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-108:ctx-f581ebda
job-9239/job-9243 ctx-5049c834) (logid:7afe088a) Hosts after re-ordering are: []
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: ERROR [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-108:ctx-f581ebda job-9239/job-9243 ctx-5049c834)
(logid:7afe088a) Invocation exception, caused by:
com.cloud.exception.InsufficientServerCapacityException: Unable to create a
deployment for VM instance
{"id":281,"instanceName":"i-2-281-VM","type":"User","uuid":"622fb948-6194-4969-a4a4-b652c22c0ec6"}Scope=interface
com.cloud.dc.DataCenter; id=1
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: INFO [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-108:ctx-f581ebda job-9239/job-9243 ctx-5049c834)
(logid:7afe088a) Rethrow exception
com.cloud.exception.InsufficientServerCapacityException: Unable to create a
deployment for VM instance
{"id":281,"instanceName":"i-2-281-VM","type":"User","uuid":"622fb948-6194-4969-a4a4-b652c22c0ec6"}Scope=interface
com.cloud.dc.DataCenter; id=1
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: ERROR [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-108:ctx-f581ebda job-9239/job-9243) (logid:7afe088a) Unable
to complete AsyncJobVO: {id:9243, userId: 2, accountId: 2, instanceType: null,
instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo:
rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAABGXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBwc3IAEWphdmEudXRpbC5IYXNoTWFwBQfawcMWYNEDAAJGAApsb2FkRmFjdG9ySQAJdGhyZXNob2xkeHA_QAAAAAAADHcIAAAAEAAAAAF0AApWbVBhc3N3b3JkdAAcck8wQUJYUUFEbk5oZG1Wa1gzQmhjM04zYjNKa3hw,
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
null, initMsid: 90520746422272, completeMsid: null, lastUpdated: null,
lastPolled: null, created: Fri Dec 01 08:43:35 MYT 2023, removed: null}, job
origin:9239
Dec 01 08:43:48 n2ncs-mcs01 java[11044]:
com.cloud.exception.InsufficientServerCapacityException: Unable to create a
deployment for VM instance
{"id":281,"instanceName":"i-2-281-VM","type":"User","uuid":"622fb948-6194-4969-a4a4-b652c22c0ec6"}Scope=interface
com.cloud.dc.DataCenter; id=1
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1226)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5412)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
jdk.internal.reflect.GeneratedMethodAccessor787.invoke(Unknown Source)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
java.base/java.lang.reflect.Method.invoke(Method.java:566)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5536)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
java.base/java.lang.Thread.run(Thread.java:829)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: INFO [o.a.c.f.j.i.AsyncJobMonitor]
(Work-Job-Executor-108:ctx-f581ebda job-9239/job-9243) (logid:7afe088a) Remove
job-9243 from job monitoring
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: ERROR [c.c.a.ApiAsyncJobDispatcher]
(API-Job-Executor-58:ctx-493a5aef job-9239) (logid:7afe088a) Unexpected
exception while executing
org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin
Dec 01 08:43:48 n2ncs-mcs01 java[11044]:
com.cloud.utils.exception.CloudRuntimeException: Unable to start a VM
[622fb948-6194-4969-a4a4-b652c22c0ec6] due to [Unable to create a deployment
for VM instance
{"id":281,"instanceName":"i-2-281-VM","type":"User","uuid":"622fb948-6194-4969-a4a4-b652c22c0ec6"}].
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:841)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:258)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:214)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:5401)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:5251)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3179)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
jdk.internal.reflect.GeneratedMethodAccessor776.invoke(Unknown Source)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
java.base/java.lang.reflect.Method.invoke(Method.java:566)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
com.sun.proxy.$Proxy189.startVirtualMachine(Unknown Source)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.api.command.user.vm.StartVMCmd.execute(StartVMCmd.java:181)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:163)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:112)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
java.base/java.lang.Thread.run(Thread.java:829)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: Caused by:
com.cloud.exception.InsufficientServerCapacityException: Unable to create a
deployment for VM instance
{"id":281,"instanceName":"i-2-281-VM","type":"User","uuid":"622fb948-6194-4969-a4a4-b652c22c0ec6"}Scope=interface
com.cloud.dc.DataCenter; id=1
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1226)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5412)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: at
jdk.internal.reflect.GeneratedMethodAccessor787.invoke(Unknown Source)
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: ... 17 more
Dec 01 08:43:48 n2ncs-mcs01 java[11044]: INFO [o.a.c.f.j.i.AsyncJobMonitor]
(API-Job-Executor-58:ctx-493a5aef job-9239) (logid:7afe088a) Remove job-9239
from job monitoring
Dec 01 08:43:50 n2ncs-mcs01 java[11044]: INFO
[o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1278852808-11127:ctx-979db7d2
ctx-4c3c8e63) (logid:8e92ebe7) Account [Account
[{"accountName":"admin","id":2,"uuid":"0e02266c-6350-11ee-a6a8-5254001f02f7"}]]
is Root Admin or Domain Admin, all APIs are allowed.
Dec 01 08:43:50 n2ncs-mcs01 java[11044]: WARN
[o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp1278852808-11127:ctx-979db7d2
ctx-4c3c8e63) (logid:8e92ebe7) Project is null,
ProjectRoleBasedApiAccessChecker only applies to projects, returning API
[queryAsyncJobResult] for user [User
{"username":"admin","uuid":"0e02fe3b-6350-11ee-a6a8-5254001f02f7"}.] as allowed.
Dec 01 08:43:50 n2ncs-mcs01 java[11044]: INFO
[o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1278852808-11005:ctx-c3db1bd6
ctx-794f7009) (logid:4343399f) Account [Account
[{"accountName":"admin","id":2,"uuid":"0e02266c-6350-11ee-a6a8-5254001f02f7"}]]
is Root Admin or Domain Admin, all APIs are allowed.
Dec 01 08:43:50 n2ncs-mcs01 java[11044]: INFO
[o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1278852808-11074:ctx-04f48f2f
ctx-42d1e1df) (logid:30adcc61) Account [Account
[{"accountName":"admin","id":2,"uuid":"0e02266c-6350-11ee-a6a8-5254001f02f7"}]]
is Root Admin or Domain Admin, all APIs are allowed.
Dec 01 08:43:50 n2ncs-mcs01 java[11044]: WARN
[o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp1278852808-11005:ctx-c3db1bd6
ctx-794f7009) (logid:4343399f) Project is null,
ProjectRoleBasedApiAccessChecker only applies to projects, returning API
[listVirtualMachinesMetrics] for user [User
{"username":"admin","uuid":"0e02fe3b-6350-11ee-a6a8-5254001f02f7"}.] as allowed.
Dec 01 08:43:50 n2ncs-mcs01 java[11044]: WARN
[o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp1278852808-11074:ctx-04f48f2f
ctx-42d1e1df) (logid:30adcc61) Project is null,
ProjectRoleBasedApiAccessChecker only applies to projects, returning API
[listVirtualMachinesMetrics] for user [User
{"username":"admin","uuid":"0e02fe3b-6350-11ee-a6a8-5254001f02f7"}.] as allowed.
Dec 01 08:43:50 n2ncs-mcs01 java[11044]: INFO
[o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1278852808-11191:ctx-59849484
ctx-b7284ccb) (logid:0516dca8) Account [Account
[{"accountName":"admin","id":2,"uuid":"0e02266c-6350-11ee-a6a8-5254001f02f7"}]]
is Root Admin or Domain Admin, all APIs are allowed.
Dec 01 08:43:50 n2ncs-mcs01 java[11044]: WARN
[o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp1278852808-11191:ctx-59849484
ctx-b7284ccb) (logid:0516dca8) Project is null,
ProjectRoleBasedApiAccessChecker only applies to projects, returning API
[listVirtualMachinesMetrics] for user [User
{"username":"admin","uuid":"0e02fe3b-6350-11ee-a6a8-5254001f02f7"}.] as allowed.
Dec 01 08:43:50 n2ncs-mcs01 java[11044]: INFO
[o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1278852808-11074:ctx-f9f36bbf
ctx-ecdec4f5) (logid:b48ce5bb) Account [Account
[{"accountName":"admin","id":2,"uuid":"0e02266c-6350-11ee-a6a8-5254001f02f7"}]]
is Root Admin or Domain Admin, all APIs are allowed.
Dec 01 08:43:50 n2ncs-mcs01 java[11044]: WARN
[o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp1278852808-11074:ctx-f9f36bbf
ctx-ecdec4f5) (logid:b48ce5bb) Project is null,
ProjectRoleBasedApiAccessChecker only applies to projects, returning API
[listVirtualMachinesMetrics] for user [User
{"username":"admin","uuid":"0e02fe3b-6350-11ee-a6a8-5254001f02f7"}.] as allowed.
Dec 01 08:43:51 n2ncs-mcs01 java[11044]: INFO
[o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1278852808-11191:ctx-0d98e05c
ctx-f2a2d747) (logid:1eaa630e) Account [Account
[{"accountName":"admin","id":2,"uuid":"0e02266c-6350-11ee-a6a8-5254001f02f7"}]]
is Root Admin or Domain Admin, all APIs are allowed.
Dec 01 08:43:51 n2ncs-mcs01 java[11044]: WARN
[o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp1278852808-11191:ctx-0d98e05c
ctx-f2a2d747) (logid:1eaa630e) Project is null,
ProjectRoleBasedApiAccessChecker only applies to projects, returning API
[listVirtualMachinesMetrics] for user [User
{"username":"admin","uuid":"0e02fe3b-6350-11ee-a6a8-5254001f02f7"}.] as allowed.
Dec 01 08:43:51 n2ncs-mcs01 java[11044]: INFO
[o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1278852808-11074:ctx-3accd589
ctx-79a7e17f) (logid:2b9e986a) Account [Account
[{"accountName":"admin","id":2,"uuid":"0e02266c-6350-11ee-a6a8-5254001f02f7"}]]
is Root Admin or Domain Admin, all APIs are allowed.
Dec 01 08:43:51 n2ncs-mcs01 java[11044]: WARN
[o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp1278852808-11074:ctx-3accd589
ctx-79a7e17f) (logid:2b9e986a) Project is null,
ProjectRoleBasedApiAccessChecker only applies to projects, returning API
[listVirtualMachinesMetrics] for user [User
{"username":"admin","uuid":"0e02fe3b-6350-11ee-a6a8-5254001f02f7"}.] as allowed.
Dec 01 08:43:51 n2ncs-mcs01 java[11044]: INFO
[o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1278852808-11191:ctx-8ee75159
ctx-4ca5b071) (logid:5f1c8bc5) Account [Account
[{"accountName":"admin","id":2,"uuid":"0e02266c-6350-11ee-a6a8-5254001f02f7"}]]
is Root Admin or Domain Admin, all APIs are allowed.
Dec 01 08:43:51 n2ncs-mcs01 java[11044]: WARN
[o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp1278852808-11191:ctx-8ee75159
ctx-4ca5b071) (logid:5f1c8bc5) Project is null,
ProjectRoleBasedApiAccessChecker only applies to projects, returning API
[listVirtualMachinesMetrics] for user [User
{"username":"admin","uuid":"0e02fe3b-6350-11ee-a6a8-5254001f02f7"}.] as allowed.
Dec 01 09:06:05 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
Dec 01 09:06:05 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
Dec 01 09:06:05 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
Dec 01 09:06:05 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
Dec 01 09:06:05 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
Dec 01 09:06:05 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
Dec 01 09:06:05 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
Dec 01 09:06:05 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
Dec 01 09:06:05 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Dec 01 09:06:05 n2ncs-mcs01 java[11044]: at
java.base/java.lang.Thread.run(Thread.java:829)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: INFO [c.c.n.NetworkModelImpl]
(VmGroup-Monitor-21-1:ctx-1bdcb1fb) (logid:45c6ea3e) Checking permission for
account PrjAcct-CGS-3 (06859235-cd2e-4506-89e3-75c0ffe53912) on network
n2n-bs-prod-cgs-sg-private_subnet01 (be8521f0-9ffa-46b4-bad9-eb0b6160e253)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: ERROR [c.c.r.ResourceLimitManagerImpl]
(VmGroup-Monitor-21-1:ctx-1bdcb1fb) (logid:45c6ea3e) Maximum amount of
resources of Type = 'primary_storage' for Project Name = CGS in Domain Id = 3
is exceeded: Account Resource Limit = (4.0000 TB) 4398046511104, Current
Account Resource Amount = (3.8086 TB) 4187607060480, Current Account Resource
Reservation = (0 bytes) 0, Requested Resource Amount = (250.00 GB) 268436779008.
Dec 01 09:06:25 n2ncs-mcs01 java[11044]:
com.cloud.exception.ResourceAllocationException: Maximum amount of resources of
Type = 'primary_storage' for Project Name = CGS in Domain Id = 3 is exceeded:
Account Resource Limit = (4.0000 TB) 4398046511104, Current Account Resource
Amount = (3.8086 TB) 4187607060480, Current Account Resource Reservation = (0
bytes) 0, Requested Resource Amount = (250.00 GB) 268436779008.
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
com.cloud.resourcelimit.ResourceLimitManagerImpl.checkAccountResourceLimit(ResourceLimitManagerImpl.java:516)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
com.cloud.resourcelimit.ResourceLimitManagerImpl$2.doInTransactionWithoutResult(ResourceLimitManagerImpl.java:584)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:25)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:21)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
com.cloud.utils.db.Transaction.execute(Transaction.java:40)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
com.cloud.resourcelimit.ResourceLimitManagerImpl.checkResourceLimit(ResourceLimitManagerImpl.java:578)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
com.cloud.resourcelimit.CheckedReservation.<init>(CheckedReservation.java:76)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
com.cloud.vm.UserVmManagerImpl.getUncheckedUserVmResource(UserVmManagerImpl.java:3987)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
com.cloud.vm.UserVmManagerImpl.getCheckedUserVmResource(UserVmManagerImpl.java:3972)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
com.cloud.vm.UserVmManagerImpl.createVirtualMachine(UserVmManagerImpl.java:3955)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
com.cloud.vm.UserVmManagerImpl.createAdvancedVirtualMachine(UserVmManagerImpl.java:3729)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
jdk.internal.reflect.GeneratedMethodAccessor336.invoke(Unknown Source)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
java.base/java.lang.reflect.Method.invoke(Method.java:566)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
com.sun.proxy.$Proxy189.createAdvancedVirtualMachine(Unknown Source)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
com.cloud.network.as.AutoScaleManagerImpl.createNewVM(AutoScaleManagerImpl.java:1812)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
com.cloud.network.as.AutoScaleManagerImpl.doScaleUp(AutoScaleManagerImpl.java:2054)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
com.cloud.network.as.AutoScaleManagerImpl.checkAsGroupMaxAndMinMembers(AutoScaleManagerImpl.java:2444)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
com.cloud.network.as.AutoScaleManagerImpl.monitorVirtualRouterAsGroup(AutoScaleManagerImpl.java:2624)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
com.cloud.network.as.AutoScaleManagerImpl$MonitorTask.runInContext(AutoScaleManagerImpl.java:2996)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
java.base/java.lang.Thread.run(Thread.java:829)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: WARN [c.c.n.a.AutoScaleManagerImpl]
(VmGroup-Monitor-21-1:ctx-1bdcb1fb) (logid:45c6ea3e) Exception:
Dec 01 09:06:25 n2ncs-mcs01 java[11044]:
com.cloud.exception.ResourceAllocationException: Maximum amount of resources of
Type = 'primary_storage' for Project Name = CGS in Domain Id = 3 is exceeded:
Account Resource Limit = (4.0000 TB) 4398046511104, Current Account Resource
Amount = (3.8086 TB) 4187607060480, Current Account Resource Reservation = (0
bytes) 0, Requested Resource Amount = (250.00 GB) 268436779008.
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
com.cloud.resourcelimit.ResourceLimitManagerImpl.checkAccountResourceLimit(ResourceLimitManagerImpl.java:516)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
com.cloud.resourcelimit.ResourceLimitManagerImpl$2.doInTransactionWithoutResult(ResourceLimitManagerImpl.java:584)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:25)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:21)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
com.cloud.utils.db.Transaction.execute(Transaction.java:40)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
com.cloud.resourcelimit.ResourceLimitManagerImpl.checkResourceLimit(ResourceLimitManagerImpl.java:578)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
com.cloud.resourcelimit.CheckedReservation.<init>(CheckedReservation.java:76)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
com.cloud.vm.UserVmManagerImpl.getUncheckedUserVmResource(UserVmManagerImpl.java:3987)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
com.cloud.vm.UserVmManagerImpl.getCheckedUserVmResource(UserVmManagerImpl.java:3972)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
com.cloud.vm.UserVmManagerImpl.createVirtualMachine(UserVmManagerImpl.java:3955)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
com.cloud.vm.UserVmManagerImpl.createAdvancedVirtualMachine(UserVmManagerImpl.java:3729)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
jdk.internal.reflect.GeneratedMethodAccessor336.invoke(Unknown Source)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
java.base/java.lang.reflect.Method.invoke(Method.java:566)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
com.sun.proxy.$Proxy189.createAdvancedVirtualMachine(Unknown Source)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
com.cloud.network.as.AutoScaleManagerImpl.createNewVM(AutoScaleManagerImpl.java:1812)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
com.cloud.network.as.AutoScaleManagerImpl.doScaleUp(AutoScaleManagerImpl.java:2054)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
com.cloud.network.as.AutoScaleManagerImpl.checkAsGroupMaxAndMinMembers(AutoScaleManagerImpl.java:2444)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
com.cloud.network.as.AutoScaleManagerImpl.monitorVirtualRouterAsGroup(AutoScaleManagerImpl.java:2624)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
com.cloud.network.as.AutoScaleManagerImpl$MonitorTask.runInContext(AutoScaleManagerImpl.java:2996)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
java.base/java.lang.Thread.run(Thread.java:829)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: WARN [c.c.n.a.AutoScaleManagerImpl]
(VmGroup-Monitor-21-1:ctx-1bdcb1fb) (logid:45c6ea3e) Caught the following
exception on monitoring AutoScale Vm Group
Dec 01 09:06:25 n2ncs-mcs01 java[11044]:
org.apache.cloudstack.api.ServerApiException: Maximum amount of resources of
Type = 'primary_storage' for Project Name = CGS in Domain Id = 3 is exceeded:
Account Resource Limit = (4.0000 TB) 4398046511104, Current Account Resource
Amount = (3.8086 TB) 4187607060480, Current Account Resource Reservation = (0
bytes) 0, Requested Resource Amount = (250.00 GB) 268436779008.
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
com.cloud.network.as.AutoScaleManagerImpl.createNewVM(AutoScaleManagerImpl.java:1837)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
com.cloud.network.as.AutoScaleManagerImpl.doScaleUp(AutoScaleManagerImpl.java:2054)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
com.cloud.network.as.AutoScaleManagerImpl.checkAsGroupMaxAndMinMembers(AutoScaleManagerImpl.java:2444)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
com.cloud.network.as.AutoScaleManagerImpl.monitorVirtualRouterAsGroup(AutoScaleManagerImpl.java:2624)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
com.cloud.network.as.AutoScaleManagerImpl$MonitorTask.runInContext(AutoScaleManagerImpl.java:2996)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Dec 01 09:06:25 n2ncs-mcs01 java[11044]: at
java.base/java.lang.Thread.run(Thread.java:829)
Dec 01 09:06:26 n2ncs-mcs01 java[11044]: INFO
[o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1278852808-11206:ctx-7609d144
ctx-6e4a9f60) (logid:adb6660f) Account [Account
[{"accountName":"admin","id":2,"uuid":"0e02266c-6350-11ee-a6a8-5254001f02f7"}]]
is Root Admin or Domain Admin, all APIs are allowed.
Dec 01 09:06:26 n2ncs-mcs01 java[11044]: WARN
[o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp1278852808-11206:ctx-7609d144
ctx-6e4a9f60) (logid:adb6660f) Project is null,
ProjectRoleBasedApiAccessChecker only applies to projects, returning API
[startVirtualMachine] for user [User
{"username":"admin","uuid":"0e02fe3b-6350-11ee-a6a8-5254001f02f7"}.] as allowed.
Dec 01 09:06:26 n2ncs-mcs01 java[11044]: INFO [o.a.c.f.j.i.AsyncJobMonitor]
(API-Job-Executor-62:ctx-efb71405 job-9263) (logid:4ab2b325) Add job-9263 into
job monitoring
Dec 01 09:06:26 n2ncs-mcs01 java[11044]: INFO
[o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1278852808-11204:ctx-b3987fd7
ctx-218540f1) (logid:23d480df) Account [Account
[{"accountName":"admin","id":2,"uuid":"0e02266c-6350-11ee-a6a8-5254001f02f7"}]]
is Root Admin or Domain Admin, all APIs are allowed.
Dec 01 09:06:26 n2ncs-mcs01 java[11044]: WARN
[o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp1278852808-11204:ctx-b3987fd7
ctx-218540f1) (logid:23d480df) Project is null,
ProjectRoleBasedApiAccessChecker only applies to projects, returning API
[queryAsyncJobResult] for user [User
{"username":"admin","uuid":"0e02fe3b-6350-11ee-a6a8-5254001f02f7"}.] as allowed.
Dec 01 09:06:26 n2ncs-mcs01 java[11044]: INFO [o.a.c.f.j.i.AsyncJobMonitor]
(Work-Job-Executor-114:ctx-b148154e job-9263/job-9265) (logid:02f57480) Add
job-9265 into job monitoring
Dec 01 09:06:27 n2ncs-mcs01 java[11044]: INFO
[c.c.n.e.VpcVirtualRouterElement] (Work-Job-Executor-114:ctx-b148154e
job-9263/job-9265 ctx-2127de32) (logid:2619f60b) Adding VPC routers to Guest
Network: 1 to be added!
Dec 01 09:06:29 n2ncs-mcs01 java[11044]: WARN
[c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-114:ctx-b148154e
job-9263/job-9265 ctx-2127de32) (logid:2619f60b) Unable to orchestrate start VM
instance
{"id":286,"instanceName":"i-2-286-VM","type":"User","uuid":"6987bdf1-7b37-41e1-bdbd-1071a9b0fa18"}
due to [Unable to get answer that is of class com.cloud.agent.api.StartAnswer].
Dec 01 09:06:29 n2ncs-mcs01 java[11044]:
com.cloud.utils.exception.CloudRuntimeException: Unable to get answer that is
of class com.cloud.agent.api.StartAnswer
Dec 01 09:06:29 n2ncs-mcs01 java[11044]: at
com.cloud.agent.manager.Commands.getAnswer(Commands.java:81)
Dec 01 09:06:29 n2ncs-mcs01 java[11044]: at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1304)
Dec 01 09:06:29 n2ncs-mcs01 java[11044]: at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5412)
Dec 01 09:06:29 n2ncs-mcs01 java[11044]: at
jdk.internal.reflect.GeneratedMethodAccessor787.invoke(Unknown Source)
Dec 01 09:06:29 n2ncs-mcs01 java[11044]: at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Dec 01 09:06:29 n2ncs-mcs01 java[11044]: at
java.base/java.lang.reflect.Method.invoke(Method.java:566)
Dec 01 09:06:29 n2ncs-mcs01 java[11044]: at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
Dec 01 09:06:29 n2ncs-mcs01 java[11044]: at
com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5536)
Dec 01 09:06:29 n2ncs-mcs01 java[11044]: at
com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
Dec 01 09:06:29 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
Dec 01 09:06:29 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
Dec 01 09:06:29 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
Dec 01 09:06:29 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
Dec 01 09:06:29 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
Dec 01 09:06:29 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
Dec 01 09:06:29 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
Dec 01 09:06:29 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
Dec 01 09:06:29 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
Dec 01 09:06:29 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
Dec 01 09:06:29 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Dec 01 09:06:29 n2ncs-mcs01 java[11044]: at
java.base/java.lang.Thread.run(Thread.java:829)
Dec 01 09:06:29 n2ncs-mcs01 java[11044]: INFO [o.a.c.f.j.i.AsyncJobMonitor]
(Work-Job-Executor-114:ctx-b148154e job-9263/job-9265) (logid:2619f60b) Remove
job-9265 from job monitoring
Dec 01 09:06:29 n2ncs-mcs01 java[11044]: INFO
[o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1278852808-11206:ctx-190579fa
ctx-6f21911d) (logid:4ddb99a7) Account [Account
[{"accountName":"admin","id":2,"uuid":"0e02266c-6350-11ee-a6a8-5254001f02f7"}]]
is Root Admin or Domain Admin, all APIs are allowed.
Dec 01 09:06:29 n2ncs-mcs01 java[11044]: WARN
[o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp1278852808-11206:ctx-190579fa
ctx-6f21911d) (logid:4ddb99a7) Project is null,
ProjectRoleBasedApiAccessChecker only applies to projects, returning API
[queryAsyncJobResult] for user [User
{"username":"admin","uuid":"0e02fe3b-6350-11ee-a6a8-5254001f02f7"}.] as allowed.
Dec 01 09:06:30 n2ncs-mcs01 java[11044]: INFO [o.a.c.f.j.i.AsyncJobMonitor]
(Work-Job-Executor-115:ctx-03a862a7 job-9263/job-9267) (logid:76ea3670) Add
job-9267 into job monitoring
Dec 01 09:06:31 n2ncs-mcs01 java[11044]: INFO
[c.c.n.e.VpcVirtualRouterElement] (Work-Job-Executor-115:ctx-03a862a7
job-9263/job-9267 ctx-f2efc47a) (logid:2619f60b) Adding VPC routers to Guest
Network: 1 to be added!
Dec 01 09:06:32 n2ncs-mcs01 java[11044]: INFO
[o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1278852808-11204:ctx-6df2ee4b
ctx-9b7264b6) (logid:006c1b84) Account [Account
[{"accountName":"admin","id":2,"uuid":"0e02266c-6350-11ee-a6a8-5254001f02f7"}]]
is Root Admin or Domain Admin, all APIs are allowed.
Dec 01 09:06:32 n2ncs-mcs01 java[11044]: WARN
[o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp1278852808-11204:ctx-6df2ee4b
ctx-9b7264b6) (logid:006c1b84) Project is null,
ProjectRoleBasedApiAccessChecker only applies to projects, returning API
[queryAsyncJobResult] for user [User
{"username":"admin","uuid":"0e02fe3b-6350-11ee-a6a8-5254001f02f7"}.] as allowed.
Dec 01 09:06:33 n2ncs-mcs01 java[11044]: INFO
[c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-115:ctx-03a862a7
job-9263/job-9267 ctx-f2efc47a) (logid:2619f60b) Unable to start VM on Host
{"id":15,"name":"n2ncs-c06","type":"Routing","uuid":"0f518498-d995-4d1b-a938-2d9ca8d433bd"}
due to missing source information for device hda
Dec 01 09:06:33 n2ncs-mcs01 java[11044]: INFO [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-115:ctx-03a862a7 job-9263/job-9267 ctx-f2efc47a
FirstFitRoutingAllocator) (logid:2619f60b) Guest VM is requested with
Custom[UEFI] Boot Type false
Dec 01 09:06:33 n2ncs-mcs01 java[11044]: INFO
[c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-115:ctx-03a862a7
job-9263/job-9267 ctx-f2efc47a) (logid:2619f60b) Re-ordering hosts [Host
{"id":3,"name":"n2ncs-c02","type":"Routing","uuid":"77defbe3-794f-407c-84d0-520db2e27708"},
Host
{"id":5,"name":"n2ncs-c03","type":"Routing","uuid":"ba19c0fd-259c-41f6-af63-8da065f0b677"}]
by priorities {}
Dec 01 09:06:33 n2ncs-mcs01 java[11044]: INFO
[c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-115:ctx-03a862a7
job-9263/job-9267 ctx-f2efc47a) (logid:2619f60b) Hosts after re-ordering are:
[Host
{"id":3,"name":"n2ncs-c02","type":"Routing","uuid":"77defbe3-794f-407c-84d0-520db2e27708"},
Host
{"id":5,"name":"n2ncs-c03","type":"Routing","uuid":"ba19c0fd-259c-41f6-af63-8da065f0b677"}]
Dec 01 09:06:33 n2ncs-mcs01 java[11044]: INFO
[c.c.n.e.VpcVirtualRouterElement] (Work-Job-Executor-115:ctx-03a862a7
job-9263/job-9267 ctx-f2efc47a) (logid:2619f60b) Adding VPC routers to Guest
Network: 1 to be added!
Dec 01 09:06:35 n2ncs-mcs01 java[11044]: INFO
[c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-115:ctx-03a862a7
job-9263/job-9267 ctx-f2efc47a) (logid:2619f60b) Unable to start VM on Host
{"id":3,"name":"n2ncs-c02","type":"Routing","uuid":"77defbe3-794f-407c-84d0-520db2e27708"}
due to missing source information for device hda
Dec 01 09:06:35 n2ncs-mcs01 java[11044]: INFO [c.c.a.m.a.i.FirstFitAllocator]
(Work-Job-Executor-115:ctx-03a862a7 job-9263/job-9267 ctx-f2efc47a
FirstFitRoutingAllocator) (logid:2619f60b) Guest VM is requested with
Custom[UEFI] Boot Type false
Dec 01 09:06:35 n2ncs-mcs01 java[11044]: INFO
[c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-115:ctx-03a862a7
job-9263/job-9267 ctx-f2efc47a) (logid:2619f60b) Re-ordering hosts [Host
{"id":7,"name":"n2ncs-c04","type":"Routing","uuid":"3c9bb524-fbfe-49de-9190-248251f96d4e"},
Host
{"id":13,"name":"n2ncs-c05","type":"Routing","uuid":"1d1ed762-ad39-4ce7-a9ca-42057e8b1a6b"}]
by priorities {}
Dec 01 09:06:35 n2ncs-mcs01 java[11044]: INFO
[c.c.d.DeploymentPlanningManagerImpl] (Work-Job-Executor-115:ctx-03a862a7
job-9263/job-9267 ctx-f2efc47a) (logid:2619f60b) Hosts after re-ordering are:
[Host
{"id":7,"name":"n2ncs-c04","type":"Routing","uuid":"3c9bb524-fbfe-49de-9190-248251f96d4e"},
Host
{"id":13,"name":"n2ncs-c05","type":"Routing","uuid":"1d1ed762-ad39-4ce7-a9ca-42057e8b1a6b"}]
Dec 01 09:06:35 n2ncs-mcs01 java[11044]: INFO
[c.c.n.e.VpcVirtualRouterElement] (Work-Job-Executor-115:ctx-03a862a7
job-9263/job-9267 ctx-f2efc47a) (logid:2619f60b) Adding VPC routers to Guest
Network: 1 to be added!
Dec 01 09:06:36 n2ncs-mcs01 java[11044]: INFO
[o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1278852808-11206:ctx-4dd7a309
ctx-efc2f3cf) (logid:f6edac32) Account [Account
[{"accountName":"admin","id":2,"uuid":"0e02266c-6350-11ee-a6a8-5254001f02f7"}]]
is Root Admin or Domain Admin, all APIs are allowed.
Dec 01 09:06:36 n2ncs-mcs01 java[11044]: WARN
[o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp1278852808-11206:ctx-4dd7a309
ctx-efc2f3cf) (logid:f6edac32) Project is null,
ProjectRoleBasedApiAccessChecker only applies to projects, returning API
[queryAsyncJobResult] for user [User
{"username":"admin","uuid":"0e02fe3b-6350-11ee-a6a8-5254001f02f7"}.] as allowed.
Dec 01 09:06:38 n2ncs-mcs01 java[11044]: WARN
[c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-115:ctx-03a862a7
job-9263/job-9267 ctx-f2efc47a) (logid:2619f60b) Unable to orchestrate start VM
instance
{"id":286,"instanceName":"i-2-286-VM","type":"User","uuid":"6987bdf1-7b37-41e1-bdbd-1071a9b0fa18"}
due to [Unable to get answer that is of class com.cloud.agent.api.StartAnswer].
Dec 01 09:06:38 n2ncs-mcs01 java[11044]:
com.cloud.utils.exception.CloudRuntimeException: Unable to get answer that is
of class com.cloud.agent.api.StartAnswer
Dec 01 09:06:38 n2ncs-mcs01 java[11044]: at
com.cloud.agent.manager.Commands.getAnswer(Commands.java:81)
Dec 01 09:06:38 n2ncs-mcs01 java[11044]: at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1304)
Dec 01 09:06:38 n2ncs-mcs01 java[11044]: at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5412)
Dec 01 09:06:38 n2ncs-mcs01 java[11044]: at
jdk.internal.reflect.GeneratedMethodAccessor787.invoke(Unknown Source)
Dec 01 09:06:38 n2ncs-mcs01 java[11044]: at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Dec 01 09:06:38 n2ncs-mcs01 java[11044]: at
java.base/java.lang.reflect.Method.invoke(Method.java:566)
Dec 01 09:06:38 n2ncs-mcs01 java[11044]: at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
Dec 01 09:06:38 n2ncs-mcs01 java[11044]: at
com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5536)
Dec 01 09:06:38 n2ncs-mcs01 java[11044]: at
com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
Dec 01 09:06:38 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
Dec 01 09:06:38 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
Dec 01 09:06:38 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
Dec 01 09:06:38 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
Dec 01 09:06:38 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
Dec 01 09:06:38 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
Dec 01 09:06:38 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
Dec 01 09:06:38 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
Dec 01 09:06:38 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
Dec 01 09:06:38 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
Dec 01 09:06:38 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Dec 01 09:06:38 n2ncs-mcs01 java[11044]: at
java.base/java.lang.Thread.run(Thread.java:829)
Dec 01 09:06:38 n2ncs-mcs01 java[11044]: INFO [o.a.c.f.j.i.AsyncJobMonitor]
(Work-Job-Executor-115:ctx-03a862a7 job-9263/job-9267) (logid:2619f60b) Remove
job-9267 from job monitoring
Dec 01 09:06:38 n2ncs-mcs01 java[11044]: ERROR [c.c.a.ApiAsyncJobDispatcher]
(API-Job-Executor-62:ctx-efb71405 job-9263) (logid:2619f60b) Unexpected
exception while executing
org.apache.cloudstack.api.command.admin.vm.StartVMCmdByAdmin
Dec 01 09:06:38 n2ncs-mcs01 java[11044]:
com.cloud.utils.exception.CloudRuntimeException: Unable to orchestrate start VM
instance
{"id":286,"instanceName":"i-2-286-VM","type":"User","uuid":"6987bdf1-7b37-41e1-bdbd-1071a9b0fa18"}
due to [Unable to get answer that is of class com.cloud.agent.api.StartAnswer].
Dec 01 09:06:38 n2ncs-mcs01 java[11044]: at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5416)
Dec 01 09:06:38 n2ncs-mcs01 java[11044]: at
jdk.internal.reflect.GeneratedMethodAccessor787.invoke(Unknown Source)
Dec 01 09:06:38 n2ncs-mcs01 java[11044]: at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Dec 01 09:06:38 n2ncs-mcs01 java[11044]: at
java.base/java.lang.reflect.Method.invoke(Method.java:566)
Dec 01 09:06:38 n2ncs-mcs01 java[11044]: at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
Dec 01 09:06:38 n2ncs-mcs01 java[11044]: at
com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5536)
Dec 01 09:06:38 n2ncs-mcs01 java[11044]: at
com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
Dec 01 09:06:38 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
Dec 01 09:06:38 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
Dec 01 09:06:38 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
Dec 01 09:06:38 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
Dec 01 09:06:38 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
Dec 01 09:06:38 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
Dec 01 09:06:38 n2ncs-mcs01 java[11044]: at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
Dec 01 09:06:38 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
Dec 01 09:06:38 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
Dec 01 09:06:38 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
Dec 01 09:06:38 n2ncs-mcs01 java[11044]: at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
Dec 01 09:06:38 n2ncs-mcs01 java[11044]: at
java.base/java.lang.Thread.run(Thread.java:829)
Dec 01 09:06:38 n2ncs-mcs01 java[11044]: INFO [o.a.c.f.j.i.AsyncJobMonitor]
(API-Job-Executor-62:ctx-efb71405 job-9263) (logid:2619f60b) Remove job-9263
from job monitoring
Dec 01 09:06:39 n2ncs-mcs01 java[11044]: INFO
[o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1278852808-11204:ctx-16e87dcc
ctx-fb1e6607) (logid:8cae1e7f) Account [Account
[{"accountName":"admin","id":2,"uuid":"0e02266c-6350-11ee-a6a8-5254001f02f7"}]]
is Root Admin or Domain Admin, all APIs are allowed.
Dec 01 09:06:39 n2ncs-mcs01 java[11044]: WARN
[o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp1278852808-11204:ctx-16e87dcc
ctx-fb1e6607) (logid:8cae1e7f) Project is null,
ProjectRoleBasedApiAccessChecker only applies to projects, returning API
[queryAsyncJobResult] for user [User
{"username":"admin","uuid":"0e02fe3b-6350-11ee-a6a8-5254001f02f7"}.] as allowed.
Dec 01 09:06:39 n2ncs-mcs01 java[11044]: INFO
[o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1278852808-11206:ctx-a86d22c7
ctx-e7ef493a) (logid:d7f48c6b) Account [Account
[{"accountName":"admin","id":2,"uuid":"0e02266c-6350-11ee-a6a8-5254001f02f7"}]]
is Root Admin or Domain Admin, all APIs are allowed.
Dec 01 09:06:39 n2ncs-mcs01 java[11044]: WARN
[o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp1278852808-11206:ctx-a86d22c7
ctx-e7ef493a) (logid:d7f48c6b) Project is null,
ProjectRoleBasedApiAccessChecker only applies to projects, returning API
[listVirtualMachinesMetrics] for user [User
{"username":"admin","uuid":"0e02fe3b-6350-11ee-a6a8-5254001f02f7"}.] as allowed.
Dec 01 09:06:39 n2ncs-mcs01 java[11044]: INFO
[o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1278852808-11190:ctx-d5b798dc
ctx-9ff9e921) (logid:cb7a6db0) Account [Account
[{"accountName":"admin","id":2,"uuid":"0e02266c-6350-11ee-a6a8-5254001f02f7"}]]
is Root Admin or Domain Admin, all APIs are allowed.
Dec 01 09:06:39 n2ncs-mcs01 java[11044]: WARN
[o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp1278852808-11190:ctx-d5b798dc
ctx-9ff9e921) (logid:cb7a6db0) Project is null,
ProjectRoleBasedApiAccessChecker only applies to projects, returning API
[listVirtualMachinesMetrics] for user [User
{"username":"admin","uuid":"0e02fe3b-6350-11ee-a6a8-5254001f02f7"}.] as allowed.
Dec 01 09:06:39 n2ncs-mcs01 java[11044]: INFO
[o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1278852808-11208:ctx-b2ec0c3f
ctx-9988c6d0) (logid:9ab0e309) Account [Account
[{"accountName":"admin","id":2,"uuid":"0e02266c-6350-11ee-a6a8-5254001f02f7"}]]
is Root Admin or Domain Admin, all APIs are allowed.
Dec 01 09:06:39 n2ncs-mcs01 java[11044]: WARN
[o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp1278852808-11208:ctx-b2ec0c3f
ctx-9988c6d0) (logid:9ab0e309) Project is null,
ProjectRoleBasedApiAccessChecker only applies to projects, returning API
[listVirtualMachinesMetrics] for user [User
{"username":"admin","uuid":"0e02fe3b-6350-11ee-a6a8-5254001f02f7"}.] as allowed.
Dec 01 09:06:39 n2ncs-mcs01 java[11044]: INFO
[o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1278852808-11094:ctx-5cf81d66
ctx-b4b08822) (logid:0f468221) Account [Account
[{"accountName":"admin","id":2,"uuid":"0e02266c-6350-11ee-a6a8-5254001f02f7"}]]
is Root Admin or Domain Admin, all APIs are allowed.
Dec 01 09:06:39 n2ncs-mcs01 java[11044]: WARN
[o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp1278852808-11094:ctx-5cf81d66
ctx-b4b08822) (logid:0f468221) Project is null,
ProjectRoleBasedApiAccessChecker only applies to projects, returning API
[listVirtualMachinesMetrics] for user [User
{"username":"admin","uuid":"0e02fe3b-6350-11ee-a6a8-5254001f02f7"}.] as allowed.
Dec 01 09:06:39 n2ncs-mcs01 java[11044]: INFO
[o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1278852808-11206:ctx-804df6a1
ctx-80d80f1f) (logid:c36a545c) Account [Account
[{"accountName":"admin","id":2,"uuid":"0e02266c-6350-11ee-a6a8-5254001f02f7"}]]
is Root Admin or Domain Admin, all APIs are allowed.
Dec 01 09:06:39 n2ncs-mcs01 java[11044]: WARN
[o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp1278852808-11206:ctx-804df6a1
ctx-80d80f1f) (logid:c36a545c) Project is null,
ProjectRoleBasedApiAccessChecker only applies to projects, returning API
[listVirtualMachinesMetrics] for user [User
{"username":"admin","uuid":"0e02fe3b-6350-11ee-a6a8-5254001f02f7"}.] as allowed.
Dec 01 09:06:39 n2ncs-mcs01 java[11044]: INFO
[o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1278852808-11094:ctx-8a3ac0ed
ctx-ad187d96) (logid:bdab0eef) Account [Account
[{"accountName":"admin","id":2,"uuid":"0e02266c-6350-11ee-a6a8-5254001f02f7"}]]
is Root Admin or Domain Admin, all APIs are allowed.
Dec 01 09:06:39 n2ncs-mcs01 java[11044]: WARN
[o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp1278852808-11094:ctx-8a3ac0ed
ctx-ad187d96) (logid:bdab0eef) Project is null,
ProjectRoleBasedApiAccessChecker only applies to projects, returning API
[listVirtualMachinesMetrics] for user [User
{"username":"admin","uuid":"0e02fe3b-6350-11ee-a6a8-5254001f02f7"}.] as allowed.
Dec 01 09:06:39 n2ncs-mcs01 java[11044]: INFO
[o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1278852808-11206:ctx-cae6dc39
ctx-d6ba8fe6) (logid:89d9ffeb) Account [Account
[{"accountName":"admin","id":2,"uuid":"0e02266c-6350-11ee-a6a8-5254001f02f7"}]]
is Root Admin or Domain Admin, all APIs are allowed.
Dec 01 09:06:39 n2ncs-mcs01 java[11044]: WARN
[o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp1278852808-11206:ctx-cae6dc39
ctx-d6ba8fe6) (logid:89d9ffeb) Project is null,
ProjectRoleBasedApiAccessChecker only applies to projects, returning API
[listVirtualMachinesMetrics] for user [User
{"username":"admin","uuid":"0e02fe3b-6350-11ee-a6a8-5254001f02f7"}.] as allowed.
Dec 01 09:06:39 n2ncs-mcs01 java[11044]: INFO
[o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1278852808-11094:ctx-704eaab8
ctx-3b16bbbd) (logid:865a39ec) Account [Account
[{"accountName":"admin","id":2,"uuid":"0e02266c-6350-11ee-a6a8-5254001f02f7"}]]
is Root Admin or Domain Admin, all APIs are allowed.
Dec 01 09:06:39 n2ncs-mcs01 java[11044]: WARN
[o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp1278852808-11094:ctx-704eaab8
ctx-3b16bbbd) (logid:865a39ec) Project is null,
ProjectRoleBasedApiAccessChecker only applies to projects, returning API
[listVirtualMachinesMetrics] for user [User
{"username":"admin","uuid":"0e02fe3b-6350-11ee-a6a8-5254001f02f7"}.] as allowed.
Dec 01 09:06:39 n2ncs-mcs01 java[11044]: INFO
[o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1278852808-11206:ctx-93aa4b38
ctx-aea0a579) (logid:541ea609) Account [Account
[{"accountName":"admin","id":2,"uuid":"0e02266c-6350-11ee-a6a8-5254001f02f7"}]]
is Root Admin or Domain Admin, all APIs are allowed.
Dec 01 09:06:39 n2ncs-mcs01 java[11044]: WARN
[o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp1278852808-11206:ctx-93aa4b38
ctx-aea0a579) (logid:541ea609) Project is null,
ProjectRoleBasedApiAccessChecker only applies to projects, returning API
[listVirtualMachinesMetrics] for user [User
{"username":"admin","uuid":"0e02fe3b-6350-11ee-a6a8-5254001f02f7"}.] as allowed.
Dec 01 09:06:40 n2ncs-mcs01 java[11044]: INFO
[o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1278852808-11094:ctx-22495840
ctx-4636f473) (logid:ab993615) Account [Account
[{"accountName":"admin","id":2,"uuid":"0e02266c-6350-11ee-a6a8-5254001f02f7"}]]
is Root Admin or Domain Admin, all APIs are allowed.
Dec 01 09:06:40 n2ncs-mcs01 java[11044]: WARN
[o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp1278852808-11094:ctx-22495840
ctx-4636f473) (logid:ab993615) Project is null,
ProjectRoleBasedApiAccessChecker only applies to projects, returning API
[listVirtualMachinesMetrics] for user [User
{"username":"admin","uuid":"0e02fe3b-6350-11ee-a6a8-5254001f02f7"}.] as allowed.
Dec 01 09:06:40 n2ncs-mcs01 java[11044]: INFO
[o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1278852808-11206:ctx-a2bf62be
ctx-0a80baf2) (logid:5dca31ec) Account [Account
[{"accountName":"admin","id":2,"uuid":"0e02266c-6350-11ee-a6a8-5254001f02f7"}]]
is Root Admin or Domain Admin, all APIs are allowed.
Dec 01 09:06:40 n2ncs-mcs01 java[11044]: WARN
[o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp1278852808-11206:ctx-a2bf62be
ctx-0a80baf2) (logid:5dca31ec) Project is null,
ProjectRoleBasedApiAccessChecker only applies to projects, returning API
[listVirtualMachinesMetrics] for user [User
{"username":"admin","uuid":"0e02fe3b-6350-11ee-a6a8-5254001f02f7"}.] as allowed.
Dec 01 09:06:40 n2ncs-mcs01 java[11044]: INFO
[o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1278852808-11094:ctx-f87e6a81
ctx-41035169) (logid:65063a30) Account [Account
[{"accountName":"admin","id":2,"uuid":"0e02266c-6350-11ee-a6a8-5254001f02f7"}]]
is Root Admin or Domain Admin, all APIs are allowed.
Dec 01 09:06:40 n2ncs-mcs01 java[11044]: WARN
[o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp1278852808-11094:ctx-f87e6a81
ctx-41035169) (logid:65063a30) Project is null,
ProjectRoleBasedApiAccessChecker only applies to projects, returning API
[listVirtualMachinesMetrics] for user [User
{"username":"admin","uuid":"0e02fe3b-6350-11ee-a6a8-5254001f02f7"}.] as allowed.
Dec 01 09:06:40 n2ncs-mcs01 java[11044]: INFO
[o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1278852808-11206:ctx-178b1d6a
ctx-d83d7a75) (logid:a5635dcd) Account [Account
[{"accountName":"admin","id":2,"uuid":"0e02266c-6350-11ee-a6a8-5254001f02f7"}]]
is Root Admin or Domain Admin, all APIs are allowed.
Dec 01 09:06:40 n2ncs-mcs01 java[11044]: WARN
[o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp1278852808-11206:ctx-178b1d6a
ctx-d83d7a75) (logid:a5635dcd) Project is null,
ProjectRoleBasedApiAccessChecker only applies to projects, returning API
[listVirtualMachinesMetrics] for user [User
{"username":"admin","uuid":"0e02fe3b-6350-11ee-a6a8-5254001f02f7"}.] as allowed.
Dec 01 09:06:40 n2ncs-mcs01 java[11044]: INFO
[o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp1278852808-11094:ctx-994d5aa0
ctx-29ea88c9) (logid:b05ed930) Account [Account
[{"accountName":"admin","id":2,"uuid":"0e02266c-6350-11ee-a6a8-5254001f02f7"}]]
is Root Admin or Domain Admin, all APIs are allowed.
Dec 01 09:06:40 n2ncs-mcs01 java[11044]: WARN
[o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp1278852808-11094:ctx-994d5aa0
ctx-29ea88c9) (logid:b05ed930) Project is null,
ProjectRoleBasedApiAccessChecker only applies to projects, returning API
[listVirtualMachinesMetrics] for user [User
{"username":"admin","uuid":"0e02fe3b-6350-11ee-a6a8-5254001f02f7"}.] as allowed.
