This was an issue with redundancy. I just set redundant flag to 0 and
restarted the network with cleanup flag set to true and it worked.
On Fri, Mar 29, 2019 at 7:03 PM Fariborz Navidan <mdvlinqu...@gmail.com>
wrote:
>
> 2019-03-29 14:28:05,448 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
> (secstorage-1:ctx-e7a148f3) (logid:953545b2) Zone 1 is ready to launch
> secondary storage VM
> 2019-03-29 14:28:05,482 DEBUG [c.c.c.ConsoleProxyManagerImpl]
> (consoleproxy-1:ctx-a238f5ea) (logid:174bb141) Zone 1 is ready to launch
> console proxy
> 2019-03-29 14:28:06,699 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentManager-Handler-7:null) (logid:) SeqA 21-7009: Processing Seq
> 21-7009: { Cmd , MgmtId: -1, via: 21, Ver: v1, Flags: 11,
> [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":97,"_loadInfo":"{\n
> \"connections\": []\n}","wait":0}}] }
> 2019-03-29 14:28:06,700 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentManager-Handler-7:null) (logid:) SeqA 21-7009: Sending Seq 21-7009:
> { Ans: , MgmtId: 279278805450993, via: 21, Ver: v1, Flags: 100010,
> [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }
> 2019-03-29 14:28:07,774 DEBUG [c.c.a.ApiServlet]
> (qtp788117692-20:ctx-ba8bac07) (logid:40cc781a) ===START=== 2.190.177.97
> -- GET
> command=restartNetwork&response=json&id=6e644551-5aee-4c9b-a75f-134f544ee97c&cleanup=true&makeredundant=false&_=1553866089613
> 2019-03-29 14:28:07,776 DEBUG [c.c.a.ApiServer]
> (qtp788117692-20:ctx-ba8bac07 ctx-a8e5fbc6) (logid:40cc781a) CIDRs from
> which account 'Acct[27cd01ef-3907-11e9-87ab-a4bf012ed1a6-admin]' is allowed
> to perform API calls: 0.0.0.0/0,::/0
> 2019-03-29 14:28:07,786 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (API-Job-Executor-2:ctx-c03e0b1a job-16346) (logid:b38d88a3) Add job-16346
> into job monitoring
> 2019-03-29 14:28:07,790 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (qtp788117692-20:ctx-ba8bac07 ctx-a8e5fbc6) (logid:40cc781a) submit async
> job-16346, details: AsyncJobVO {id:16346, userId: 2, accountId: 2,
> instanceType: None, instanceId: null, cmd:
> org.apache.cloudstack.api.command.user.network.RestartNetworkCmd, cmdInfo:
> {"makeredundant":"false","cleanup":"true","response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"3261","id":"6e644551-5aee-4c9b-a75f-134f544ee97c","ctxDetails":"{\"interface
> com.cloud.network.Network\":\"6e644551-5aee-4c9b-a75f-134f544ee97c\"}","ctxAccountId":"2","uuid":"6e644551-5aee-4c9b-a75f-134f544ee97c","cmdEventType":"NETWORK.RESTART","_":"1553866089613"},
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> result: null, initMsid: 279278805450993, completeMsid: null, lastUpdated:
> null, lastPolled: null, created: null}
> 2019-03-29 14:28:07,790 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-2:ctx-c03e0b1a job-16346) (logid:241b7d3f) Executing
> AsyncJobVO {id:16346, userId: 2, accountId: 2, instanceType: None,
> instanceId: null, cmd:
> org.apache.cloudstack.api.command.user.network.RestartNetworkCmd, cmdInfo:
> {"makeredundant":"false","cleanup":"true","response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"3261","id":"6e644551-5aee-4c9b-a75f-134f544ee97c","ctxDetails":"{\"interface
> com.cloud.network.Network\":\"6e644551-5aee-4c9b-a75f-134f544ee97c\"}","ctxAccountId":"2","uuid":"6e644551-5aee-4c9b-a75f-134f544ee97c","cmdEventType":"NETWORK.RESTART","_":"1553866089613"},
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> result: null, initMsid: 279278805450993, completeMsid: null, lastUpdated:
> null, lastPolled: null, created: null}
> 2019-03-29 14:28:07,791 DEBUG [c.c.a.ApiServlet]
> (qtp788117692-20:ctx-ba8bac07 ctx-a8e5fbc6) (logid:40cc781a) ===END===
> 2.190.177.97 -- GET
> command=restartNetwork&response=json&id=6e644551-5aee-4c9b-a75f-134f544ee97c&cleanup=true&makeredundant=false&_=1553866089613
> 2019-03-29 14:28:07,801 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (API-Job-Executor-2:ctx-c03e0b1a job-16346 ctx-bbbd2b27) (logid:241b7d3f)
> Restarting network 204...
> 2019-03-29 14:28:07,802 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (API-Job-Executor-2:ctx-c03e0b1a job-16346 ctx-bbbd2b27) (logid:241b7d3f)
> Performing rolling restart of routers of network Ntwk[204|Guest|7]
> 2019-03-29 14:28:07,803 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (API-Job-Executor-2:ctx-c03e0b1a job-16346 ctx-bbbd2b27) (logid:241b7d3f)
> Destroying old router VM[DomainRouter|r-90-VM]
> 2019-03-29 14:28:07,803 DEBUG [c.c.n.r.NetworkHelperImpl]
> (API-Job-Executor-2:ctx-c03e0b1a job-16346 ctx-bbbd2b27) (logid:241b7d3f)
> Attempting to destroy router 90
> 2019-03-29 14:28:07,810 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-2:ctx-c03e0b1a job-16346 ctx-bbbd2b27) (logid:241b7d3f)
> Sync job-16347 execution on object VmWorkJobQueue.90
> 2019-03-29 14:28:08,568 DEBUG [c.c.s.StatsCollector]
> (StatsCollector-6:ctx-b4679e35) (logid:380c977e) StorageCollector is
> running...
> 2019-03-29 14:28:08,572 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
> (StatsCollector-6:ctx-b4679e35) (logid:380c977e) getCommandHostDelegation:
> class com.cloud.agent.api.GetStorageStatsCommand
> 2019-03-29 14:28:08,572 DEBUG [c.c.h.XenServerGuru]
> (StatsCollector-6:ctx-b4679e35) (logid:380c977e) We are returning the
> default host to execute commands because the command is not of Copy type.
> 2019-03-29 14:28:08,650 DEBUG [c.c.a.t.Request]
> (StatsCollector-6:ctx-b4679e35) (logid:380c977e) Seq
> 23-9107122871473274930: Received: { Ans: , MgmtId: 279278805450993, via:
> 23(s-135-VM), Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
> 2019-03-29 14:28:08,651 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
> (StatsCollector-6:ctx-b4679e35) (logid:380c977e) getCommandHostDelegation:
> class com.cloud.agent.api.GetStorageStatsCommand
> 2019-03-29 14:28:08,651 DEBUG [c.c.h.XenServerGuru]
> (StatsCollector-6:ctx-b4679e35) (logid:380c977e) We are returning the
> default host to execute commands because the command is not of Copy type.
> 2019-03-29 14:28:08,710 DEBUG [c.c.a.t.Request]
> (StatsCollector-6:ctx-b4679e35) (logid:380c977e) Seq
> 20-7123568710593282239: Received: { Ans: , MgmtId: 279278805450993, via:
> 20(fr-kvm1), Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
> 2019-03-29 14:28:08,711 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
> (StatsCollector-6:ctx-b4679e35) (logid:380c977e) getCommandHostDelegation:
> class com.cloud.agent.api.GetStorageStatsCommand
> 2019-03-29 14:28:08,711 DEBUG [c.c.h.XenServerGuru]
> (StatsCollector-6:ctx-b4679e35) (logid:380c977e) We are returning the
> default host to execute commands because the command is not of Copy type.
> 2019-03-29 14:28:08,774 DEBUG [c.c.a.t.Request]
> (StatsCollector-6:ctx-b4679e35) (logid:380c977e) Seq
> 20-7123568710593282240: Received: { Ans: , MgmtId: 279278805450993, via:
> 20(fr-kvm1), Ver: v1, Flags: 10, { GetStorageStatsAnswer } }
> 2019-03-29 14:28:08,777 DEBUG [c.c.a.t.Request]
> (StatsCollector-5:ctx-3c915db5) (logid:8775a686) Seq
> 20-7123568710593282238: Received: { Ans: , MgmtId: 279278805450993, via:
> 20(fr-kvm1), Ver: v1, Flags: 10, { GetVmStatsAnswer } }
> 2019-03-29 14:28:08,874 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (AsyncJobMgr-Heartbeat-1:ctx-acc21ccd) (logid:4909a327) Execute sync-queue
> item: SyncQueueItemVO {id:15042, queueId: 13061, contentType: AsyncJob,
> contentId: 16347, lastProcessMsid: 279278805450993, lastprocessNumber: 12,
> lastProcessTime: Fri Mar 29 14:28:08 CET 2019, created: Fri Mar 29 14:28:07
> CET 2019}
> 2019-03-29 14:28:08,875 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (AsyncJobMgr-Heartbeat-1:ctx-acc21ccd) (logid:4909a327) Schedule queued
> job-16347
> 2019-03-29 14:28:08,877 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (Work-Job-Executor-2:ctx-3cff0259 job-16346/job-16347) (logid:d66d0b38) Add
> job-16347 into job monitoring
> 2019-03-29 14:28:08,880 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-2:ctx-3cff0259 job-16346/job-16347) (logid:241b7d3f)
> Executing AsyncJobVO {id:16347, userId: 2, accountId: 2, instanceType:
> null, instanceId: null, cmd: com.cloud.vm.VmWorkStop, cmdInfo:
> rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNjb20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlkTAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAAAWnQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwA,
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> result: null, initMsid: 279278805450993, completeMsid: null, lastUpdated:
> null, lastPolled: null, created: Fri Mar 29 14:28:07 CET 2019}
> 2019-03-29 14:28:08,881 DEBUG [c.c.v.VmWorkJobDispatcher]
> (Work-Job-Executor-2:ctx-3cff0259 job-16346/job-16347) (logid:241b7d3f) Run
> VM work job: com.cloud.vm.VmWorkStop for VM 90, job origin: 16346
> 2019-03-29 14:28:08,882 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-2:ctx-3cff0259 job-16346/job-16347 ctx-99c6000f)
> (logid:241b7d3f) Execute VM work job:
> com.cloud.vm.VmWorkStop{"cleanup":false,"userId":2,"accountId":2,"vmId":90,"handlerName":"VirtualMachineManagerImpl"}
> 2019-03-29 14:28:08,884 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-2:ctx-3cff0259 job-16346/job-16347 ctx-99c6000f)
> (logid:241b7d3f) VM is already stopped: VM[DomainRouter|r-90-VM]
> 2019-03-29 14:28:08,884 DEBUG [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-2:ctx-3cff0259 job-16346/job-16347 ctx-99c6000f)
> (logid:241b7d3f) Done executing VM work job:
> com.cloud.vm.VmWorkStop{"cleanup":false,"userId":2,"accountId":2,"vmId":90,"handlerName":"VirtualMachineManagerImpl"}
> 2019-03-29 14:28:08,884 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-2:ctx-3cff0259 job-16346/job-16347 ctx-99c6000f)
> (logid:241b7d3f) Complete async job-16347, jobStatus: SUCCEEDED,
> resultCode: 0, result: null
> 2019-03-29 14:28:08,884 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-2:ctx-3cff0259 job-16346/job-16347 ctx-99c6000f)
> (logid:241b7d3f) Publish async job-16347 complete on message bus
> 2019-03-29 14:28:08,884 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-2:ctx-3cff0259 job-16346/job-16347 ctx-99c6000f)
> (logid:241b7d3f) Wake up jobs related to job-16347
> 2019-03-29 14:28:08,884 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-2:ctx-3cff0259 job-16346/job-16347 ctx-99c6000f)
> (logid:241b7d3f) Update db status for job-16347
> 2019-03-29 14:28:08,885 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-2:ctx-3cff0259 job-16346/job-16347 ctx-99c6000f)
> (logid:241b7d3f) Wake up jobs joined with job-16347 and disjoin all subjobs
> created from job- 16347
> 2019-03-29 14:28:08,888 DEBUG [c.c.v.VmWorkJobDispatcher]
> (Work-Job-Executor-2:ctx-3cff0259 job-16346/job-16347) (logid:241b7d3f)
> Done with run of VM work job: com.cloud.vm.VmWorkStop for VM 90, job
> origin: 16346
> 2019-03-29 14:28:08,888 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-2:ctx-3cff0259 job-16346/job-16347) (logid:241b7d3f)
> Done executing com.cloud.vm.VmWorkStop for job-16347
> 2019-03-29 14:28:08,889 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (Work-Job-Executor-2:ctx-3cff0259 job-16346/job-16347) (logid:241b7d3f)
> Remove job-16347 from job monitoring
> 2019-03-29 14:28:08,897 DEBUG [c.c.c.CapacityManagerImpl]
> (API-Job-Executor-2:ctx-c03e0b1a job-16346 ctx-bbbd2b27) (logid:241b7d3f)
> VM state transitted from :Stopped to Expunging with event:
> ExpungeOperationvm's original host id: 20 new host id: null host id before
> state transition: null
> 2019-03-29 14:28:08,900 DEBUG [c.c.c.CapacityManagerImpl]
> (API-Job-Executor-2:ctx-c03e0b1a job-16346 ctx-bbbd2b27) (logid:241b7d3f)
> Hosts's actual total CPU: 33600 and CPU after applying overprovisioning:
> 168000
> 2019-03-29 14:28:08,900 DEBUG [c.c.c.CapacityManagerImpl]
> (API-Job-Executor-2:ctx-c03e0b1a job-16346 ctx-bbbd2b27) (logid:241b7d3f)
> Hosts's actual total RAM: 67312107520 and RAM after applying
> overprovisioning: 67312107520
> 2019-03-29 14:28:08,900 DEBUG [c.c.c.CapacityManagerImpl]
> (API-Job-Executor-2:ctx-c03e0b1a job-16346 ctx-bbbd2b27) (logid:241b7d3f)
> release cpu from host: 20, old used: 67000,reserved: 0, actual total:
> 33600, total with overprovisioning: 168000; new used: 67000,reserved:0;
> movedfromreserved: true,moveToReserveredfalse
> 2019-03-29 14:28:08,900 DEBUG [c.c.c.CapacityManagerImpl]
> (API-Job-Executor-2:ctx-c03e0b1a job-16346 ctx-bbbd2b27) (logid:241b7d3f)
> release mem from host: 20, old used: 35165044736,reserved: 0, total:
> 67312107520; new used: 35165044736,reserved:0; movedfromreserved:
> true,moveToReserveredfalse
> 2019-03-29 14:28:08,902 DEBUG [c.c.n.NetworkModelImpl]
> (API-Job-Executor-2:ctx-c03e0b1a job-16346 ctx-bbbd2b27) (logid:241b7d3f)
> Unable to find default network for the vm; vm doesn't have default nic
> 2019-03-29 14:28:08,903 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (API-Job-Executor-2:ctx-c03e0b1a job-16346 ctx-bbbd2b27) (logid:241b7d3f)
> Destroying vm VM[DomainRouter|r-90-VM]
> 2019-03-29 14:28:08,903 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (API-Job-Executor-2:ctx-c03e0b1a job-16346 ctx-bbbd2b27) (logid:241b7d3f)
> Cleaning up NICS
> 2019-03-29 14:28:08,903 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (API-Job-Executor-2:ctx-c03e0b1a job-16346 ctx-bbbd2b27) (logid:241b7d3f)
> Cleaning network for vm: 90
> 2019-03-29 14:28:08,903 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (API-Job-Executor-2:ctx-c03e0b1a job-16346 ctx-bbbd2b27) (logid:241b7d3f)
> Cleaning up hypervisor data structures (ex. SRs in XenServer) for managed
> storage
> 2019-03-29 14:28:08,905 DEBUG [o.a.c.e.o.VolumeOrchestrator]
> (API-Job-Executor-2:ctx-c03e0b1a job-16346 ctx-bbbd2b27) (logid:241b7d3f)
> Cleaning storage for vm: 90
> 2019-03-29 14:28:08,907 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (API-Job-Executor-2:ctx-c03e0b1a job-16346 ctx-bbbd2b27) (logid:241b7d3f)
> Expunged VM[DomainRouter|r-90-VM]
>
> 2019-03-29 14:28:08,912 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (API-Job-Executor-2:ctx-c03e0b1a job-16346 ctx-bbbd2b27) (logid:241b7d3f)
> Asking VirtualRouter to implemenet Ntwk[204|Guest|7]
> 2019-03-29 14:28:08,917 ERROR [o.c.n.r.d.RouterDeploymentDefinition]
> (API-Job-Executor-2:ctx-c03e0b1a job-16346 ctx-bbbd2b27) (logid:241b7d3f)
> Didn't support redundant virtual router without public network!
> 2019-03-29 14:28:08,923 ERROR [c.c.a.ApiAsyncJobDispatcher]
> (API-Job-Executor-2:ctx-c03e0b1a job-16346) (logid:241b7d3f) Unexpected
> exception while executing
> org.apache.cloudstack.api.command.user.network.RestartNetworkCmd
> com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1]
> is unreachable: Can't find all necessary running routers!
> at
> com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:243)
> at
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElements(NetworkOrchestrator.java:1203)
> at
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.rollingRestartRouters(NetworkOrchestrator.java:2948)
> at
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:2853)
> at
> com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1883)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> at
> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> at
> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
> at com.sun.proxy.$Proxy229.restartNetwork(Unknown Source)
> at
> org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:99)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
> at
> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2019-03-29 14:28:08,924 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-2:ctx-c03e0b1a job-16346) (logid:241b7d3f) Complete async
> job-16346, jobStatus: FAILED, resultCode: 530, result:
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Resource
> [DataCenter:1] is unreachable: Can\u0027t find all necessary running
> routers!"}
> 2019-03-29 14:28:08,925 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-2:ctx-c03e0b1a job-16346) (logid:241b7d3f) Publish async
> job-16346 complete on message bus
> 2019-03-29 14:28:08,925 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-2:ctx-c03e0b1a job-16346) (logid:241b7d3f) Wake up jobs
> related to job-16346
> 2019-03-29 14:28:08,925 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-2:ctx-c03e0b1a job-16346) (logid:241b7d3f) Update db
> status for job-16346
> 2019-03-29 14:28:08,926 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-2:ctx-c03e0b1a job-16346) (logid:241b7d3f) Wake up jobs
> joined with job-16346 and disjoin all subjobs created from job- 16346
> 2019-03-29 14:28:08,928 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-2:ctx-c03e0b1a job-16346) (logid:241b7d3f) Done executing
> org.apache.cloudstack.api.command.user.network.RestartNetworkCmd for
> job-16346
> 2019-03-29 14:28:08,928 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (API-Job-Executor-2:ctx-c03e0b1a job-16346) (logid:241b7d3f) Remove
> job-16346 from job monitoring
> 2019-03-29 14:28:10,944 DEBUG [c.c.a.ApiServlet]
> (qtp788117692-18:ctx-8726f195) (logid:737a3a42) ===START=== 2.190.177.97
> -- GET
> command=queryAsyncJobResult&jobId=241b7d3f-7f79-4683-903f-bb1fbbb75929&response=json&_=1553866092909
> 2019-03-29 14:28:10,946 DEBUG [c.c.a.ApiServer]
> (qtp788117692-18:ctx-8726f195 ctx-67ab5d8c) (logid:737a3a42) CIDRs from
> which account 'Acct[27cd01ef-3907-11e9-87ab-a4bf012ed1a6-admin]' is allowed
> to perform API calls: 0.0.0.0/0,::/0
> 2019-03-29 14:28:10,953 DEBUG [c.c.a.ApiServlet]
> (qtp788117692-18:ctx-8726f195 ctx-67ab5d8c) (logid:737a3a42) ===END===
> 2.190.177.97 -- GET
> command=queryAsyncJobResult&jobId=241b7d3f-7f79-4683-903f-bb1fbbb75929&response=json&_=1553866092909
> 2019-03-29 14:28:14,878 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
> (AsyncJobMgr-Heartbeat-1:ctx-4dc5b694) (logid:513a416d) Begin cleanup
> expired async-jobs
> 2019-03-29 14:28:14,881 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
> (AsyncJobMgr-Heartbeat-1:ctx-4dc5b694) (logid:513a416d) End cleanup expired
> async-jobs
> 2019-03-29 14:28:15,572 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> (RouterStatusMonitor-1:ctx-adab5f84) (logid:ff323865) Found 0 routers to
> update status.
> 2019-03-29 14:28:15,572 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> (RouterStatusMonitor-1:ctx-adab5f84) (logid:ff323865) Found 0 VPC networks
> to update Redundant State.
> 2019-03-29 14:28:15,573 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> (RouterStatusMonitor-1:ctx-adab5f84) (logid:ff323865) Found 1 networks to
> update RvR status.
> 2019-03-29 14:28:15,627 DEBUG [c.c.a.AlertManagerImpl]
> (CapacityChecker:ctx-9754bf04) (logid:833ca1e8) Running Capacity Checker
> ...
> 2019-03-29 14:28:15,627 DEBUG [c.c.a.AlertManagerImpl]
> (CapacityChecker:ctx-9754bf04) (logid:833ca1e8) recalculating system
> capacity
> 2019-03-29 14:28:15,627 DEBUG [c.c.a.AlertManagerImpl]
> (CapacityChecker:ctx-9754bf04) (logid:833ca1e8) Executing cpu/ram capacity
> update
> 2019-03-29 14:28:15,636 DEBUG [c.c.c.CapacityManagerImpl]
> (CapacityChecker:ctx-9754bf04) (logid:833ca1e8) Found 22 VMs on host 20
> 2019-03-29 14:28:15,643 DEBUG [c.c.c.CapacityManagerImpl]
> (CapacityChecker:ctx-9754bf04) (logid:833ca1e8) Found 5 VM, not running on
> host 20
> 2019-03-29 14:28:15,647 DEBUG [c.c.c.CapacityManagerImpl]
> (CapacityChecker:ctx-9754bf04) (logid:833ca1e8) No need to calibrate cpu
> capacity, host:20 usedCpuCore: 39 reservedCpuCore: 0
> 2019-03-29 14:28:15,647 DEBUG [c.c.c.CapacityManagerImpl]
> (CapacityChecker:ctx-9754bf04) (logid:833ca1e8) No need to calibrate cpu
> capacity, host:20 usedCpu: 67000 reservedCpu: 0
> 2019-03-29 14:28:15,647 DEBUG [c.c.c.CapacityManagerImpl]
> (CapacityChecker:ctx-9754bf04) (logid:833ca1e8) No need to calibrate memory
> capacity, host:20 usedMem: 35165044736 reservedMem: 0
> 2019-03-29 14:28:15,648 DEBUG [c.c.a.AlertManagerImpl]
> (CapacityChecker:ctx-9754bf04) (logid:833ca1e8) Done executing cpu/ram
> capacity update
> 2019-03-29 14:28:15,648 DEBUG [c.c.a.AlertManagerImpl]
> (CapacityChecker:ctx-9754bf04) (logid:833ca1e8) Executing storage capacity
> update
> 2019-03-29 14:28:15,651 DEBUG [c.c.s.StorageManagerImpl]
> (CapacityChecker:ctx-9754bf04) (logid:833ca1e8) Found storage pool Primary1
> of type NetworkFilesystem with overprovisioning factor 2
> 2019-03-29 14:28:15,651 DEBUG [c.c.s.StorageManagerImpl]
> (CapacityChecker:ctx-9754bf04) (logid:833ca1e8) Total over provisioned
> capacity calculated is 2 * 855968317440
> 2019-03-29 14:28:15,651 DEBUG [c.c.s.StorageManagerImpl]
> (CapacityChecker:ctx-9754bf04) (logid:833ca1e8) Total over provisioned
> capacity of the pool Primary1 id: 1 is 1711936634880
> 2019-03-29 14:28:15,652 DEBUG [c.c.s.StorageManagerImpl]
> (CapacityChecker:ctx-9754bf04) (logid:833ca1e8) Successfully set Capacity -
> 1711936634880 for capacity type - 3 , DataCenterId - 1, HostOrPoolId - 1,
> PodId 1
> 2019-03-29 14:28:15,654 DEBUG [c.c.s.StorageManagerImpl]
> (CapacityChecker:ctx-9754bf04) (logid:833ca1e8) Found storage pool Primary2
> of type NetworkFilesystem with overprovisioning factor 2
> 2019-03-29 14:28:15,654 DEBUG [c.c.s.StorageManagerImpl]
> (CapacityChecker:ctx-9754bf04) (logid:833ca1e8) Total over provisioned
> capacity calculated is 2 * 3936540950528
> 2019-03-29 14:28:15,654 DEBUG [c.c.s.StorageManagerImpl]
> (CapacityChecker:ctx-9754bf04) (logid:833ca1e8) Total over provisioned
> capacity of the pool Primary2 id: 3 is 7873081901056
> 2019-03-29 14:28:15,655 DEBUG [c.c.s.StorageManagerImpl]
> (CapacityChecker:ctx-9754bf04) (logid:833ca1e8) Successfully set Capacity -
> 7873081901056 for capacity type - 3 , DataCenterId - 1, HostOrPoolId - 3,
> PodId 1
> 2019-03-29 14:28:15,655 DEBUG [c.c.a.AlertManagerImpl]
> (CapacityChecker:ctx-9754bf04) (logid:833ca1e8) Done executing storage
> capacity update
> 2019-03-29 14:28:15,655 DEBUG [c.c.a.AlertManagerImpl]
> (CapacityChecker:ctx-9754bf04) (logid:833ca1e8) Executing capacity updates
> for public ip and Vlans
> 2019-03-29 14:28:15,660 DEBUG [c.c.a.AlertManagerImpl]
> (CapacityChecker:ctx-9754bf04) (logid:833ca1e8) Done capacity updates for
> public ip and Vlans
> 2019-03-29 14:28:15,660 DEBUG [c.c.a.AlertManagerImpl]
> (CapacityChecker:ctx-9754bf04) (logid:833ca1e8) Executing capacity updates
> for private ip
> 2019-03-29 14:28:15,663 DEBUG [c.c.a.AlertManagerImpl]
> (CapacityChecker:ctx-9754bf04) (logid:833ca1e8) Done executing capacity
> updates for private ip
> 2019-03-29 14:28:15,663 DEBUG [c.c.a.AlertManagerImpl]
> (CapacityChecker:ctx-9754bf04) (logid:833ca1e8) Done recalculating system
> capacity
> 2019-03-29 14:28:15,677 DEBUG [c.c.a.AlertManagerImpl]
> (CapacityChecker:ctx-9754bf04) (logid:833ca1e8) Done running Capacity
> Checker ...
> 2019-03-29 14:28:16,699 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentManager-Handler-3:null) (logid:) SeqA 21-7010: Processing Seq
> 21-7010: { Cmd , MgmtId: -1, via: 21, Ver: v1, Flags: 11,
> [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":97,"_loadInfo":"{\n
> \"connections\": []\n}","wait":0}}] }
> 2019-03-29 14:28:16,700 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentManager-Handler-3:null) (logid:) SeqA 21-7010: Sending Seq 21-7010:
> { Ans: , MgmtId: 279278805450993, via: 21, Ver: v1, Flags: 100010,
> [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }
> 2019-03-29 14:28:18,575 DEBUG [c.c.a.ApiServlet]
> (qtp788117692-20:ctx-2b46bfb6) (logid:74fe6466) ===START=== 2.190.177.97
> -- GET command=listInfrastructure&response=json&_=1553866100546
> 2019-03-29 14:28:18,577 DEBUG [c.c.a.ApiServer]
> (qtp788117692-20:ctx-2b46bfb6 ctx-d0ed5d62) (logid:74fe6466) CIDRs from
> which account 'Acct[27cd01ef-3907-11e9-87ab-a4bf012ed1a6-admin]' is allowed
> to perform API calls: 0.0.0.0/0,::/0
> 2019-03-29 14:28:18,582 DEBUG [c.c.a.ApiServlet]
> (qtp788117692-20:ctx-2b46bfb6 ctx-d0ed5d62) (logid:74fe6466) ===END===
> 2.190.177.97 -- GET
> command=listInfrastructure&response=json&_=1553866100546
> 2019-03-29 14:28:24,878 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
> (AsyncJobMgr-Heartbeat-1:ctx-016e261d) (logid:8a7dc6cc) Begin cleanup
> expired async-jobs
> 2019-03-29 14:28:24,881 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
> (AsyncJobMgr-Heartbeat-1:ctx-016e261d) (logid:8a7dc6cc) End cleanup expired
> async-jobs
>
>
> On Fri, Mar 29, 2019 at 6:52 PM Ivan Kudryavtsev <kudryavtsev...@bw-sw.com>
> wrote:
>
>> Investigate the stacktrace in logs, share all related when you'll find.
>>
>> пт, 29 мар. 2019 г., 10:19 Fariborz Navidan <mdvlinqu...@gmail.com>:
>>
>> > Not for me! I think my CS DB is inconsistent somehow! I cannot figure
>> out
>> > which tables should I manipulate to fix it.
>> >
>> > On Fri, Mar 29, 2019 at 6:45 PM Ivan Kudryavtsev <
>> kudryavtsev...@bw-sw.com
>> > >
>> > wrote:
>> >
>> > > VR deletion is OK for Basic Zone in 4.11.2, work normally, VR is
>> created
>> > > automatically.
>> > >
>> > > пт, 29 мар. 2019 г., 10:04 Fariborz Navidan <mdvlinqu...@gmail.com>:
>> > >
>> > > > There should be inconsistency in DB. Because I did a wrong before
>> and
>> > > have
>> > > > deleted all records in domain_router and router_network_ref tables
>> > > > manually. I thought it will cause CS to think network is not
>> > implemented
>> > > > yet and will re-implement but it didn't. Indeed it still shows in
>> logs
>> > > > "Asking VirtualRouter to implement network...". "Setup" state of the
>> > > > network is preventing from re-creation of router. What can I chnage
>> > state
>> > > > of network to so that cause CS to create new router for it?
>> > > >
>> > > > Thanks
>> > > >
>> > > >
>> > > > On Fri, Mar 29, 2019 at 5:12 PM Fariborz Navidan <
>> > mdvlinqu...@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > 4.11.2 with KVM.
>> > > > >
>> > > > > On Fri, Mar 29, 2019 at 5:08 PM Andrija Panic <
>> > andrija.pa...@gmail.com
>> > > >
>> > > > > wrote:
>> > > > >
>> > > > >> cloudstack version, hypervisor (version) ?
>> > > > >>
>> > > > >> On Fri, 29 Mar 2019 at 13:34, Fariborz Navidan <
>> > mdvlinqu...@gmail.com
>> > > >
>> > > > >> wrote:
>> > > > >>
>> > > > >> > I did marked router vm instance to Stopped and set remoed to
>> NULL.
>> > > In
>> > > > >> > infrastructure overview it shows 1 virtual router but clicking
>> on
>> > > > View,
>> > > > >> > shows no data. Then I restarted the network with clleanup, It
>> said
>> > > > >> > [Datacenter:1] is unreachable and automatically marked the vm
>> > > instance
>> > > > >> to
>> > > > >> > destroyed and removed it (set removed field). But it did not
>> > create
>> > > > the
>> > > > >> > router again!
>> > > > >> >
>> > > > >> > On Fri, Mar 29, 2019 at 4:46 PM Thomas Joseph <
>> > > thomas.jo...@gmail.com
>> > > > >
>> > > > >> > wrote:
>> > > > >> >
>> > > > >> > > They running the restart network with cleanup=true
>> > > > >> > >
>> > > > >> > > On Fri, 29 Mar 2019, 10:39 am Fariborz Navidan, <
>> > > > >> mdvlinqu...@gmail.com>
>> > > > >> > > wrote:
>> > > > >> > >
>> > > > >> > > > It fails to restart networl. Log says it cannot find
>> virtual
>> > > > router.
>> > > > >> > > Bellow
>> > > > >> > > > is the log:
>> > > > >> > > >
>> > > > >> > > > 2019-03-29 10:53:41,621 DEBUG [c.c.c.ClusterManagerImpl]
>> > > > >> > > > (Cluster-Heartbeat-1:ctx-277a1d9d) (logid:9902f1d7)
>> Detected
>> > > > >> management
>> > > > >> > > > node left, id:8, nodeIP:178.33.230.41
>> > > > >> > > > 2019-03-29 10:53:41,621 INFO [c.c.c.ClusterManagerImpl]
>> > > > >> > > > (Cluster-Heartbeat-1:ctx-277a1d9d) (logid:9902f1d7) Trying
>> to
>> > > > >> connect
>> > > > >> > to
>> > > > >> > > > 178.33.230.41
>> > > > >> > > > 2019-03-29 10:53:41,621 INFO [c.c.c.ClusterManagerImpl]
>> > > > >> > > > (Cluster-Heartbeat-1:ctx-277a1d9d) (logid:9902f1d7)
>> Management
>> > > > node
>> > > > >> 8
>> > > > >> > is
>> > > > >> > > > detected inactive by timestamp but is pingable
>> > > > >> > > > 2019-03-29 10:53:42,526 DEBUG [c.c.a.ApiServlet]
>> > > > >> > > > (qtp788117692-21:ctx-219919fa) (logid:c3be28b0) ===START===
>> > > > >> > 2.190.177.97
>> > > > >> > > > -- GET
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> command=restartNetwork&response=json&id=6e644551-5aee-4c9b-a75f-134f544ee97c&cleanup=false&makeredundant=false&_=1553853224533
>> > > > >> > > > 2019-03-29 10:53:42,528 DEBUG [c.c.a.ApiServer]
>> > > > >> > > > (qtp788117692-21:ctx-219919fa ctx-3b77ed02)
>> (logid:c3be28b0)
>> > > CIDRs
>> > > > >> from
>> > > > >> > > > which account
>> > 'Acct[27cd01ef-3907-11e9-87ab-a4bf012ed1a6-admin]'
>> > > > is
>> > > > >> > > allowed
>> > > > >> > > > to perform API calls: 0.0.0.0/0,::/0
>> > > > >> > > > 2019-03-29 10:53:42,536 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>> > > > >> > > > (API-Job-Executor-2:ctx-5a2a6bbe job-16321)
>> (logid:b95430aa)
>> > Add
>> > > > >> > > job-16321
>> > > > >> > > > into job monitoring
>> > > > >> > > > 2019-03-29 10:53:42,539 DEBUG
>> > [o.a.c.f.j.i.AsyncJobManagerImpl]
>> > > > >> > > > (qtp788117692-21:ctx-219919fa ctx-3b77ed02)
>> (logid:c3be28b0)
>> > > > submit
>> > > > >> > async
>> > > > >> > > > job-16321, details: AsyncJobVO {id:16321, userId: 2,
>> > accountId:
>> > > 2,
>> > > > >> > > > instanceType: None, instanceId: null, cmd:
>> > > > >> > > >
>> > > org.apache.cloudstack.api.command.user.network.RestartNetworkCmd,
>> > > > >> > > cmdInfo:
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> {"makeredundant":"false","cleanup":"false","response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"3200","id":"6e644551-5aee-4c9b-a75f-134f544ee97c","ctxDetails":"{\"interface
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> com.cloud.network.Network\":\"6e644551-5aee-4c9b-a75f-134f544ee97c\"}","ctxAccountId":"2","uuid":"6e644551-5aee-4c9b-a75f-134f544ee97c","cmdEventType":"NETWORK.RESTART","_":"1553853224533"},
>> > > > >> > > > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0,
>> > > resultCode:
>> > > > 0,
>> > > > >> > > > result: null, initMsid: 279278805450993, completeMsid:
>> null,
>> > > > >> > lastUpdated:
>> > > > >> > > > null, lastPolled: null, created: null}
>> > > > >> > > > 2019-03-29 10:53:42,540 DEBUG
>> > [o.a.c.f.j.i.AsyncJobManagerImpl]
>> > > > >> > > > (API-Job-Executor-2:ctx-5a2a6bbe job-16321)
>> (logid:155f349b)
>> > > > >> Executing
>> > > > >> > > > AsyncJobVO {id:16321, userId: 2, accountId: 2,
>> instanceType:
>> > > None,
>> > > > >> > > > instanceId: null, cmd:
>> > > > >> > > >
>> > > org.apache.cloudstack.api.command.user.network.RestartNetworkCmd,
>> > > > >> > > cmdInfo:
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> {"makeredundant":"false","cleanup":"false","response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"3200","id":"6e644551-5aee-4c9b-a75f-134f544ee97c","ctxDetails":"{\"interface
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> com.cloud.network.Network\":\"6e644551-5aee-4c9b-a75f-134f544ee97c\"}","ctxAccountId":"2","uuid":"6e644551-5aee-4c9b-a75f-134f544ee97c","cmdEventType":"NETWORK.RESTART","_":"1553853224533"},
>> > > > >> > > > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0,
>> > > resultCode:
>> > > > 0,
>> > > > >> > > > result: null, initMsid: 279278805450993, completeMsid:
>> null,
>> > > > >> > lastUpdated:
>> > > > >> > > > null, lastPolled: null, created: null}
>> > > > >> > > > 2019-03-29 10:53:42,541 DEBUG [c.c.a.ApiServlet]
>> > > > >> > > > (qtp788117692-21:ctx-219919fa ctx-3b77ed02)
>> (logid:c3be28b0)
>> > > > >> ===END===
>> > > > >> > > > 2.190.177.97 -- GET
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> command=restartNetwork&response=json&id=6e644551-5aee-4c9b-a75f-134f544ee97c&cleanup=false&makeredundant=false&_=1553853224533
>> > > > >> > > > 2019-03-29 10:53:42,551 DEBUG
>> [o.a.c.e.o.NetworkOrchestrator]
>> > > > >> > > > (API-Job-Executor-2:ctx-5a2a6bbe job-16321 ctx-a30b75bf)
>> > > > >> > (logid:155f349b)
>> > > > >> > > > Restarting network 204...
>> > > > >> > > > 2019-03-29 10:53:42,552 DEBUG
>> [o.a.c.e.o.NetworkOrchestrator]
>> > > > >> > > > (API-Job-Executor-2:ctx-5a2a6bbe job-16321 ctx-a30b75bf)
>> > > > >> > (logid:155f349b)
>> > > > >> > > > Implementing the network Ntwk[204|Guest|7] elements and
>> > > resources
>> > > > >> as a
>> > > > >> > > part
>> > > > >> > > > of network restart without cleanup
>> > > > >> > > > 2019-03-29 10:53:42,554 DEBUG
>> [o.a.c.e.o.NetworkOrchestrator]
>> > > > >> > > > (API-Job-Executor-2:ctx-5a2a6bbe job-16321 ctx-a30b75bf)
>> > > > >> > (logid:155f349b)
>> > > > >> > > > Asking VirtualRouter to implemenet Ntwk[204|Guest|7]
>> > > > >> > > > 2019-03-29 10:53:42,558 ERROR
>> > > > [o.c.n.r.d.RouterDeploymentDefinition]
>> > > > >> > > > (API-Job-Executor-2:ctx-5a2a6bbe job-16321 ctx-a30b75bf)
>> > > > >> > (logid:155f349b)
>> > > > >> > > > Didn't support redundant virtual router without public
>> > network!
>> > > > >> > > > 2019-03-29 10:53:42,560 WARN
>> [o.a.c.e.o.NetworkOrchestrator]
>> > > > >> > > > (API-Job-Executor-2:ctx-5a2a6bbe job-16321 ctx-a30b75bf)
>> > > > >> > (logid:155f349b)
>> > > > >> > > > Failed to implement network Ntwk[204|Guest|7] elements and
>> > > > resources
>> > > > >> > as a
>> > > > >> > > > part of network restart due to
>> > > > >> > > > com.cloud.exception.ResourceUnavailableException: Resource
>> > > > >> > [DataCenter:1]
>> > > > >> > > > is unreachable: Can't find all necessary running routers!
>> > > > >> > > > at
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:243)
>> > > > >> > > > at
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElements(NetworkOrchestrator.java:1203)
>> > > > >> > > > at
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElementsAndResources(NetworkOrchestrator.java:1138)
>> > > > >> > > > at
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:2862)
>> > > > >> > > > at
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1883)
>> > > > >> > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)
>> > > > >> > > > at
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> > > > >> > > > at
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> > > > >> > > > at java.lang.reflect.Method.invoke(Method.java:498)
>> > > > >> > > > at
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
>> > > > >> > > > at
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
>> > > > >> > > > at
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
>> > > > >> > > > at
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
>> > > > >> > > > at
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
>> > > > >> > > > at
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
>> > > > >> > > > at
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
>> > > > >> > > > at
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
>> > > > >> > > > at
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
>> > > > >> > > > at
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
>> > > > >> > > > at com.sun.proxy.$Proxy229.restartNetwork(Unknown Source)
>> > > > >> > > > at
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:99)
>> > > > >> > > > at
>> > com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
>> > > > >> > > > at
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
>> > > > >> > > > at
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
>> > > > >> > > > at
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> > > > >> > > > at
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> > > > >> > > > at
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> > > > >> > > > at
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> > > > >> > > > at
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> > > > >> > > > at
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
>> > > > >> > > > at
>> > > > >> > >
>> > > > >>
>> > >
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> > > > >> > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> > > > >> > > > at
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> > > > >> > > > at
>> > > > >> > > >
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> > > > >> > > > at java.lang.Thread.run(Thread.java:748)
>> > > > >> > > >
>> > > > >> > > > On Fri, Mar 29, 2019 at 11:08 AM Thomas Joseph <
>> > > > >> thomas.jo...@gmail.com
>> > > > >> > >
>> > > > >> > > > wrote:
>> > > > >> > > >
>> > > > >> > > > > Restart the network.
>> > > > >> > > > >
>> > > > >> > > > > On Thu, 28 Mar 2019, 10:59 pm Fariborz Navidan, <
>> > > > >> > mdvlinqu...@gmail.com
>> > > > >> > > >
>> > > > >> > > > > wrote:
>> > > > >> > > > >
>> > > > >> > > > > > Hello,
>> > > > >> > > > > > I stopped and deleted virtual router to see how it
>> affects
>> > > the
>> > > > >> > > changed
>> > > > >> > > > > > network settings. I tought CS will recreate it
>> > automatically
>> > > > >> but it
>> > > > >> > > > > didn't
>> > > > >> > > > > > and now I cannot start new VMs because it complains
>> about
>> > > > >> > > insufficient
>> > > > >> > > > > data
>> > > > >> > > > > > capacity. How do I force CS to re-create the router?
>> > > > >> > > > > >
>> > > > >> > > > >
>> > > > >> > > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > > >>
>> > > > >> --
>> > > > >>
>> > > > >> Andrija Panić
>> > > > >>
>> > > > >
>> > > >
>> > >
>> >
>>
>
