Somewhere in your error, I think that I saw a reference to rolling reboot. Try
disabling that in the global settings.
( network.rolling.restart )
paul.an...@shapeblue.com
www.shapeblue.com
Amadeus House, Floral Street, London WC2E 9DPUK
@shapeblue
-----Original Message-----
From: Jevgeni Zolotarjov <j.zolotar...@gmail.com>
Sent: 19 March 2019 17:29
To: users@cloudstack.apache.org
Subject: Re: Disaster after maintenance
Guys, please help with it.
What can be done here?
There is too much valuable data.
On Tue, Mar 19, 2019 at 4:21 PM Jevgeni Zolotarjov <j.zolotar...@gmail.com>
wrote:
> Tried that just now and got error:
> Resource [DataCenter:1] is unreachable: Can't find all necessary
> running routers!
>
> In the log I see:
> =============================================
>
> 2019-03-19 14:20:39,644 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (API-Job-Executor-4:ctx-7b6b69eb job-5093 ctx-9be30648)
> (logid:265a6099) Restarting network 204...
> 2019-03-19 14:20:39,645 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (API-Job-Executor-4:ctx-7b6b69eb job-5093 ctx-9be30648)
> (logid:265a6099) Performing rolling restart of routers of network
> Ntwk[204|Guest|6]
> 2019-03-19 14:20:39,649 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (API-Job-Executor-4:ctx-7b6b69eb job-5093 ctx-9be30648)
> (logid:265a6099) Asking VirtualRouter to implemenet Ntwk[204|Guest|6]
> 2019-03-19 14:20:39,658 ERROR [c.c.a.ApiAsyncJobDispatcher]
> (API-Job-Executor-4:ctx-7b6b69eb job-5093) (logid:265a6099) Unexpected
> exception while executing
> org.apache.cloudstack.api.command.user.network.RestartNetworkCmd
> com.cloud.exception.ResourceUnavailableException: Resource
> [DataCenter:1] is unreachable: Can't find all necessary running routers!
> at
> com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:243)
> at
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElements(NetworkOrchestrator.java:1203)
> at
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.rollingRestartRouters(NetworkOrchestrator.java:2948)
> at
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:2853)
> at
> com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1883)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> at
> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> at
> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
> at com.sun.proxy.$Proxy229.restartNetwork(Unknown Source)
> at
> org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:99)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
> at
> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown
> Source)
> at java.util.concurrent.FutureTask.run(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
> at java.lang.Thread.run(Unknown Source)
> 2019-03-19 14:20:39,659 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-4:ctx-7b6b69eb job-5093) (logid:265a6099) Complete
> async job-5093, jobStatus: FAILED, resultCode: 530, result:
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":
> [],"errorcode":530,"errortext":"Resource
> [DataCenter:1] is un
>
> On Tue, Mar 19, 2019 at 4:19 PM Andrija Panic
> <andrija.pa...@shapeblue.com>
> wrote:
>
>>
>> Your network can't be deleted due to "Can't delete the network, not
>> all user vms are expunged. Vm VM[User|i-2-11-VM] is in Stopped state"
>> - which is fine.
>>
>> You should be able to just start the user VM - but if you have
>> actually delete the VR itself, then just do Network restart with
>> "cleanup" and it will recreate a new VR, after which you should be able to
>> start the VM.
>>
>> Andrija
>>
>> andrija.pa...@shapeblue.com
>> www.shapeblue.com
>> Amadeus House, Floral Street, London WC2E 9DPUK @shapeblue
>>
>>
>>
>>
>> -----Original Message-----
>> From: Jevgeni Zolotarjov <j.zolotar...@gmail.com>
>> Sent: 19 March 2019 15:10
>> To: users@cloudstack.apache.org
>> Subject: Re: Disaster after maintenance
>>
>> I mean I cannot delete network: In the management server log I see
>>
>> ==========================================================
>> 019-03-19 14:06:36,316 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-1:ctx-1c0fd4dc job-5090) (logid:c734edfc) Executing
>> AsyncJobVO {id:5090, userId: 2, accountId: 2, instanceType: Network,
>> instanceId: 204, cmd:
>> org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd, cmdInfo:
>>
>> {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"2641","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\"interface
>>
>> com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}","ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmdEventType":"NETWORK.DELETE","_":"1553004396247"},
>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
>> result: null, initMsid: 264216221068220, completeMsid: null, lastUpdated:
>> null, lastPolled: null, created: null}
>> 2019-03-19 14:06:36,351 WARN [o.a.c.e.o.NetworkOrchestrator]
>> (API-Job-Executor-1:ctx-1c0fd4dc job-5090 ctx-134954fa) (logid:c734edfc)
>> Can't delete the network, not all user vms are expunged. Vm
>> VM[User|i-2-11-VM] is in Stopped state
>> 2019-03-19 14:06:36,356 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-1:ctx-1c0fd4dc job-5090) (logid:c734edfc) Complete async
>> job-5090, jobStatus: FAILED, resultCode: 530, result:
>>
>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
>> to delete network"}
>> ==========================================================
>>
>>
>> I deleted a router, expecting it to be recreated on deleting network. But
>> I am unable to delete network because of above error
>>
>> On Tue, Mar 19, 2019 at 3:58 PM Jevgeni Zolotarjov <
>> j.zolotar...@gmail.com>
>> wrote:
>>
>> > I've managed to make libvirtd running
>> > Now cloudstack console shows both hosts - running
>> >
>> > But now as I have removed network, VMs are unable to start.
>> >
>> > How can I recreate the network now?
>> >
>> > On Tue, Mar 19, 2019 at 3:14 PM Ivan Kudryavtsev
>> > <kudryavtsev...@bw-sw.com>
>> > wrote:
>> >
>> >> Jevgeniy, it may be a documentation bug. Take s look:
>> >> https://github.com/apache/cloudstack-documentation/pull/27/files
>> >>
>> >> вт, 19 мар. 2019 г., 9:09 Jevgeni Zolotarjov <j.zolotar...@gmail.com>:
>> >>
>> >> > That's it - libvirtd failed to start on second host.
>> >> > Tried restarting, but it does not start.
>> >> >
>> >> >
>> >> > >> Do you have some NUMA constraints or anything which requires
>> >> particular
>> >> > RAM configuration?
>> >> > No
>> >> >
>> >> > libvirtd.service - Virtualization daemon
>> >> > Loaded: loaded (/usr/lib/systemd/system/libvirtd.service;
>> >> > enabled; vendor preset: enabled)
>> >> > Active: failed (Result: start-limit) since Tue 2019-03-19
>> >> > 13:03:07
>> >> GMT;
>> >> > 12s ago
>> >> > Docs: man:libvirtd(8)
>> >> > https://libvirt.org
>> >> > Process: 892 ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS
>> >> > (code=exited,
>> >> > status=1/FAILURE)
>> >> > Main PID: 892 (code=exited, status=1/FAILURE)
>> >> > Tasks: 19 (limit: 32768)
>> >> > CGroup: /system.slice/libvirtd.service
>> >> > ├─11338 /usr/sbin/libvirtd -d -l
>> >> > ├─11909 /usr/sbin/dnsmasq
>> >> > --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro
>> >> > --dhcp-script=/usr/libexec/libvirt_leaseshelper
>> >> > └─11910 /usr/sbin/dnsmasq
>> >> > --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro
>> >> > --dhcp-script=/usr/libexec/libvirt_leaseshelper
>> >> >
>> >> > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Failed to
>> >> > start Virtualization daemon.
>> >> > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Unit
>> >> > libvirtd.service entered failed state.
>> >> > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]:
>> >> libvirtd.service
>> >> > failed.
>> >> > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]:
>> >> libvirtd.service
>> >> > holdoff time over, scheduling restart.
>> >> > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Stopped
>> >> > Virtualization daemon.
>> >> > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: start
>> >> > request repeated too quickly for libvirtd.service Mar 19 13:03:07
>> >> > mtl1-apphst04.mt.pbt.com.mt systemd[1]: Failed to start
>> >> > Virtualization daemon.
>> >> > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Unit
>> >> > libvirtd.service entered failed state.
>> >> > Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]:
>> >> libvirtd.service
>> >> > failed.
>> >> >
>> >> >
>> >> > On Tue, Mar 19, 2019 at 3:04 PM Paul Angus
>> >> > <paul.an...@shapeblue.com>
>> >> > wrote:
>> >> >
>> >> > > Can you check that the cloudstack agent is running on the host
>> >> > > and the agent logs (usual logs directory) Also worth checking
>> >> > > that libvirt has started ok. Do you have some
>> >> NUMA
>> >> > > constraints or anything which requires particular RAM
>> configuration?
>> >> > >
>> >> > > paul.an...@shapeblue.com
>> >> > > www.shapeblue.com
>> >> > > Amadeus House, Floral Street, London WC2E 9DPUK @shapeblue
>> >> > >
>> >> > >
>> >> > >
>> >> > >
>> >> > > -----Original Message-----
>> >> > > From: Jevgeni Zolotarjov <j.zolotar...@gmail.com>
>> >> > > Sent: 19 March 2019 14:49
>> >> > > To: users@cloudstack.apache.org
>> >> > > Subject: Re: Disaster after maintenance
>> >> > >
>> >> > > Can you try migrating a VM to the server that you changed the RAM
>> >> amount?
>> >> > >
>> >> > > Also:
>> >> > > What is the hypervisor version?
>> >> > > KVM
>> >> > > QEMU Version : 2.0.0
>> >> > > Release : 1.el7.6
>> >> > >
>> >> > >
>> >> > > Host status in ACS?
>> >> > > 1st server: Unsecure
>> >> > > 2nd server: Disconnected
>> >> > >
>> >> > > Did you try to force a VM to start/deploy in this server where you
>> >> > changed
>> >> > > the RAM?
>> >> > > Host status became disconnected. I don't know how to make it
>> >> "connected"
>> >> > > again
>> >> > >
>> >> > >
>> >> > >
>> >> > > On Tue, Mar 19, 2019 at 2:42 PM Rafael Weingärtner <
>> >> > > rafaelweingart...@gmail.com> wrote:
>> >> > >
>> >> > > > Can you try migrating a VM to the server that you changed the RAM
>> >> > amount?
>> >> > > >
>> >> > > > Also:
>> >> > > > What is the hypervisor version?
>> >> > > > Host status in ACS?
>> >> > > > Did you try to force a VM to start/deploy in this server where
>> you
>> >> > > > changed the RAM?
>> >> > > >
>> >> > > >
>> >> > > > On Tue, Mar 19, 2019 at 9:39 AM Jevgeni Zolotarjov
>> >> > > > <j.zolotar...@gmail.com
>> >> > > > >
>> >> > > > wrote:
>> >> > > >
>> >> > > > > We have Cloudstack 4.11.2 setup running fine for few months
>> (>4)
>> >> The
>> >> > > > > setup is very simple: 2 hosts We decided to do a maintenance to
>> >> > > > > increase RAM on both servers
>> >> > > > >
>> >> > > > > For this we put first server to maintenance. All VMS moved to
>> >> second
>> >> > > > > host after a while.
>> >> > > > >
>> >> > > > > Then first server was shutdown, RAM increased, server turned
>> ON.
>> >> > > > > Now nothing starts on first server.
>> >> > > > >
>> >> > > > >
>> >> > > > > Tried to delete network, but this fails as well
>> >> > > > >
>> >> > > > > Please help !
>> >> > > > >
>> >> > > > > Here is extract from log:
>> >> > > > > ======================================
>> >> > > > > 2019-03-19 12:27:53,064 DEBUG
>> >> [o.a.c.s.SecondaryStorageManagerImpl]
>> >> > > > > (secstorage-1:ctx-16d6c797) (logid:7e3160ce) Zone 1 is ready to
>> >> > > > > launch secondary storage VM
>> >> > > > > 2019-03-19 12:27:53,125 DEBUG [c.c.c.ConsoleProxyManagerImpl]
>> >> > > > > (consoleproxy-1:ctx-cbd034b9) (logid:0a8c8bf4) Zone 1 is ready
>> to
>> >> > > > > launch console proxy
>> >> > > > > 2019-03-19 12:27:53,181 DEBUG [c.c.a.ApiServlet]
>> >> > > > > (qtp510113906-285:ctx-6c5e11c3) (logid:cd8e30be) ===START===
>> >> > > > 192.168.5.140
>> >> > > > > -- GET
>> >> > > > >
>> >> > > > >
>> >> > > >
>> >> command=deleteNetwork&id=4ba834ed-48f3-468f-b667-9bb2d2c258f1&response
>> >> > > > =json&_=1552998473154
>> >> > > > > 2019-03-19 12:27:53,186 DEBUG [c.c.a.ApiServer]
>> >> > > > > (qtp510113906-285:ctx-6c5e11c3 ctx-0cc34dc6) (logid:cd8e30be)
>> >> CIDRs
>> >> > > > > from which account
>> >> > > > > 'Acct[15863393-8e8d-11e7-8f52-f04da2002bbe-admin]' is
>> >> > > > allowed
>> >> > > > > to perform API calls: 0.0.0.0/0,::/0
>> >> > > > > 2019-03-19 12:27:53,208 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>> >> > > > > (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:f6751fa7) Add
>> >> > > > > job-5081 into job monitoring
>> >> > > > > 2019-03-19 12:27:53,209 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> >> > > > > (qtp510113906-285:ctx-6c5e11c3 ctx-0cc34dc6) (logid:cd8e30be)
>> >> submit
>> >> > > > async
>> >> > > > > job-5081, details: AsyncJobVO {id:5081, userId: 2, accountId:
>> 2,
>> >> > > > > instanceType: Network, instanceId: 204, cmd:
>> >> > > > >
>> org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd,
>> >> > > cmdInfo:
>> >> > > > >
>> >> > > > >
>> >> > > >
>> >> {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId
>> >> > > >
>> >> ":"2615","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\"
>> >> > > > interface
>> >> > > > >
>> >> > > > >
>> >> > > >
>> >> com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}"
>> >> > > >
>> >> ,"ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmd
>> >> > > > EventType":"NETWORK.DELETE","_":"1552998473154"},
>> >> > > > > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0,
>> resultCode:
>> >> 0,
>> >> > > > > result: null, initMsid: 264216221068220, completeMsid: null,
>> >> > > lastUpdated:
>> >> > > > > null, lastPolled: null, created: null}
>> >> > > > > 2019-03-19 12:27:53,211 DEBUG [c.c.a.ApiServlet]
>> >> > > > > (qtp510113906-285:ctx-6c5e11c3 ctx-0cc34dc6) (logid:cd8e30be)
>> >> > > > > ===END===
>> >> > > > > 192.168.5.140 -- GET
>> >> > > > >
>> >> > > > >
>> >> > > >
>> >> command=deleteNetwork&id=4ba834ed-48f3-468f-b667-9bb2d2c258f1&response
>> >> > > > =json&_=1552998473154
>> >> > > > > 2019-03-19 12:27:53,212 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> >> > > > > (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6)
>> >> > > > > Executing AsyncJobVO {id:5081, userId: 2, accountId: 2,
>> >> > > > > instanceType: Network,
>> >> > > > > instanceId: 204, cmd:
>> >> > > > >
>> org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd,
>> >> > > cmdInfo:
>> >> > > > >
>> >> > > > >
>> >> > > >
>> >> {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId
>> >> > > >
>> >> ":"2615","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\"
>> >> > > > interface
>> >> > > > >
>> >> > > > >
>> >> > > >
>> >> com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}"
>> >> > > >
>> >> ,"ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmd
>> >> > > > EventType":"NETWORK.DELETE","_":"1552998473154"},
>> >> > > > > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0,
>> resultCode:
>> >> 0,
>> >> > > > > result: null, initMsid: 264216221068220, completeMsid: null,
>> >> > > lastUpdated:
>> >> > > > > null, lastPolled: null, created: null}
>> >> > > > > 2019-03-19 12:27:53,257 WARN [o.a.c.e.o.NetworkOrchestrator]
>> >> > > > > (API-Job-Executor-1:ctx-d4970c19 job-5081 ctx-d5de7979)
>> >> > > > > (logid:16897ea6) Can't delete the network, not all user vms are
>> >> > > > > expunged. Vm VM[User|i-2-11-VM] is in Stopped state
>> >> > > > > 2019-03-19 12:27:53,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> >> > > > > (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6)
>> >> Complete
>> >> > > > async
>> >> > > > > job-5081, jobStatus: FAILED, resultCode: 530, result:
>> >> > > > >
>> >> > > > >
>> >> > > >
>> >> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":
>> >> > > > [],"errorcode":530,"errortext":"Failed
>> >> > > > > to delete network"}
>> >> > > > > 2019-03-19 12:27:53,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> >> > > > > (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6)
>> >> Publish
>> >> > > > > async
>> >> > > > > job-5081 complete on message bus
>> >> > > > > 2019-03-19 12:27:53,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> >> > > > > (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6)
>> Wake
>> >> up
>> >> > > > > jobs related to job-5081
>> >> > > > > 2019-03-19 12:27:53,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> >> > > > > (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6)
>> Update
>> >> > > > > db status for job-5081
>> >> > > > > 2019-03-19 12:27:53,265 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> >> > > > > (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6)
>> Wake
>> >> up
>> >> > > > > jobs joined with job-5081 and disjoin all subjobs created from
>> >> job-
>> >> > > > > 5081
>> >> > > > > 2019-03-19 12:27:53,267 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> >> > > > > (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6)
>> Done
>> >> > > > executing
>> >> > > > > org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd
>> >> for
>> >> > > > > job-5081
>> >> > > > > 2019-03-19 12:27:53,267 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>> >> > > > > (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6)
>> Remove
>> >> > > > job-5081
>> >> > > > > from job monitoring
>> >> > > > > 2019-03-19 12:27:56,230 DEBUG [c.c.a.ApiServlet]
>> >> > > > > (qtp510113906-28:ctx-e6c5bc85) (logid:7fe68f75) ===START===
>> >> > > > 192.168.5.140
>> >> > > > > -- GET
>> >> > > > >
>> >> > > > >
>> >> > > >
>> >> command=queryAsyncJobResult&jobId=16897ea6-27c3-45b9-a0df-ab217bb5393c
>> >> > > > &response=json&_=1552998476202
>> >> > > > > 2019-03-19 12:27:56,238 DEBUG [c.c.a.ApiServer]
>> >> > > > > (qtp510113906-28:ctx-e6c5bc85 ctx-da1f4cbd) (logid:7fe68f75)
>> CIDRs
>> >> > > > > from which account
>> >> > > > > 'Acct[15863393-8e8d-11e7-8f52-f04da2002bbe-admin]' is
>> >> > > > allowed
>> >> > > > > to perform API calls: 0.0.0.0/0,::/0
>> >> > > > > 2019-03-19 12:27:56,260 DEBUG [c.c.a.ApiServlet]
>> >> > > > > (qtp510113906-28:ctx-e6c5bc85 ctx-da1f4cbd) (logid:7fe68f75)
>> >> > > > > ===END===
>> >> > > > > 192.168.5.140 -- GET
>> >> > > > >
>> >> > > > >
>> >> > > >
>> >> command=queryAsyncJobResult&jobId=16897ea6-27c3-45b9-a0df-ab217bb5393c
>> >> > > > &response=json&_=1552998476202
>> >> > > > > 2019-03-19 12:28:00,946 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
>> >> > > > > (AsyncJobMgr-Heartbeat-1:ctx-9b43d1fd) (logid:a605267a) Begin
>> >> > > > > cleanup expired async-jobs
>> >> > > > > 2019-03-19 12:28:00,951 INFO [o.a.c.f.j.i.AsyncJobManagerImpl]
>> >> > > > > (AsyncJobMgr-Heartbeat-1:ctx-9b43d1fd) (logid:a605267a) End
>> >> cleanup
>> >> > > > expired
>> >> > > > > async-jobs
>> >> > > > > 2019-03-19 12:28:01,142 DEBUG
>> >> > > > [c.c.n.r.VirtualNetworkApplianceManagerImpl]
>> >> > > > > (RouterStatusMonitor-1:ctx-ad6bbe7e) (logid:04e4c72b) Found 0
>> >> > > > > routers to update status.
>> >> > > > >
>> >> > > >
>> >> > > >
>> >> > > > --
>> >> > > > Rafael Weingärtner
>> >> > > >
>> >> > >
>> >> >
>> >>
>> >
>>
>
