Disclaimer: from mobile device. I don't see any special failure in agent log, except some long running migration, timeout for graceful VM shutdown etc (and agent restart)
You did not send mgmt log after last network restart failure ? On Tue, Mar 19, 2019, 17:29 Jevgeni Zolotarjov <j.zolotar...@gmail.com> wrote: > Under > Infrastructure / Hosts: > both hosts are enabled and green (Unsecure though) > > agent.log - > https://drive.google.com/open?id=1rATxHKqgNKo2kD23BtlrZy_9gFXC-Bq- > > I set network.rolling.restart to false now. Restarted management server - > same problem - cannot restart not delete network > > > > On Tue, Mar 19, 2019 at 5:48 PM Boris Stoyanov < > boris.stoya...@shapeblue.com> > wrote: > > > Hi, you shouldn’t be worried about data as long as you use a separate > > shared storage. > > > > What does the cloudstack console say about your host? Is it up? If it’s > up > > then you should be able to deploy a VM on it. > > > > If not, you’ll need to investigate the reason cloudstack-agent is not > > connected, is the agent running? If not just start it and see if it > > connects. > > > > If it’s running but still not connected please share the agent.log. > > > > If nothing helps at all you can remove the host and add it again, but > > let’s try to troubleshoot first. > > > > Bobby. > > > > > > boris.stoya...@shapeblue.com > > www.shapeblue.com > > Amadeus House, Floral Street, London WC2E 9DPUK > > @shapeblue > > > > > > > > > On 19 Mar 2019, at 17:29, Jevgeni Zolotarjov <j.zolotar...@gmail.com> > > wrote: > > > > > > Guys, please help with it. > > > What can be done here? > > > There is too much valuable data. > > > > > > On Tue, Mar 19, 2019 at 4:21 PM Jevgeni Zolotarjov < > > j.zolotar...@gmail.com> > > > wrote: > > > > > >> Tried that just now and got error: > > >> Resource [DataCenter:1] is unreachable: Can't find all necessary > running > > >> routers! > > >> > > >> In the log I see: > > >> ============================================= > > >> > > >> 2019-03-19 14:20:39,644 DEBUG [o.a.c.e.o.NetworkOrchestrator] > > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093 ctx-9be30648) > (logid:265a6099) > > >> Restarting network 204... > > >> 2019-03-19 14:20:39,645 DEBUG [o.a.c.e.o.NetworkOrchestrator] > > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093 ctx-9be30648) > (logid:265a6099) > > >> Performing rolling restart of routers of network Ntwk[204|Guest|6] > > >> 2019-03-19 14:20:39,649 DEBUG [o.a.c.e.o.NetworkOrchestrator] > > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093 ctx-9be30648) > (logid:265a6099) > > >> Asking VirtualRouter to implemenet Ntwk[204|Guest|6] > > >> 2019-03-19 14:20:39,658 ERROR [c.c.a.ApiAsyncJobDispatcher] > > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093) (logid:265a6099) Unexpected > > >> exception while executing > > >> org.apache.cloudstack.api.command.user.network.RestartNetworkCmd > > >> com.cloud.exception.ResourceUnavailableException: Resource > > [DataCenter:1] > > >> is unreachable: Can't find all necessary running routers! > > >> at > > >> > > > com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:243) > > >> at > > >> > > > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElements(NetworkOrchestrator.java:1203) > > >> at > > >> > > > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.rollingRestartRouters(NetworkOrchestrator.java:2948) > > >> at > > >> > > > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:2853) > > >> at > > >> > > > com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1883) > > >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > >> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) > > >> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown > > Source) > > >> at java.lang.reflect.Method.invoke(Unknown Source) > > >> at > > >> > > > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338) > > >> at > > >> > > > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197) > > >> at > > >> > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) > > >> at > > >> > > > org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107) > > >> at > > >> > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) > > >> at > > >> > > > com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51) > > >> at > > >> > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) > > >> at > > >> > > > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) > > >> at > > >> > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) > > >> at > > >> > > > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) > > >> at com.sun.proxy.$Proxy229.restartNetwork(Unknown Source) > > >> at > > >> > > > org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:99) > > >> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150) > > >> at > > >> > > > com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108) > > >> at > > >> > > > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581) > > >> at > > >> > > > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > > >> at > > >> > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > > >> at > > >> > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > > >> at > > >> > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > > >> at > > >> > > > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > > >> at > > >> > > > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529) > > >> at java.util.concurrent.Executors$RunnableAdapter.call(Unknown > > >> Source) > > >> at java.util.concurrent.FutureTask.run(Unknown Source) > > >> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown > > >> Source) > > >> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown > > >> Source) > > >> at java.lang.Thread.run(Unknown Source) > > >> 2019-03-19 14:20:39,659 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093) (logid:265a6099) Complete > > async > > >> job-5093, jobStatus: FAILED, resultCode: 530, result: > > >> > > > org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Resource > > >> [DataCenter:1] is un > > >> > > >> On Tue, Mar 19, 2019 at 4:19 PM Andrija Panic < > > andrija.pa...@shapeblue.com> > > >> wrote: > > >> > > >>> > > >>> Your network can't be deleted due to "Can't delete the network, not > all > > >>> user vms are expunged. Vm > > >>> VM[User|i-2-11-VM] is in Stopped state" - which is fine. > > >>> > > >>> You should be able to just start the user VM - but if you have > actually > > >>> delete the VR itself, then just do Network restart with "cleanup" and > > it > > >>> will recreate a new VR, after which you should be able to start the > VM. > > >>> > > >>> Andrija > > >>> > > >>> andrija.pa...@shapeblue.com > > >>> www.shapeblue.com > > >>> Amadeus House, Floral Street, London WC2E 9DPUK > > >>> @shapeblue > > >>> > > >>> > > >>> > > >>> > > >>> -----Original Message----- > > >>> From: Jevgeni Zolotarjov <j.zolotar...@gmail.com> > > >>> Sent: 19 March 2019 15:10 > > >>> To: users@cloudstack.apache.org > > >>> Subject: Re: Disaster after maintenance > > >>> > > >>> I mean I cannot delete network: In the management server log I see > > >>> > > >>> ========================================================== > > >>> 019-03-19 14:06:36,316 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > > >>> (API-Job-Executor-1:ctx-1c0fd4dc job-5090) (logid:c734edfc) Executing > > >>> AsyncJobVO {id:5090, userId: 2, accountId: 2, instanceType: Network, > > >>> instanceId: 204, cmd: > > >>> org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd, > > cmdInfo: > > >>> > > >>> > > > {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"2641","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\"interface > > >>> > > >>> > > > com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}","ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmdEventType":"NETWORK.DELETE","_":"1553004396247"}, > > >>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, > > >>> result: null, initMsid: 264216221068220, completeMsid: null, > > lastUpdated: > > >>> null, lastPolled: null, created: null} > > >>> 2019-03-19 14:06:36,351 WARN [o.a.c.e.o.NetworkOrchestrator] > > >>> (API-Job-Executor-1:ctx-1c0fd4dc job-5090 ctx-134954fa) > > (logid:c734edfc) > > >>> Can't delete the network, not all user vms are expunged. Vm > > >>> VM[User|i-2-11-VM] is in Stopped state > > >>> 2019-03-19 14:06:36,356 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > > >>> (API-Job-Executor-1:ctx-1c0fd4dc job-5090) (logid:c734edfc) Complete > > async > > >>> job-5090, jobStatus: FAILED, resultCode: 530, result: > > >>> > > >>> > > > org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed > > >>> to delete network"} > > >>> ========================================================== > > >>> > > >>> > > >>> I deleted a router, expecting it to be recreated on deleting network. > > But > > >>> I am unable to delete network because of above error > > >>> > > >>> On Tue, Mar 19, 2019 at 3:58 PM Jevgeni Zolotarjov < > > >>> j.zolotar...@gmail.com> > > >>> wrote: > > >>> > > >>>> I've managed to make libvirtd running > > >>>> Now cloudstack console shows both hosts - running > > >>>> > > >>>> But now as I have removed network, VMs are unable to start. > > >>>> > > >>>> How can I recreate the network now? > > >>>> > > >>>> On Tue, Mar 19, 2019 at 3:14 PM Ivan Kudryavtsev > > >>>> <kudryavtsev...@bw-sw.com> > > >>>> wrote: > > >>>> > > >>>>> Jevgeniy, it may be a documentation bug. Take s look: > > >>>>> https://github.com/apache/cloudstack-documentation/pull/27/files > > >>>>> > > >>>>> вт, 19 мар. 2019 г., 9:09 Jevgeni Zolotarjov < > j.zolotar...@gmail.com > > >: > > >>>>> > > >>>>>> That's it - libvirtd failed to start on second host. > > >>>>>> Tried restarting, but it does not start. > > >>>>>> > > >>>>>> > > >>>>>>>> Do you have some NUMA constraints or anything which requires > > >>>>> particular > > >>>>>> RAM configuration? > > >>>>>> No > > >>>>>> > > >>>>>> libvirtd.service - Virtualization daemon > > >>>>>> Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; > > >>>>>> enabled; vendor preset: enabled) > > >>>>>> Active: failed (Result: start-limit) since Tue 2019-03-19 > > >>>>>> 13:03:07 > > >>>>> GMT; > > >>>>>> 12s ago > > >>>>>> Docs: man:libvirtd(8) > > >>>>>> https://libvirt.org > > >>>>>> Process: 892 ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS > > >>>>>> (code=exited, > > >>>>>> status=1/FAILURE) > > >>>>>> Main PID: 892 (code=exited, status=1/FAILURE) > > >>>>>> Tasks: 19 (limit: 32768) > > >>>>>> CGroup: /system.slice/libvirtd.service > > >>>>>> ├─11338 /usr/sbin/libvirtd -d -l > > >>>>>> ├─11909 /usr/sbin/dnsmasq > > >>>>>> --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro > > >>>>>> --dhcp-script=/usr/libexec/libvirt_leaseshelper > > >>>>>> └─11910 /usr/sbin/dnsmasq > > >>>>>> --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro > > >>>>>> --dhcp-script=/usr/libexec/libvirt_leaseshelper > > >>>>>> > > >>>>>> Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Failed to > > >>>>>> start Virtualization daemon. > > >>>>>> Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Unit > > >>>>>> libvirtd.service entered failed state. > > >>>>>> Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: > > >>>>> libvirtd.service > > >>>>>> failed. > > >>>>>> Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: > > >>>>> libvirtd.service > > >>>>>> holdoff time over, scheduling restart. > > >>>>>> Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Stopped > > >>>>>> Virtualization daemon. > > >>>>>> Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: start > > >>>>>> request repeated too quickly for libvirtd.service Mar 19 13:03:07 > > >>>>>> mtl1-apphst04.mt.pbt.com.mt systemd[1]: Failed to start > > >>>>>> Virtualization daemon. > > >>>>>> Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: Unit > > >>>>>> libvirtd.service entered failed state. > > >>>>>> Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: > > >>>>> libvirtd.service > > >>>>>> failed. > > >>>>>> > > >>>>>> > > >>>>>> On Tue, Mar 19, 2019 at 3:04 PM Paul Angus > > >>>>>> <paul.an...@shapeblue.com> > > >>>>>> wrote: > > >>>>>> > > >>>>>>> Can you check that the cloudstack agent is running on the host > > >>>>>>> and the agent logs (usual logs directory) Also worth checking > > >>>>>>> that libvirt has started ok. Do you have some > > >>>>> NUMA > > >>>>>>> constraints or anything which requires particular RAM > > >>> configuration? > > >>>>>>> > > >>>>>>> paul.an...@shapeblue.com > > >>>>>>> www.shapeblue.com > > >>>>>>> Amadeus House, Floral Street, London WC2E 9DPUK @shapeblue > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> -----Original Message----- > > >>>>>>> From: Jevgeni Zolotarjov <j.zolotar...@gmail.com> > > >>>>>>> Sent: 19 March 2019 14:49 > > >>>>>>> To: users@cloudstack.apache.org > > >>>>>>> Subject: Re: Disaster after maintenance > > >>>>>>> > > >>>>>>> Can you try migrating a VM to the server that you changed the RAM > > >>>>> amount? > > >>>>>>> > > >>>>>>> Also: > > >>>>>>> What is the hypervisor version? > > >>>>>>> KVM > > >>>>>>> QEMU Version : 2.0.0 > > >>>>>>> Release : 1.el7.6 > > >>>>>>> > > >>>>>>> > > >>>>>>> Host status in ACS? > > >>>>>>> 1st server: Unsecure > > >>>>>>> 2nd server: Disconnected > > >>>>>>> > > >>>>>>> Did you try to force a VM to start/deploy in this server where > you > > >>>>>> changed > > >>>>>>> the RAM? > > >>>>>>> Host status became disconnected. I don't know how to make it > > >>>>> "connected" > > >>>>>>> again > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> On Tue, Mar 19, 2019 at 2:42 PM Rafael Weingärtner < > > >>>>>>> rafaelweingart...@gmail.com> wrote: > > >>>>>>> > > >>>>>>>> Can you try migrating a VM to the server that you changed the > RAM > > >>>>>> amount? > > >>>>>>>> > > >>>>>>>> Also: > > >>>>>>>> What is the hypervisor version? > > >>>>>>>> Host status in ACS? > > >>>>>>>> Did you try to force a VM to start/deploy in this server where > > >>> you > > >>>>>>>> changed the RAM? > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> On Tue, Mar 19, 2019 at 9:39 AM Jevgeni Zolotarjov > > >>>>>>>> <j.zolotar...@gmail.com > > >>>>>>>>> > > >>>>>>>> wrote: > > >>>>>>>> > > >>>>>>>>> We have Cloudstack 4.11.2 setup running fine for few months > > >>> (>4) > > >>>>> The > > >>>>>>>>> setup is very simple: 2 hosts We decided to do a maintenance to > > >>>>>>>>> increase RAM on both servers > > >>>>>>>>> > > >>>>>>>>> For this we put first server to maintenance. All VMS moved to > > >>>>> second > > >>>>>>>>> host after a while. > > >>>>>>>>> > > >>>>>>>>> Then first server was shutdown, RAM increased, server turned > > >>> ON. > > >>>>>>>>> Now nothing starts on first server. > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> Tried to delete network, but this fails as well > > >>>>>>>>> > > >>>>>>>>> Please help ! > > >>>>>>>>> > > >>>>>>>>> Here is extract from log: > > >>>>>>>>> ====================================== > > >>>>>>>>> 2019-03-19 12:27:53,064 DEBUG > > >>>>> [o.a.c.s.SecondaryStorageManagerImpl] > > >>>>>>>>> (secstorage-1:ctx-16d6c797) (logid:7e3160ce) Zone 1 is ready to > > >>>>>>>>> launch secondary storage VM > > >>>>>>>>> 2019-03-19 12:27:53,125 DEBUG [c.c.c.ConsoleProxyManagerImpl] > > >>>>>>>>> (consoleproxy-1:ctx-cbd034b9) (logid:0a8c8bf4) Zone 1 is ready > > >>> to > > >>>>>>>>> launch console proxy > > >>>>>>>>> 2019-03-19 12:27:53,181 DEBUG [c.c.a.ApiServlet] > > >>>>>>>>> (qtp510113906-285:ctx-6c5e11c3) (logid:cd8e30be) ===START=== > > >>>>>>>> 192.168.5.140 > > >>>>>>>>> -- GET > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>> > > >>>>> > > command=deleteNetwork&id=4ba834ed-48f3-468f-b667-9bb2d2c258f1&response > > >>>>>>>> =json&_=1552998473154 > > >>>>>>>>> 2019-03-19 12:27:53,186 DEBUG [c.c.a.ApiServer] > > >>>>>>>>> (qtp510113906-285:ctx-6c5e11c3 ctx-0cc34dc6) (logid:cd8e30be) > > >>>>> CIDRs > > >>>>>>>>> from which account > > >>>>>>>>> 'Acct[15863393-8e8d-11e7-8f52-f04da2002bbe-admin]' is > > >>>>>>>> allowed > > >>>>>>>>> to perform API calls: 0.0.0.0/0,::/0 > > >>>>>>>>> 2019-03-19 12:27:53,208 INFO [o.a.c.f.j.i.AsyncJobMonitor] > > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:f6751fa7) Add > > >>>>>>>>> job-5081 into job monitoring > > >>>>>>>>> 2019-03-19 12:27:53,209 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > > >>>>>>>>> (qtp510113906-285:ctx-6c5e11c3 ctx-0cc34dc6) (logid:cd8e30be) > > >>>>> submit > > >>>>>>>> async > > >>>>>>>>> job-5081, details: AsyncJobVO {id:5081, userId: 2, accountId: > > >>> 2, > > >>>>>>>>> instanceType: Network, instanceId: 204, cmd: > > >>>>>>>>> > > >>> org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd, > > >>>>>>> cmdInfo: > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>> > > >>>>> > > {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId > > >>>>>>>> > > >>>>> > > ":"2615","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\" > > >>>>>>>> interface > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>> > > >>>>> > > com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}" > > >>>>>>>> > > >>>>> > > ,"ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmd > > >>>>>>>> EventType":"NETWORK.DELETE","_":"1552998473154"}, > > >>>>>>>>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, > > >>> resultCode: > > >>>>> 0, > > >>>>>>>>> result: null, initMsid: 264216221068220, completeMsid: null, > > >>>>>>> lastUpdated: > > >>>>>>>>> null, lastPolled: null, created: null} > > >>>>>>>>> 2019-03-19 12:27:53,211 DEBUG [c.c.a.ApiServlet] > > >>>>>>>>> (qtp510113906-285:ctx-6c5e11c3 ctx-0cc34dc6) (logid:cd8e30be) > > >>>>>>>>> ===END=== > > >>>>>>>>> 192.168.5.140 -- GET > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>> > > >>>>> > > command=deleteNetwork&id=4ba834ed-48f3-468f-b667-9bb2d2c258f1&response > > >>>>>>>> =json&_=1552998473154 > > >>>>>>>>> 2019-03-19 12:27:53,212 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6) > > >>>>>>>>> Executing AsyncJobVO {id:5081, userId: 2, accountId: 2, > > >>>>>>>>> instanceType: Network, > > >>>>>>>>> instanceId: 204, cmd: > > >>>>>>>>> > > >>> org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd, > > >>>>>>> cmdInfo: > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>> > > >>>>> > > {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId > > >>>>>>>> > > >>>>> > > ":"2615","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\" > > >>>>>>>> interface > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>> > > >>>>> > > com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}" > > >>>>>>>> > > >>>>> > > ,"ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmd > > >>>>>>>> EventType":"NETWORK.DELETE","_":"1552998473154"}, > > >>>>>>>>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, > > >>> resultCode: > > >>>>> 0, > > >>>>>>>>> result: null, initMsid: 264216221068220, completeMsid: null, > > >>>>>>> lastUpdated: > > >>>>>>>>> null, lastPolled: null, created: null} > > >>>>>>>>> 2019-03-19 12:27:53,257 WARN [o.a.c.e.o.NetworkOrchestrator] > > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081 ctx-d5de7979) > > >>>>>>>>> (logid:16897ea6) Can't delete the network, not all user vms are > > >>>>>>>>> expunged. Vm VM[User|i-2-11-VM] is in Stopped state > > >>>>>>>>> 2019-03-19 12:27:53,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6) > > >>>>> Complete > > >>>>>>>> async > > >>>>>>>>> job-5081, jobStatus: FAILED, resultCode: 530, result: > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>> > > >>>>> > > org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList": > > >>>>>>>> [],"errorcode":530,"errortext":"Failed > > >>>>>>>>> to delete network"} > > >>>>>>>>> 2019-03-19 12:27:53,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6) > > >>>>> Publish > > >>>>>>>>> async > > >>>>>>>>> job-5081 complete on message bus > > >>>>>>>>> 2019-03-19 12:27:53,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6) > > >>> Wake > > >>>>> up > > >>>>>>>>> jobs related to job-5081 > > >>>>>>>>> 2019-03-19 12:27:53,264 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6) > > >>> Update > > >>>>>>>>> db status for job-5081 > > >>>>>>>>> 2019-03-19 12:27:53,265 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6) > > >>> Wake > > >>>>> up > > >>>>>>>>> jobs joined with job-5081 and disjoin all subjobs created from > > >>>>> job- > > >>>>>>>>> 5081 > > >>>>>>>>> 2019-03-19 12:27:53,267 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6) > > >>> Done > > >>>>>>>> executing > > >>>>>>>>> org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd > > >>>>> for > > >>>>>>>>> job-5081 > > >>>>>>>>> 2019-03-19 12:27:53,267 INFO [o.a.c.f.j.i.AsyncJobMonitor] > > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081) (logid:16897ea6) > > >>> Remove > > >>>>>>>> job-5081 > > >>>>>>>>> from job monitoring > > >>>>>>>>> 2019-03-19 12:27:56,230 DEBUG [c.c.a.ApiServlet] > > >>>>>>>>> (qtp510113906-28:ctx-e6c5bc85) (logid:7fe68f75) ===START=== > > >>>>>>>> 192.168.5.140 > > >>>>>>>>> -- GET > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>> > > >>>>> > > command=queryAsyncJobResult&jobId=16897ea6-27c3-45b9-a0df-ab217bb5393c > > >>>>>>>> &response=json&_=1552998476202 > > >>>>>>>>> 2019-03-19 12:27:56,238 DEBUG [c.c.a.ApiServer] > > >>>>>>>>> (qtp510113906-28:ctx-e6c5bc85 ctx-da1f4cbd) (logid:7fe68f75) > > >>> CIDRs > > >>>>>>>>> from which account > > >>>>>>>>> 'Acct[15863393-8e8d-11e7-8f52-f04da2002bbe-admin]' is > > >>>>>>>> allowed > > >>>>>>>>> to perform API calls: 0.0.0.0/0,::/0 > > >>>>>>>>> 2019-03-19 12:27:56,260 DEBUG [c.c.a.ApiServlet] > > >>>>>>>>> (qtp510113906-28:ctx-e6c5bc85 ctx-da1f4cbd) (logid:7fe68f75) > > >>>>>>>>> ===END=== > > >>>>>>>>> 192.168.5.140 -- GET > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>> > > >>>>> > > command=queryAsyncJobResult&jobId=16897ea6-27c3-45b9-a0df-ab217bb5393c > > >>>>>>>> &response=json&_=1552998476202 > > >>>>>>>>> 2019-03-19 12:28:00,946 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] > > >>>>>>>>> (AsyncJobMgr-Heartbeat-1:ctx-9b43d1fd) (logid:a605267a) Begin > > >>>>>>>>> cleanup expired async-jobs > > >>>>>>>>> 2019-03-19 12:28:00,951 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] > > >>>>>>>>> (AsyncJobMgr-Heartbeat-1:ctx-9b43d1fd) (logid:a605267a) End > > >>>>> cleanup > > >>>>>>>> expired > > >>>>>>>>> async-jobs > > >>>>>>>>> 2019-03-19 12:28:01,142 DEBUG > > >>>>>>>> [c.c.n.r.VirtualNetworkApplianceManagerImpl] > > >>>>>>>>> (RouterStatusMonitor-1:ctx-ad6bbe7e) (logid:04e4c72b) Found 0 > > >>>>>>>>> routers to update status. > > >>>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> -- > > >>>>>>>> Rafael Weingärtner > > >>>>>>>> > > >>>>>>> > > >>>>>> > > >>>>> > > >>>> > > >>> > > >> > > > > >
