>>>Network offering needs to be change to non-redundant How do I do that?
On Tue, Mar 19, 2019 at 11:47 PM Sergey Levitskiy <serg...@hotmail.com> wrote: > Network offering needs to be change to non-redundant. Most likely old bug > is resurfaced. > https://issues.apache.org/jira/browse/CLOUDSTACK-9024 > > > > On 3/19/19, 2:19 PM, "Jevgeni Zolotarjov" <j.zolotar...@gmail.com> wrote: > > Hello > > I did exactly like you suggested. > > After UPDATE on db, I can see the router in cloudstack console. > But attempt to restart network fails. > I get error: > Resource [DataCenter:1] is unreachable: Can't find all necessary > running > routers! > > I rechecked agents on both servers. They look running OK. > > On Tue, Mar 19, 2019 at 10:00 PM Andrija Panic < > andrija.pa...@gmail.com> > wrote: > > > Ok, so: > > > > 1.BACKUP YOUR DB - in case there is issues, you will want to restore > > 2. I tried to reproduce your case on 4.11.2 (though didn't do any > > maintenance etc.) - and could not - I have artificially marked > existing VR > > as removed in DB - and tried to restart network and it worked just > fine. > > > > Let's try to "undelete" the VR for that network - though I can't be > sure if > > this will work or not. > > > > Find VR based on the network UUID: > > SELECT * FROM vm_instance WHERE id IN (SELECT instance_id from > nics > > where network_id IN (SELECT id FROM networks WHERE > > UUID="65ca9a05-ff96-4563-ab9c-ffb610dc8b73")); > > Obviously, replace this UUID with your network UUID. > > SQL above will return all VMs (user VMs, VRs, etc, delete or not...) > that > > have NIC inside the network with that UUID - look for the r-XX-VM > being a > > router. > > > > Now that you have the name of the VR (i.e. r-4-VM) - you want to set > 2 > > fields to some other value than what it might be already, as below: > > > > Need to set field "state" to Stopped: > > UPDATE cloud.vm_instance SET state='Stopped' WHERE > name="r-4-VM"; > > (what state is set currently ???) > > > > Need to set "removed" to NULL value: > > UPDATE cloud.vm_instance SET removed = NULL WHERE > NAME="r-4-VM"; (is > > there a removed date already set ???) > > > > Obviously make sure that this VR is not running on any hypervisor > and if > > running, virsh destroy it... > > > > AFTER above has been done (VR is considered to exist but stopped, > from > > CloudStack point of view) - try to RESTART the network (don't bother > > deleteing it, since there are VMs in that network) > > > > I'm not 100% positive this will fix your issue, but doesn't hurt to > try > > > > If above doesn't work - I would still take a look into agents and if > they > > are still connected - optionally, restart agents on both hosts once > more > > and confirm they are connected and up. > > > > Let us know how it goes (and please backup DB once more before any > actions > > !) > > > > Andrija > > > > > > On Tue, 19 Mar 2019 at 20:41, Jevgeni Zolotarjov < > j.zolotar...@gmail.com> > > wrote: > > > > > Yes. Just a single network. > > > > > > On Tue, 19 Mar 2019, 21:39 Andrija Panic, <andrija.pa...@gmail.com > > > > wrote: > > > > > > > Just one more clarification - this is Isolate single network (not > > Shared > > > > Network, not VPC) ? > > > > > > > > > > > > > > > > On Tue, Mar 19, 2019, 19:36 Jevgeni Zolotarjov < > j.zolotar...@gmail.com > > > > > > > wrote: > > > > > > > > > Name defaultGuestNetwork > > > > > ID 4ba834ed-48f3-468f-b667-9bb2d2c258f1 > > > > > Zone PBT zone 1 > > > > > Description defaultGuestNetwork > > > > > Type Shared > > > > > State Setup > > > > > VPC ID N/A > > > > > Persistent No > > > > > > > > > > On Tue, Mar 19, 2019 at 8:29 PM Andrija Panic < > > andrija.pa...@gmail.com > > > > > > > > > wrote: > > > > > > > > > > > Share the network id and name as seen from GUI... > > > > > > > > > > > > On Tue, Mar 19, 2019, 19:27 Jevgeni Zolotarjov < > > > j.zolotar...@gmail.com > > > > > > > > > > > wrote: > > > > > > > > > > > > > >>> 1. Confirm please rolling restart is set to false > please - > > > double > > > > > > check > > > > > > > Double checked - It is set to false > > > > > > > > > > > > > > >>>>2. If so - do you know the name of VR which you > deleted ? Is > > it > > > > > last > > > > > > > one > > > > > > > >>> ever created - if so we can find it easily... > > > > > > > I dont know the name > > > > > > > Is there a way to fetch it from DB? > > > > > > > > > > > > > > On Tue, Mar 19, 2019 at 7:58 PM Andrija Panic < > > > > andrija.pa...@gmail.com > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Right...still complaining on missing running routers. > > > > > > > > > > > > > > > > 1. Confirm please rolling restart is set to false please > - > > double > > > > > check > > > > > > > > 2. If so - do you know the name of VR which you deleted > ? Is it > > > > last > > > > > > one > > > > > > > > ever created - if so we can find it easily... > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 19, 2019, 18:40 Jevgeni Zolotarjov < > > > > > j.zolotar...@gmail.com > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Here is management server log > > > > > > > > > > > > > > https://drive.google.com/open?id=1H2jI0roeiWxtzReB8qV6QxDkNpaki99A > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 19, 2019 at 7:29 PM Andrija Panic < > > > > > > andrija.pa...@gmail.com > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > I would also try to "undelete" VR in DB, but let's > keep > > this > > > as > > > > > > last > > > > > > > > > step. > > > > > > > > > > > > > > > > > > > > On Tue, Mar 19, 2019, 18:24 Andrija Panic < > > > > > andrija.pa...@gmail.com > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Disclaimer: from mobile device. > > > > > > > > > > > > > > > > > > > > > > I don't see any special failure in agent log, > except some > > > > long > > > > > > > > running > > > > > > > > > > > migration, timeout for graceful VM shutdown etc > (and > > agent > > > > > > restart) > > > > > > > > > > > > > > > > > > > > > > You did not send mgmt log after last network > restart > > > failure > > > > ? > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 19, 2019, 17:29 Jevgeni Zolotarjov < > > > > > > > > j.zolotar...@gmail.com > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > >> Under > > > > > > > > > > >> Infrastructure / Hosts: > > > > > > > > > > >> both hosts are enabled and green (Unsecure though) > > > > > > > > > > >> > > > > > > > > > > >> agent.log - > > > > > > > > > > >> > > > > > > > > > https://drive.google.com/open?id=1rATxHKqgNKo2kD23BtlrZy_9gFXC-Bq- > > > > > > > > > > >> > > > > > > > > > > >> I set network.rolling.restart to false now. > Restarted > > > > > management > > > > > > > > > server > > > > > > > > > > - > > > > > > > > > > >> same problem - cannot restart not delete network > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> On Tue, Mar 19, 2019 at 5:48 PM Boris Stoyanov < > > > > > > > > > > >> boris.stoya...@shapeblue.com> > > > > > > > > > > >> wrote: > > > > > > > > > > >> > > > > > > > > > > >> > Hi, you shouldn’t be worried about data as long > as you > > > > use a > > > > > > > > > separate > > > > > > > > > > >> > shared storage. > > > > > > > > > > >> > > > > > > > > > > > >> > What does the cloudstack console say about your > host? > > Is > > > > it > > > > > > up? > > > > > > > If > > > > > > > > > > it’s > > > > > > > > > > >> up > > > > > > > > > > >> > then you should be able to deploy a VM on it. > > > > > > > > > > >> > > > > > > > > > > > >> > If not, you’ll need to investigate the reason > > > > > cloudstack-agent > > > > > > > is > > > > > > > > > not > > > > > > > > > > >> > connected, is the agent running? If not just > start it > > > and > > > > > see > > > > > > if > > > > > > > > it > > > > > > > > > > >> > connects. > > > > > > > > > > >> > > > > > > > > > > > >> > If it’s running but still not connected please > share > > the > > > > > > > > agent.log. > > > > > > > > > > >> > > > > > > > > > > > >> > If nothing helps at all you can remove the host > and > > add > > > it > > > > > > > again, > > > > > > > > > but > > > > > > > > > > >> > let’s try to troubleshoot first. > > > > > > > > > > >> > > > > > > > > > > > >> > Bobby. > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > boris.stoya...@shapeblue.com > > > > > > > > > > >> > www.shapeblue.com > > > > > > > > > > >> > Amadeus House, Floral Street, London WC2E 9DPUK > > > > > > > > > > >> > @shapeblue > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > On 19 Mar 2019, at 17:29, Jevgeni Zolotarjov < > > > > > > > > > > j.zolotar...@gmail.com> > > > > > > > > > > >> > wrote: > > > > > > > > > > >> > > > > > > > > > > > > >> > > Guys, please help with it. > > > > > > > > > > >> > > What can be done here? > > > > > > > > > > >> > > There is too much valuable data. > > > > > > > > > > >> > > > > > > > > > > > > >> > > On Tue, Mar 19, 2019 at 4:21 PM Jevgeni > Zolotarjov < > > > > > > > > > > >> > j.zolotar...@gmail.com> > > > > > > > > > > >> > > wrote: > > > > > > > > > > >> > > > > > > > > > > > > >> > >> Tried that just now and got error: > > > > > > > > > > >> > >> Resource [DataCenter:1] is unreachable: > Can't find > > > all > > > > > > > > necessary > > > > > > > > > > >> running > > > > > > > > > > >> > >> routers! > > > > > > > > > > >> > >> > > > > > > > > > > >> > >> In the log I see: > > > > > > > > > > >> > >> ============================================= > > > > > > > > > > >> > >> > > > > > > > > > > >> > >> 2019-03-19 14:20:39,644 DEBUG > > > > > > [o.a.c.e.o.NetworkOrchestrator] > > > > > > > > > > >> > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093 > > > ctx-9be30648) > > > > > > > > > > >> (logid:265a6099) > > > > > > > > > > >> > >> Restarting network 204... > > > > > > > > > > >> > >> 2019-03-19 14:20:39,645 DEBUG > > > > > > [o.a.c.e.o.NetworkOrchestrator] > > > > > > > > > > >> > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093 > > > ctx-9be30648) > > > > > > > > > > >> (logid:265a6099) > > > > > > > > > > >> > >> Performing rolling restart of routers of > network > > > > > > > > > Ntwk[204|Guest|6] > > > > > > > > > > >> > >> 2019-03-19 14:20:39,649 DEBUG > > > > > > [o.a.c.e.o.NetworkOrchestrator] > > > > > > > > > > >> > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093 > > > ctx-9be30648) > > > > > > > > > > >> (logid:265a6099) > > > > > > > > > > >> > >> Asking VirtualRouter to implemenet > > Ntwk[204|Guest|6] > > > > > > > > > > >> > >> 2019-03-19 14:20:39,658 ERROR > > > > > [c.c.a.ApiAsyncJobDispatcher] > > > > > > > > > > >> > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093) > > > > > (logid:265a6099) > > > > > > > > > > >> Unexpected > > > > > > > > > > >> > >> exception while executing > > > > > > > > > > >> > >> > > > > > > > > > > org.apache.cloudstack.api.command.user.network.RestartNetworkCmd > > > > > > > > > > >> > >> > com.cloud.exception.ResourceUnavailableException: > > > > > Resource > > > > > > > > > > >> > [DataCenter:1] > > > > > > > > > > >> > >> is unreachable: Can't find all necessary > running > > > > routers! > > > > > > > > > > >> > >> at > > > > > > > > > > >> > >> > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:243) > > > > > > > > > > >> > >> at > > > > > > > > > > >> > >> > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElements(NetworkOrchestrator.java:1203) > > > > > > > > > > >> > >> at > > > > > > > > > > >> > >> > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.rollingRestartRouters(NetworkOrchestrator.java:2948) > > > > > > > > > > >> > >> at > > > > > > > > > > >> > >> > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:2853) > > > > > > > > > > >> > >> at > > > > > > > > > > >> > >> > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1883) > > > > > > > > > > >> > >> at > > > > > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native > > > > > > > > > > Method) > > > > > > > > > > >> > >> at > > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(Unknown > > > > > > > > > > Source) > > > > > > > > > > >> > >> at > > > > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown > > > > > > > > > > >> > Source) > > > > > > > > > > >> > >> at > java.lang.reflect.Method.invoke(Unknown > > > > Source) > > > > > > > > > > >> > >> at > > > > > > > > > > >> > >> > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338) > > > > > > > > > > >> > >> at > > > > > > > > > > >> > >> > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197) > > > > > > > > > > >> > >> at > > > > > > > > > > >> > >> > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) > > > > > > > > > > >> > >> at > > > > > > > > > > >> > >> > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107) > > > > > > > > > > >> > >> at > > > > > > > > > > >> > >> > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) > > > > > > > > > > >> > >> at > > > > > > > > > > >> > >> > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51) > > > > > > > > > > >> > >> at > > > > > > > > > > >> > >> > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) > > > > > > > > > > >> > >> at > > > > > > > > > > >> > >> > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) > > > > > > > > > > >> > >> at > > > > > > > > > > >> > >> > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) > > > > > > > > > > >> > >> at > > > > > > > > > > >> > >> > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) > > > > > > > > > > >> > >> at > > > > com.sun.proxy.$Proxy229.restartNetwork(Unknown > > > > > > > > Source) > > > > > > > > > > >> > >> at > > > > > > > > > > >> > >> > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:99) > > > > > > > > > > >> > >> at > > > > > > > > > > >> > > > com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150) > > > > > > > > > > >> > >> at > > > > > > > > > > >> > >> > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108) > > > > > > > > > > >> > >> at > > > > > > > > > > >> > >> > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581) > > > > > > > > > > >> > >> at > > > > > > > > > > >> > >> > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > > > > > > > > > > >> > >> at > > > > > > > > > > >> > >> > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > > > > > > > > > > >> > >> at > > > > > > > > > > >> > >> > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > > > > > > > > > > >> > >> at > > > > > > > > > > >> > >> > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > > > > > > > > > > >> > >> at > > > > > > > > > > >> > >> > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > > > > > > > > > > >> > >> at > > > > > > > > > > >> > >> > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529) > > > > > > > > > > >> > >> at > > > > > > > > > > > java.util.concurrent.Executors$RunnableAdapter.call(Unknown > > > > > > > > > > >> > >> Source) > > > > > > > > > > >> > >> at > > java.util.concurrent.FutureTask.run(Unknown > > > > > > Source) > > > > > > > > > > >> > >> at > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown > > > > > > > > > > >> > >> Source) > > > > > > > > > > >> > >> at > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown > > > > > > > > > > >> > >> Source) > > > > > > > > > > >> > >> at java.lang.Thread.run(Unknown > Source) > > > > > > > > > > >> > >> 2019-03-19 14:20:39,659 DEBUG > > > > > > > [o.a.c.f.j.i.AsyncJobManagerImpl] > > > > > > > > > > >> > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093) > > > > > (logid:265a6099) > > > > > > > > > > Complete > > > > > > > > > > >> > async > > > > > > > > > > >> > >> job-5093, jobStatus: FAILED, resultCode: 530, > > result: > > > > > > > > > > >> > >> > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Resource > > > > > > > > > > >> > >> [DataCenter:1] is un > > > > > > > > > > >> > >> > > > > > > > > > > >> > >> On Tue, Mar 19, 2019 at 4:19 PM Andrija > Panic < > > > > > > > > > > >> > andrija.pa...@shapeblue.com> > > > > > > > > > > >> > >> wrote: > > > > > > > > > > >> > >> > > > > > > > > > > >> > >>> > > > > > > > > > > >> > >>> Your network can't be deleted due to "Can't > delete > > > the > > > > > > > > network, > > > > > > > > > > not > > > > > > > > > > >> all > > > > > > > > > > >> > >>> user vms are expunged. Vm > > > > > > > > > > >> > >>> VM[User|i-2-11-VM] is in Stopped state" - > which is > > > > fine. > > > > > > > > > > >> > >>> > > > > > > > > > > >> > >>> You should be able to just start the user > VM - but > > > if > > > > > you > > > > > > > have > > > > > > > > > > >> actually > > > > > > > > > > >> > >>> delete the VR itself, then just do Network > restart > > > > with > > > > > > > > > "cleanup" > > > > > > > > > > >> and > > > > > > > > > > >> > it > > > > > > > > > > >> > >>> will recreate a new VR, after which you > should be > > > able > > > > > to > > > > > > > > start > > > > > > > > > > the > > > > > > > > > > >> VM. > > > > > > > > > > >> > >>> > > > > > > > > > > >> > >>> Andrija > > > > > > > > > > >> > >>> > > > > > > > > > > >> > >>> andrija.pa...@shapeblue.com > > > > > > > > > > >> > >>> www.shapeblue.com > > > > > > > > > > >> > >>> Amadeus House, Floral Street, London WC2E > 9DPUK > > > > > > > > > > >> > >>> @shapeblue > > > > > > > > > > >> > >>> > > > > > > > > > > >> > >>> > > > > > > > > > > >> > >>> > > > > > > > > > > >> > >>> > > > > > > > > > > >> > >>> -----Original Message----- > > > > > > > > > > >> > >>> From: Jevgeni Zolotarjov < > j.zolotar...@gmail.com> > > > > > > > > > > >> > >>> Sent: 19 March 2019 15:10 > > > > > > > > > > >> > >>> To: users@cloudstack.apache.org > > > > > > > > > > >> > >>> Subject: Re: Disaster after maintenance > > > > > > > > > > >> > >>> > > > > > > > > > > >> > >>> I mean I cannot delete network: In the > management > > > > server > > > > > > > log I > > > > > > > > > see > > > > > > > > > > >> > >>> > > > > > > > > > > >> > >>> > > > > > ========================================================== > > > > > > > > > > >> > >>> 019-03-19 14:06:36,316 DEBUG > > > > > > > [o.a.c.f.j.i.AsyncJobManagerImpl] > > > > > > > > > > >> > >>> (API-Job-Executor-1:ctx-1c0fd4dc job-5090) > > > > > > (logid:c734edfc) > > > > > > > > > > >> Executing > > > > > > > > > > >> > >>> AsyncJobVO {id:5090, userId: 2, accountId: > 2, > > > > > > instanceType: > > > > > > > > > > Network, > > > > > > > > > > >> > >>> instanceId: 204, cmd: > > > > > > > > > > >> > >>> > > > > > > > > > > org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd, > > > > > > > > > > >> > cmdInfo: > > > > > > > > > > >> > >>> > > > > > > > > > > >> > >>> > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"2641","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\"interface > > > > > > > > > > >> > >>> > > > > > > > > > > >> > >>> > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}","ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmdEventType":"NETWORK.DELETE","_":"1553004396247"}, > > > > > > > > > > >> > >>> cmdVersion: 0, status: IN_PROGRESS, > processStatus: > > > 0, > > > > > > > > > resultCode: > > > > > > > > > > 0, > > > > > > > > > > >> > >>> result: null, initMsid: 264216221068220, > > > completeMsid: > > > > > > null, > > > > > > > > > > >> > lastUpdated: > > > > > > > > > > >> > >>> null, lastPolled: null, created: null} > > > > > > > > > > >> > >>> 2019-03-19 14:06:36,351 WARN > > > > > > > [o.a.c.e.o.NetworkOrchestrator] > > > > > > > > > > >> > >>> (API-Job-Executor-1:ctx-1c0fd4dc job-5090 > > > > ctx-134954fa) > > > > > > > > > > >> > (logid:c734edfc) > > > > > > > > > > >> > >>> Can't delete the network, not all user vms > are > > > > expunged. > > > > > > Vm > > > > > > > > > > >> > >>> VM[User|i-2-11-VM] is in Stopped state > > > > > > > > > > >> > >>> 2019-03-19 14:06:36,356 DEBUG > > > > > > > > [o.a.c.f.j.i.AsyncJobManagerImpl] > > > > > > > > > > >> > >>> (API-Job-Executor-1:ctx-1c0fd4dc job-5090) > > > > > > (logid:c734edfc) > > > > > > > > > > Complete > > > > > > > > > > >> > async > > > > > > > > > > >> > >>> job-5090, jobStatus: FAILED, resultCode: > 530, > > > result: > > > > > > > > > > >> > >>> > > > > > > > > > > >> > >>> > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed > > > > > > > > > > >> > >>> to delete network"} > > > > > > > > > > >> > >>> > > > > > ========================================================== > > > > > > > > > > >> > >>> > > > > > > > > > > >> > >>> > > > > > > > > > > >> > >>> I deleted a router, expecting it to be > recreated > > on > > > > > > deleting > > > > > > > > > > >> network. > > > > > > > > > > >> > But > > > > > > > > > > >> > >>> I am unable to delete network because of > above > > error > > > > > > > > > > >> > >>> > > > > > > > > > > >> > >>> On Tue, Mar 19, 2019 at 3:58 PM Jevgeni > > Zolotarjov < > > > > > > > > > > >> > >>> j.zolotar...@gmail.com> > > > > > > > > > > >> > >>> wrote: > > > > > > > > > > >> > >>> > > > > > > > > > > >> > >>>> I've managed to make libvirtd running > > > > > > > > > > >> > >>>> Now cloudstack console shows both hosts - > running > > > > > > > > > > >> > >>>> > > > > > > > > > > >> > >>>> But now as I have removed network, VMs are > unable > > > to > > > > > > start. > > > > > > > > > > >> > >>>> > > > > > > > > > > >> > >>>> How can I recreate the network now? > > > > > > > > > > >> > >>>> > > > > > > > > > > >> > >>>> On Tue, Mar 19, 2019 at 3:14 PM Ivan > Kudryavtsev > > > > > > > > > > >> > >>>> <kudryavtsev...@bw-sw.com> > > > > > > > > > > >> > >>>> wrote: > > > > > > > > > > >> > >>>> > > > > > > > > > > >> > >>>>> Jevgeniy, it may be a documentation bug. > Take s > > > > look: > > > > > > > > > > >> > >>>>> > > > > > > > > > > > > > > https://github.com/apache/cloudstack-documentation/pull/27/files > > > > > > > > > > >> > >>>>> > > > > > > > > > > >> > >>>>> вт, 19 мар. 2019 г., 9:09 Jevgeni > Zolotarjov < > > > > > > > > > > >> j.zolotar...@gmail.com > > > > > > > > > > >> > >: > > > > > > > > > > >> > >>>>> > > > > > > > > > > >> > >>>>>> That's it - libvirtd failed to start on > second > > > > host. > > > > > > > > > > >> > >>>>>> Tried restarting, but it does not start. > > > > > > > > > > >> > >>>>>> > > > > > > > > > > >> > >>>>>> > > > > > > > > > > >> > >>>>>>>> Do you have some NUMA constraints or > anything > > > > which > > > > > > > > > requires > > > > > > > > > > >> > >>>>> particular > > > > > > > > > > >> > >>>>>> RAM configuration? > > > > > > > > > > >> > >>>>>> No > > > > > > > > > > >> > >>>>>> > > > > > > > > > > >> > >>>>>> libvirtd.service - Virtualization daemon > > > > > > > > > > >> > >>>>>> Loaded: loaded > > > > > > > (/usr/lib/systemd/system/libvirtd.service; > > > > > > > > > > >> > >>>>>> enabled; vendor preset: enabled) > > > > > > > > > > >> > >>>>>> Active: failed (Result: start-limit) > since > > Tue > > > > > > > 2019-03-19 > > > > > > > > > > >> > >>>>>> 13:03:07 > > > > > > > > > > >> > >>>>> GMT; > > > > > > > > > > >> > >>>>>> 12s ago > > > > > > > > > > >> > >>>>>> Docs: man:libvirtd(8) > > > > > > > > > > >> > >>>>>> https://libvirt.org > > > > > > > > > > >> > >>>>>> Process: 892 > ExecStart=/usr/sbin/libvirtd > > > > > > $LIBVIRTD_ARGS > > > > > > > > > > >> > >>>>>> (code=exited, > > > > > > > > > > >> > >>>>>> status=1/FAILURE) > > > > > > > > > > >> > >>>>>> Main PID: 892 (code=exited, > status=1/FAILURE) > > > > > > > > > > >> > >>>>>> Tasks: 19 (limit: 32768) > > > > > > > > > > >> > >>>>>> CGroup: /system.slice/libvirtd.service > > > > > > > > > > >> > >>>>>> ├─11338 /usr/sbin/libvirtd -d > -l > > > > > > > > > > >> > >>>>>> ├─11909 /usr/sbin/dnsmasq > > > > > > > > > > >> > >>>>>> > > --conf-file=/var/lib/libvirt/dnsmasq/default.conf > > > > > > > > > > --leasefile-ro > > > > > > > > > > >> > >>>>>> > --dhcp-script=/usr/libexec/libvirt_leaseshelper > > > > > > > > > > >> > >>>>>> └─11910 /usr/sbin/dnsmasq > > > > > > > > > > >> > >>>>>> > > --conf-file=/var/lib/libvirt/dnsmasq/default.conf > > > > > > > > > > --leasefile-ro > > > > > > > > > > >> > >>>>>> > --dhcp-script=/usr/libexec/libvirt_leaseshelper > > > > > > > > > > >> > >>>>>> > > > > > > > > > > >> > >>>>>> Mar 19 13:03:07 > mtl1-apphst04.mt.pbt.com.mt > > > > > > systemd[1]: > > > > > > > > > Failed > > > > > > > > > > >> to > > > > > > > > > > >> > >>>>>> start Virtualization daemon. > > > > > > > > > > >> > >>>>>> Mar 19 13:03:07 > mtl1-apphst04.mt.pbt.com.mt > > > > > > systemd[1]: > > > > > > > > Unit > > > > > > > > > > >> > >>>>>> libvirtd.service entered failed state. > > > > > > > > > > >> > >>>>>> Mar 19 13:03:07 > mtl1-apphst04.mt.pbt.com.mt > > > > > > systemd[1]: > > > > > > > > > > >> > >>>>> libvirtd.service > > > > > > > > > > >> > >>>>>> failed. > > > > > > > > > > >> > >>>>>> Mar 19 13:03:07 > mtl1-apphst04.mt.pbt.com.mt > > > > > > systemd[1]: > > > > > > > > > > >> > >>>>> libvirtd.service > > > > > > > > > > >> > >>>>>> holdoff time over, scheduling restart. > > > > > > > > > > >> > >>>>>> Mar 19 13:03:07 > mtl1-apphst04.mt.pbt.com.mt > > > > > > systemd[1]: > > > > > > > > > > Stopped > > > > > > > > > > >> > >>>>>> Virtualization daemon. > > > > > > > > > > >> > >>>>>> Mar 19 13:03:07 > mtl1-apphst04.mt.pbt.com.mt > > > > > > systemd[1]: > > > > > > > > > start > > > > > > > > > > >> > >>>>>> request repeated too quickly for > > libvirtd.service > > > > Mar > > > > > > 19 > > > > > > > > > > 13:03:07 > > > > > > > > > > >> > >>>>>> mtl1-apphst04.mt.pbt.com.mt systemd[1]: > Failed > > > to > > > > > > start > > > > > > > > > > >> > >>>>>> Virtualization daemon. > > > > > > > > > > >> > >>>>>> Mar 19 13:03:07 > mtl1-apphst04.mt.pbt.com.mt > > > > > > systemd[1]: > > > > > > > > Unit > > > > > > > > > > >> > >>>>>> libvirtd.service entered failed state. > > > > > > > > > > >> > >>>>>> Mar 19 13:03:07 > mtl1-apphst04.mt.pbt.com.mt > > > > > > systemd[1]: > > > > > > > > > > >> > >>>>> libvirtd.service > > > > > > > > > > >> > >>>>>> failed. > > > > > > > > > > >> > >>>>>> > > > > > > > > > > >> > >>>>>> > > > > > > > > > > >> > >>>>>> On Tue, Mar 19, 2019 at 3:04 PM Paul > Angus > > > > > > > > > > >> > >>>>>> <paul.an...@shapeblue.com> > > > > > > > > > > >> > >>>>>> wrote: > > > > > > > > > > >> > >>>>>> > > > > > > > > > > >> > >>>>>>> Can you check that the cloudstack agent > is > > > running > > > > > on > > > > > > > the > > > > > > > > > host > > > > > > > > > > >> > >>>>>>> and the agent logs (usual logs > directory) Also > > > > worth > > > > > > > > > checking > > > > > > > > > > >> > >>>>>>> that libvirt has started ok. Do you > have some > > > > > > > > > > >> > >>>>> NUMA > > > > > > > > > > >> > >>>>>>> constraints or anything which requires > > > particular > > > > > RAM > > > > > > > > > > >> > >>> configuration? > > > > > > > > > > >> > >>>>>>> > > > > > > > > > > >> > >>>>>>> paul.an...@shapeblue.com > > > > > > > > > > >> > >>>>>>> www.shapeblue.com > > > > > > > > > > >> > >>>>>>> Amadeus House, Floral Street, London > WC2E > > 9DPUK > > > > > > > > @shapeblue > > > > > > > > > > >> > >>>>>>> > > > > > > > > > > >> > >>>>>>> > > > > > > > > > > >> > >>>>>>> > > > > > > > > > > >> > >>>>>>> > > > > > > > > > > >> > >>>>>>> -----Original Message----- > > > > > > > > > > >> > >>>>>>> From: Jevgeni Zolotarjov < > > > j.zolotar...@gmail.com> > > > > > > > > > > >> > >>>>>>> Sent: 19 March 2019 14:49 > > > > > > > > > > >> > >>>>>>> To: users@cloudstack.apache.org > > > > > > > > > > >> > >>>>>>> Subject: Re: Disaster after maintenance > > > > > > > > > > >> > >>>>>>> > > > > > > > > > > >> > >>>>>>> Can you try migrating a VM to the > server that > > > you > > > > > > > changed > > > > > > > > > the > > > > > > > > > > >> RAM > > > > > > > > > > >> > >>>>> amount? > > > > > > > > > > >> > >>>>>>> > > > > > > > > > > >> > >>>>>>> Also: > > > > > > > > > > >> > >>>>>>> What is the hypervisor version? > > > > > > > > > > >> > >>>>>>> KVM > > > > > > > > > > >> > >>>>>>> QEMU Version : 2.0.0 > > > > > > > > > > >> > >>>>>>> Release : 1.el7.6 > > > > > > > > > > >> > >>>>>>> > > > > > > > > > > >> > >>>>>>> > > > > > > > > > > >> > >>>>>>> Host status in ACS? > > > > > > > > > > >> > >>>>>>> 1st server: Unsecure > > > > > > > > > > >> > >>>>>>> 2nd server: Disconnected > > > > > > > > > > >> > >>>>>>> > > > > > > > > > > >> > >>>>>>> Did you try to force a VM to > start/deploy in > > > this > > > > > > server > > > > > > > > > where > > > > > > > > > > >> you > > > > > > > > > > >> > >>>>>> changed > > > > > > > > > > >> > >>>>>>> the RAM? > > > > > > > > > > >> > >>>>>>> Host status became disconnected. I > don't know > > > how > > > > to > > > > > > > make > > > > > > > > it > > > > > > > > > > >> > >>>>> "connected" > > > > > > > > > > >> > >>>>>>> again > > > > > > > > > > >> > >>>>>>> > > > > > > > > > > >> > >>>>>>> > > > > > > > > > > >> > >>>>>>> > > > > > > > > > > >> > >>>>>>> On Tue, Mar 19, 2019 at 2:42 PM Rafael > > > > Weingärtner < > > > > > > > > > > >> > >>>>>>> rafaelweingart...@gmail.com> wrote: > > > > > > > > > > >> > >>>>>>> > > > > > > > > > > >> > >>>>>>>> Can you try migrating a VM to the > server that > > > you > > > > > > > changed > > > > > > > > > the > > > > > > > > > > >> RAM > > > > > > > > > > >> > >>>>>> amount? > > > > > > > > > > >> > >>>>>>>> > > > > > > > > > > >> > >>>>>>>> Also: > > > > > > > > > > >> > >>>>>>>> What is the hypervisor version? > > > > > > > > > > >> > >>>>>>>> Host status in ACS? > > > > > > > > > > >> > >>>>>>>> Did you try to force a VM to > start/deploy in > > > this > > > > > > > server > > > > > > > > > > where > > > > > > > > > > >> > >>> you > > > > > > > > > > >> > >>>>>>>> changed the RAM? > > > > > > > > > > >> > >>>>>>>> > > > > > > > > > > >> > >>>>>>>> > > > > > > > > > > >> > >>>>>>>> On Tue, Mar 19, 2019 at 9:39 AM Jevgeni > > > > Zolotarjov > > > > > > > > > > >> > >>>>>>>> <j.zolotar...@gmail.com > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > >> > >>>>>>>> wrote: > > > > > > > > > > >> > >>>>>>>> > > > > > > > > > > >> > >>>>>>>>> We have Cloudstack 4.11.2 setup > running fine > > > for > > > > > few > > > > > > > > > months > > > > > > > > > > >> > >>> (>4) > > > > > > > > > > >> > >>>>> The > > > > > > > > > > >> > >>>>>>>>> setup is very simple: 2 hosts We > decided to > > > do a > > > > > > > > > maintenance > > > > > > > > > > >> to > > > > > > > > > > >> > >>>>>>>>> increase RAM on both servers > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > >> > >>>>>>>>> For this we put first server to > maintenance. > > > All > > > > > VMS > > > > > > > > moved > > > > > > > > > > to > > > > > > > > > > >> > >>>>> second > > > > > > > > > > >> > >>>>>>>>> host after a while. > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > >> > >>>>>>>>> Then first server was shutdown, RAM > > increased, > > > > > > server > > > > > > > > > turned > > > > > > > > > > >> > >>> ON. > > > > > > > > > > >> > >>>>>>>>> Now nothing starts on first server. > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > >> > >>>>>>>>> Tried to delete network, but this > fails as > > > well > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > >> > >>>>>>>>> Please help ! > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > >> > >>>>>>>>> Here is extract from log: > > > > > > > > > > >> > >>>>>>>>> ====================================== > > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,064 DEBUG > > > > > > > > > > >> > >>>>> [o.a.c.s.SecondaryStorageManagerImpl] > > > > > > > > > > >> > >>>>>>>>> (secstorage-1:ctx-16d6c797) > (logid:7e3160ce) > > > > Zone > > > > > 1 > > > > > > is > > > > > > > > > ready > > > > > > > > > > >> to > > > > > > > > > > >> > >>>>>>>>> launch secondary storage VM > > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,125 DEBUG > > > > > > > > > > [c.c.c.ConsoleProxyManagerImpl] > > > > > > > > > > >> > >>>>>>>>> (consoleproxy-1:ctx-cbd034b9) > > (logid:0a8c8bf4) > > > > > Zone > > > > > > 1 > > > > > > > is > > > > > > > > > > ready > > > > > > > > > > >> > >>> to > > > > > > > > > > >> > >>>>>>>>> launch console proxy > > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,181 DEBUG > > > [c.c.a.ApiServlet] > > > > > > > > > > >> > >>>>>>>>> (qtp510113906-285:ctx-6c5e11c3) > > > (logid:cd8e30be) > > > > > > > > > ===START=== > > > > > > > > > > >> > >>>>>>>> 192.168.5.140 > > > > > > > > > > >> > >>>>>>>>> -- GET > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > >> > >>>>>>>> > > > > > > > > > > >> > >>>>> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > command=deleteNetwork&id=4ba834ed-48f3-468f-b667-9bb2d2c258f1&response > > > > > > > > > > >> > >>>>>>>> =json&_=1552998473154 > > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,186 DEBUG > > > [c.c.a.ApiServer] > > > > > > > > > > >> > >>>>>>>>> (qtp510113906-285:ctx-6c5e11c3 > ctx-0cc34dc6) > > > > > > > > > > (logid:cd8e30be) > > > > > > > > > > >> > >>>>> CIDRs > > > > > > > > > > >> > >>>>>>>>> from which account > > > > > > > > > > >> > >>>>>>>>> > > > > 'Acct[15863393-8e8d-11e7-8f52-f04da2002bbe-admin]' > > > > > > is > > > > > > > > > > >> > >>>>>>>> allowed > > > > > > > > > > >> > >>>>>>>>> to perform API calls: 0.0.0.0/0,::/0 > > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,208 INFO > > > > > > > > > [o.a.c.f.j.i.AsyncJobMonitor] > > > > > > > > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 > job-5081) > > > > > > > > > (logid:f6751fa7) > > > > > > > > > > >> Add > > > > > > > > > > >> > >>>>>>>>> job-5081 into job monitoring > > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,209 DEBUG > > > > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl] > > > > > > > > > > >> > >>>>>>>>> (qtp510113906-285:ctx-6c5e11c3 > ctx-0cc34dc6) > > > > > > > > > > (logid:cd8e30be) > > > > > > > > > > >> > >>>>> submit > > > > > > > > > > >> > >>>>>>>> async > > > > > > > > > > >> > >>>>>>>>> job-5081, details: AsyncJobVO > {id:5081, > > > userId: > > > > 2, > > > > > > > > > > accountId: > > > > > > > > > > >> > >>> 2, > > > > > > > > > > >> > >>>>>>>>> instanceType: Network, instanceId: > 204, cmd: > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > >> > >>> > > > > > > > > > > org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd, > > > > > > > > > > >> > >>>>>>> cmdInfo: > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > >> > >>>>>>>> > > > > > > > > > > >> > >>>>> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId > > > > > > > > > > >> > >>>>>>>> > > > > > > > > > > >> > >>>>> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > ":"2615","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\" > > > > > > > > > > >> > >>>>>>>> interface > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > >> > >>>>>>>> > > > > > > > > > > >> > >>>>> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}" > > > > > > > > > > >> > >>>>>>>> > > > > > > > > > > >> > >>>>> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > ,"ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmd > > > > > > > > > > >> > >>>>>>>> > > > EventType":"NETWORK.DELETE","_":"1552998473154"}, > > > > > > > > > > >> > >>>>>>>>> cmdVersion: 0, status: IN_PROGRESS, > > > > processStatus: > > > > > > 0, > > > > > > > > > > >> > >>> resultCode: > > > > > > > > > > >> > >>>>> 0, > > > > > > > > > > >> > >>>>>>>>> result: null, initMsid: > 264216221068220, > > > > > > completeMsid: > > > > > > > > > null, > > > > > > > > > > >> > >>>>>>> lastUpdated: > > > > > > > > > > >> > >>>>>>>>> null, lastPolled: null, created: null} > > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,211 DEBUG > > > [c.c.a.ApiServlet] > > > > > > > > > > >> > >>>>>>>>> (qtp510113906-285:ctx-6c5e11c3 > ctx-0cc34dc6) > > > > > > > > > > (logid:cd8e30be) > > > > > > > > > > >> > >>>>>>>>> ===END=== > > > > > > > > > > >> > >>>>>>>>> 192.168.5.140 -- GET > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > >> > >>>>>>>> > > > > > > > > > > >> > >>>>> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > command=deleteNetwork&id=4ba834ed-48f3-468f-b667-9bb2d2c258f1&response > > > > > > > > > > >> > >>>>>>>> =json&_=1552998473154 > > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,212 DEBUG > > > > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl] > > > > > > > > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 > job-5081) > > > > > > > > > (logid:16897ea6) > > > > > > > > > > >> > >>>>>>>>> Executing AsyncJobVO {id:5081, > userId: 2, > > > > > accountId: > > > > > > > 2, > > > > > > > > > > >> > >>>>>>>>> instanceType: Network, > > > > > > > > > > >> > >>>>>>>>> instanceId: 204, cmd: > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > >> > >>> > > > > > > > > > > org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd, > > > > > > > > > > >> > >>>>>>> cmdInfo: > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > >> > >>>>>>>> > > > > > > > > > > >> > >>>>> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId > > > > > > > > > > >> > >>>>>>>> > > > > > > > > > > >> > >>>>> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > ":"2615","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\" > > > > > > > > > > >> > >>>>>>>> interface > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > >> > >>>>>>>> > > > > > > > > > > >> > >>>>> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}" > > > > > > > > > > >> > >>>>>>>> > > > > > > > > > > >> > >>>>> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > ,"ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmd > > > > > > > > > > >> > >>>>>>>> > > > EventType":"NETWORK.DELETE","_":"1552998473154"}, > > > > > > > > > > >> > >>>>>>>>> cmdVersion: 0, status: IN_PROGRESS, > > > > processStatus: > > > > > > 0, > > > > > > > > > > >> > >>> resultCode: > > > > > > > > > > >> > >>>>> 0, > > > > > > > > > > >> > >>>>>>>>> result: null, initMsid: > 264216221068220, > > > > > > completeMsid: > > > > > > > > > null, > > > > > > > > > > >> > >>>>>>> lastUpdated: > > > > > > > > > > >> > >>>>>>>>> null, lastPolled: null, created: null} > > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,257 WARN > > > > > > > > > > [o.a.c.e.o.NetworkOrchestrator] > > > > > > > > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 > job-5081 > > > > > > > ctx-d5de7979) > > > > > > > > > > >> > >>>>>>>>> (logid:16897ea6) Can't delete the > network, > > not > > > > all > > > > > > > user > > > > > > > > > vms > > > > > > > > > > >> are > > > > > > > > > > >> > >>>>>>>>> expunged. Vm VM[User|i-2-11-VM] is in > > Stopped > > > > > state > > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,263 DEBUG > > > > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl] > > > > > > > > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 > job-5081) > > > > > > > > > (logid:16897ea6) > > > > > > > > > > >> > >>>>> Complete > > > > > > > > > > >> > >>>>>>>> async > > > > > > > > > > >> > >>>>>>>>> job-5081, jobStatus: FAILED, > resultCode: > > 530, > > > > > > result: > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > >> > >>>>>>>> > > > > > > > > > > >> > >>>>> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList": > > > > > > > > > > >> > >>>>>>>> [],"errorcode":530,"errortext":"Failed > > > > > > > > > > >> > >>>>>>>>> to delete network"} > > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,264 DEBUG > > > > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl] > > > > > > > > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 > job-5081) > > > > > > > > > (logid:16897ea6) > > > > > > > > > > >> > >>>>> Publish > > > > > > > > > > >> > >>>>>>>>> async > > > > > > > > > > >> > >>>>>>>>> job-5081 complete on message bus > > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,264 DEBUG > > > > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl] > > > > > > > > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 > job-5081) > > > > > > > > > (logid:16897ea6) > > > > > > > > > > >> > >>> Wake > > > > > > > > > > >> > >>>>> up > > > > > > > > > > >> > >>>>>>>>> jobs related to job-5081 > > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,264 DEBUG > > > > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl] > > > > > > > > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 > job-5081) > > > > > > > > > (logid:16897ea6) > > > > > > > > > > >> > >>> Update > > > > > > > > > > >> > >>>>>>>>> db status for job-5081 > > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,265 DEBUG > > > > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl] > > > > > > > > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 > job-5081) > > > > > > > > > (logid:16897ea6) > > > > > > > > > > >> > >>> Wake > > > > > > > > > > >> > >>>>> up > > > > > > > > > > >> > >>>>>>>>> jobs joined with job-5081 and disjoin > all > > > > subjobs > > > > > > > > created > > > > > > > > > > from > > > > > > > > > > >> > >>>>> job- > > > > > > > > > > >> > >>>>>>>>> 5081 > > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,267 DEBUG > > > > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl] > > > > > > > > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 > job-5081) > > > > > > > > > (logid:16897ea6) > > > > > > > > > > >> > >>> Done > > > > > > > > > > >> > >>>>>>>> executing > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > >> > > > > > org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd > > > > > > > > > > >> > >>>>> for > > > > > > > > > > >> > >>>>>>>>> job-5081 > > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,267 INFO > > > > > > > > > [o.a.c.f.j.i.AsyncJobMonitor] > > > > > > > > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 > job-5081) > > > > > > > > > (logid:16897ea6) > > > > > > > > > > >> > >>> Remove > > > > > > > > > > >> > >>>>>>>> job-5081 > > > > > > > > > > >> > >>>>>>>>> from job monitoring > > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:56,230 DEBUG > > > [c.c.a.ApiServlet] > > > > > > > > > > >> > >>>>>>>>> (qtp510113906-28:ctx-e6c5bc85) > > > (logid:7fe68f75) > > > > > > > > > ===START=== > > > > > > > > > > >> > >>>>>>>> 192.168.5.140 > > > > > > > > > > >> > >>>>>>>>> -- GET > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > >> > >>>>>>>> > > > > > > > > > > >> > >>>>> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > command=queryAsyncJobResult&jobId=16897ea6-27c3-45b9-a0df-ab217bb5393c > > > > > > > > > > >> > >>>>>>>> &response=json&_=1552998476202 > > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:56,238 DEBUG > > > [c.c.a.ApiServer] > > > > > > > > > > >> > >>>>>>>>> (qtp510113906-28:ctx-e6c5bc85 > ctx-da1f4cbd) > > > > > > > > > (logid:7fe68f75) > > > > > > > > > > >> > >>> CIDRs > > > > > > > > > > >> > >>>>>>>>> from which account > > > > > > > > > > >> > >>>>>>>>> > > > > 'Acct[15863393-8e8d-11e7-8f52-f04da2002bbe-admin]' > > > > > > is > > > > > > > > > > >> > >>>>>>>> allowed > > > > > > > > > > >> > >>>>>>>>> to perform API calls: 0.0.0.0/0,::/0 > > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:56,260 DEBUG > > > [c.c.a.ApiServlet] > > > > > > > > > > >> > >>>>>>>>> (qtp510113906-28:ctx-e6c5bc85 > ctx-da1f4cbd) > > > > > > > > > (logid:7fe68f75) > > > > > > > > > > >> > >>>>>>>>> ===END=== > > > > > > > > > > >> > >>>>>>>>> 192.168.5.140 -- GET > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > >> > >>>>>>>> > > > > > > > > > > >> > >>>>> > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > command=queryAsyncJobResult&jobId=16897ea6-27c3-45b9-a0df-ab217bb5393c > > > > > > > > > > >> > >>>>>>>> &response=json&_=1552998476202 > > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:28:00,946 INFO > > > > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl] > > > > > > > > > > >> > >>>>>>>>> (AsyncJobMgr-Heartbeat-1:ctx-9b43d1fd) > > > > > > > (logid:a605267a) > > > > > > > > > > Begin > > > > > > > > > > >> > >>>>>>>>> cleanup expired async-jobs > > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:28:00,951 INFO > > > > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl] > > > > > > > > > > >> > >>>>>>>>> (AsyncJobMgr-Heartbeat-1:ctx-9b43d1fd) > > > > > > > (logid:a605267a) > > > > > > > > > End > > > > > > > > > > >> > >>>>> cleanup > > > > > > > > > > >> > >>>>>>>> expired > > > > > > > > > > >> > >>>>>>>>> async-jobs > > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:28:01,142 DEBUG > > > > > > > > > > >> > >>>>>>>> > [c.c.n.r.VirtualNetworkApplianceManagerImpl] > > > > > > > > > > >> > >>>>>>>>> (RouterStatusMonitor-1:ctx-ad6bbe7e) > > > > > > (logid:04e4c72b) > > > > > > > > > Found > > > > > > > > > > 0 > > > > > > > > > > >> > >>>>>>>>> routers to update status. > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > >> > >>>>>>>> > > > > > > > > > > >> > >>>>>>>> > > > > > > > > > > >> > >>>>>>>> -- > > > > > > > > > > >> > >>>>>>>> Rafael Weingärtner > > > > > > > > > > >> > >>>>>>>> > > > > > > > > > > >> > >>>>>>> > > > > > > > > > > >> > >>>>>> > > > > > > > > > > >> > >>>>> > > > > > > > > > > >> > >>>> > > > > > > > > > > >> > >>> > > > > > > > > > > >> > >> > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Andrija Panić > > > > >
