Ok, so:
1.BACKUP YOUR DB - in case there is issues, you will want to restore
2. I tried to reproduce your case on 4.11.2 (though didn't do any
maintenance etc.) - and could not - I have artificially marked existing VR
as removed in DB - and tried to restart network and it worked just fine.
Let's try to "undelete" the VR for that network - though I can't be sure if
this will work or not.
Find VR based on the network UUID:
SELECT * FROM vm_instance WHERE id IN (SELECT instance_id from nics
where network_id IN (SELECT id FROM networks WHERE
UUID="65ca9a05-ff96-4563-ab9c-ffb610dc8b73"));
Obviously, replace this UUID with your network UUID.
SQL above will return all VMs (user VMs, VRs, etc, delete or not...) that
have NIC inside the network with that UUID - look for the r-XX-VM being a
router.
Now that you have the name of the VR (i.e. r-4-VM) - you want to set 2
fields to some other value than what it might be already, as below:
Need to set field "state" to Stopped:
UPDATE cloud.vm_instance SET state='Stopped' WHERE name="r-4-VM";
(what state is set currently ???)
Need to set "removed" to NULL value:
UPDATE cloud.vm_instance SET removed = NULL WHERE NAME="r-4-VM"; (is
there a removed date already set ???)
Obviously make sure that this VR is not running on any hypervisor and if
running, virsh destroy it...
AFTER above has been done (VR is considered to exist but stopped, from
CloudStack point of view) - try to RESTART the network (don't bother
deleteing it, since there are VMs in that network)
I'm not 100% positive this will fix your issue, but doesn't hurt to try
If above doesn't work - I would still take a look into agents and if they
are still connected - optionally, restart agents on both hosts once more
and confirm they are connected and up.
Let us know how it goes (and please backup DB once more before any actions
!)
Andrija
On Tue, 19 Mar 2019 at 20:41, Jevgeni Zolotarjov <j.zolotar...@gmail.com>
wrote:
> Yes. Just a single network.
>
> On Tue, 19 Mar 2019, 21:39 Andrija Panic, <andrija.pa...@gmail.com> wrote:
>
> > Just one more clarification - this is Isolate single network (not Shared
> > Network, not VPC) ?
> >
> >
> >
> > On Tue, Mar 19, 2019, 19:36 Jevgeni Zolotarjov <j.zolotar...@gmail.com>
> > wrote:
> >
> > > Name defaultGuestNetwork
> > > ID 4ba834ed-48f3-468f-b667-9bb2d2c258f1
> > > Zone PBT zone 1
> > > Description defaultGuestNetwork
> > > Type Shared
> > > State Setup
> > > VPC ID N/A
> > > Persistent No
> > >
> > > On Tue, Mar 19, 2019 at 8:29 PM Andrija Panic <andrija.pa...@gmail.com
> >
> > > wrote:
> > >
> > > > Share the network id and name as seen from GUI...
> > > >
> > > > On Tue, Mar 19, 2019, 19:27 Jevgeni Zolotarjov <
> j.zolotar...@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > >>> 1. Confirm please rolling restart is set to false please -
> double
> > > > check
> > > > > Double checked - It is set to false
> > > > >
> > > > > >>>>2. If so - do you know the name of VR which you deleted ? Is it
> > > last
> > > > > one
> > > > > >>> ever created - if so we can find it easily...
> > > > > I dont know the name
> > > > > Is there a way to fetch it from DB?
> > > > >
> > > > > On Tue, Mar 19, 2019 at 7:58 PM Andrija Panic <
> > andrija.pa...@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Right...still complaining on missing running routers.
> > > > > >
> > > > > > 1. Confirm please rolling restart is set to false please - double
> > > check
> > > > > > 2. If so - do you know the name of VR which you deleted ? Is it
> > last
> > > > one
> > > > > > ever created - if so we can find it easily...
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Mar 19, 2019, 18:40 Jevgeni Zolotarjov <
> > > j.zolotar...@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Here is management server log
> > > > > > >
> > https://drive.google.com/open?id=1H2jI0roeiWxtzReB8qV6QxDkNpaki99A
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Mar 19, 2019 at 7:29 PM Andrija Panic <
> > > > andrija.pa...@gmail.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I would also try to "undelete" VR in DB, but let's keep this
> as
> > > > last
> > > > > > > step.
> > > > > > > >
> > > > > > > > On Tue, Mar 19, 2019, 18:24 Andrija Panic <
> > > andrija.pa...@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Disclaimer: from mobile device.
> > > > > > > > >
> > > > > > > > > I don't see any special failure in agent log, except some
> > long
> > > > > > running
> > > > > > > > > migration, timeout for graceful VM shutdown etc (and agent
> > > > restart)
> > > > > > > > >
> > > > > > > > > You did not send mgmt log after last network restart
> failure
> > ?
> > > > > > > > >
> > > > > > > > > On Tue, Mar 19, 2019, 17:29 Jevgeni Zolotarjov <
> > > > > > j.zolotar...@gmail.com
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Under
> > > > > > > > >> Infrastructure / Hosts:
> > > > > > > > >> both hosts are enabled and green (Unsecure though)
> > > > > > > > >>
> > > > > > > > >> agent.log -
> > > > > > > > >>
> > > > > https://drive.google.com/open?id=1rATxHKqgNKo2kD23BtlrZy_9gFXC-Bq-
> > > > > > > > >>
> > > > > > > > >> I set network.rolling.restart to false now. Restarted
> > > management
> > > > > > > server
> > > > > > > > -
> > > > > > > > >> same problem - cannot restart not delete network
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> On Tue, Mar 19, 2019 at 5:48 PM Boris Stoyanov <
> > > > > > > > >> boris.stoya...@shapeblue.com>
> > > > > > > > >> wrote:
> > > > > > > > >>
> > > > > > > > >> > Hi, you shouldn’t be worried about data as long as you
> > use a
> > > > > > > separate
> > > > > > > > >> > shared storage.
> > > > > > > > >> >
> > > > > > > > >> > What does the cloudstack console say about your host? Is
> > it
> > > > up?
> > > > > If
> > > > > > > > it’s
> > > > > > > > >> up
> > > > > > > > >> > then you should be able to deploy a VM on it.
> > > > > > > > >> >
> > > > > > > > >> > If not, you’ll need to investigate the reason
> > > cloudstack-agent
> > > > > is
> > > > > > > not
> > > > > > > > >> > connected, is the agent running? If not just start it
> and
> > > see
> > > > if
> > > > > > it
> > > > > > > > >> > connects.
> > > > > > > > >> >
> > > > > > > > >> > If it’s running but still not connected please share the
> > > > > > agent.log.
> > > > > > > > >> >
> > > > > > > > >> > If nothing helps at all you can remove the host and add
> it
> > > > > again,
> > > > > > > but
> > > > > > > > >> > let’s try to troubleshoot first.
> > > > > > > > >> >
> > > > > > > > >> > Bobby.
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> > boris.stoya...@shapeblue.com
> > > > > > > > >> > www.shapeblue.com
> > > > > > > > >> > Amadeus House, Floral Street, London WC2E 9DPUK
> > > > > > > > >> > @shapeblue
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> > > On 19 Mar 2019, at 17:29, Jevgeni Zolotarjov <
> > > > > > > > j.zolotar...@gmail.com>
> > > > > > > > >> > wrote:
> > > > > > > > >> > >
> > > > > > > > >> > > Guys, please help with it.
> > > > > > > > >> > > What can be done here?
> > > > > > > > >> > > There is too much valuable data.
> > > > > > > > >> > >
> > > > > > > > >> > > On Tue, Mar 19, 2019 at 4:21 PM Jevgeni Zolotarjov <
> > > > > > > > >> > j.zolotar...@gmail.com>
> > > > > > > > >> > > wrote:
> > > > > > > > >> > >
> > > > > > > > >> > >> Tried that just now and got error:
> > > > > > > > >> > >> Resource [DataCenter:1] is unreachable: Can't find
> all
> > > > > > necessary
> > > > > > > > >> running
> > > > > > > > >> > >> routers!
> > > > > > > > >> > >>
> > > > > > > > >> > >> In the log I see:
> > > > > > > > >> > >> =============================================
> > > > > > > > >> > >>
> > > > > > > > >> > >> 2019-03-19 14:20:39,644 DEBUG
> > > > [o.a.c.e.o.NetworkOrchestrator]
> > > > > > > > >> > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093
> ctx-9be30648)
> > > > > > > > >> (logid:265a6099)
> > > > > > > > >> > >> Restarting network 204...
> > > > > > > > >> > >> 2019-03-19 14:20:39,645 DEBUG
> > > > [o.a.c.e.o.NetworkOrchestrator]
> > > > > > > > >> > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093
> ctx-9be30648)
> > > > > > > > >> (logid:265a6099)
> > > > > > > > >> > >> Performing rolling restart of routers of network
> > > > > > > Ntwk[204|Guest|6]
> > > > > > > > >> > >> 2019-03-19 14:20:39,649 DEBUG
> > > > [o.a.c.e.o.NetworkOrchestrator]
> > > > > > > > >> > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093
> ctx-9be30648)
> > > > > > > > >> (logid:265a6099)
> > > > > > > > >> > >> Asking VirtualRouter to implemenet Ntwk[204|Guest|6]
> > > > > > > > >> > >> 2019-03-19 14:20:39,658 ERROR
> > > [c.c.a.ApiAsyncJobDispatcher]
> > > > > > > > >> > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093)
> > > (logid:265a6099)
> > > > > > > > >> Unexpected
> > > > > > > > >> > >> exception while executing
> > > > > > > > >> > >>
> > > > > > org.apache.cloudstack.api.command.user.network.RestartNetworkCmd
> > > > > > > > >> > >> com.cloud.exception.ResourceUnavailableException:
> > > Resource
> > > > > > > > >> > [DataCenter:1]
> > > > > > > > >> > >> is unreachable: Can't find all necessary running
> > routers!
> > > > > > > > >> > >> at
> > > > > > > > >> > >>
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:243)
> > > > > > > > >> > >> at
> > > > > > > > >> > >>
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElements(NetworkOrchestrator.java:1203)
> > > > > > > > >> > >> at
> > > > > > > > >> > >>
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.rollingRestartRouters(NetworkOrchestrator.java:2948)
> > > > > > > > >> > >> at
> > > > > > > > >> > >>
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:2853)
> > > > > > > > >> > >> at
> > > > > > > > >> > >>
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1883)
> > > > > > > > >> > >> at
> > > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> > > > > > > > Method)
> > > > > > > > >> > >> at
> > > > sun.reflect.NativeMethodAccessorImpl.invoke(Unknown
> > > > > > > > Source)
> > > > > > > > >> > >> at
> > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
> > > > > > > > >> > Source)
> > > > > > > > >> > >> at java.lang.reflect.Method.invoke(Unknown
> > Source)
> > > > > > > > >> > >> at
> > > > > > > > >> > >>
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
> > > > > > > > >> > >> at
> > > > > > > > >> > >>
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
> > > > > > > > >> > >> at
> > > > > > > > >> > >>
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> > > > > > > > >> > >> at
> > > > > > > > >> > >>
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
> > > > > > > > >> > >> at
> > > > > > > > >> > >>
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> > > > > > > > >> > >> at
> > > > > > > > >> > >>
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
> > > > > > > > >> > >> at
> > > > > > > > >> > >>
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> > > > > > > > >> > >> at
> > > > > > > > >> > >>
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
> > > > > > > > >> > >> at
> > > > > > > > >> > >>
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
> > > > > > > > >> > >> at
> > > > > > > > >> > >>
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
> > > > > > > > >> > >> at
> > com.sun.proxy.$Proxy229.restartNetwork(Unknown
> > > > > > Source)
> > > > > > > > >> > >> at
> > > > > > > > >> > >>
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:99)
> > > > > > > > >> > >> at
> > > > > > > > >>
> com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
> > > > > > > > >> > >> at
> > > > > > > > >> > >>
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
> > > > > > > > >> > >> at
> > > > > > > > >> > >>
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
> > > > > > > > >> > >> at
> > > > > > > > >> > >>
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> > > > > > > > >> > >> at
> > > > > > > > >> > >>
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> > > > > > > > >> > >> at
> > > > > > > > >> > >>
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> > > > > > > > >> > >> at
> > > > > > > > >> > >>
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> > > > > > > > >> > >> at
> > > > > > > > >> > >>
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> > > > > > > > >> > >> at
> > > > > > > > >> > >>
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
> > > > > > > > >> > >> at
> > > > > > > > java.util.concurrent.Executors$RunnableAdapter.call(Unknown
> > > > > > > > >> > >> Source)
> > > > > > > > >> > >> at java.util.concurrent.FutureTask.run(Unknown
> > > > Source)
> > > > > > > > >> > >> at
> > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> > > > > > > > >> > >> Source)
> > > > > > > > >> > >> at
> > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> > > > > > > > >> > >> Source)
> > > > > > > > >> > >> at java.lang.Thread.run(Unknown Source)
> > > > > > > > >> > >> 2019-03-19 14:20:39,659 DEBUG
> > > > > [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > > > >> > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093)
> > > (logid:265a6099)
> > > > > > > > Complete
> > > > > > > > >> > async
> > > > > > > > >> > >> job-5093, jobStatus: FAILED, resultCode: 530, result:
> > > > > > > > >> > >>
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Resource
> > > > > > > > >> > >> [DataCenter:1] is un
> > > > > > > > >> > >>
> > > > > > > > >> > >> On Tue, Mar 19, 2019 at 4:19 PM Andrija Panic <
> > > > > > > > >> > andrija.pa...@shapeblue.com>
> > > > > > > > >> > >> wrote:
> > > > > > > > >> > >>
> > > > > > > > >> > >>>
> > > > > > > > >> > >>> Your network can't be deleted due to "Can't delete
> the
> > > > > > network,
> > > > > > > > not
> > > > > > > > >> all
> > > > > > > > >> > >>> user vms are expunged. Vm
> > > > > > > > >> > >>> VM[User|i-2-11-VM] is in Stopped state" - which is
> > fine.
> > > > > > > > >> > >>>
> > > > > > > > >> > >>> You should be able to just start the user VM - but
> if
> > > you
> > > > > have
> > > > > > > > >> actually
> > > > > > > > >> > >>> delete the VR itself, then just do Network restart
> > with
> > > > > > > "cleanup"
> > > > > > > > >> and
> > > > > > > > >> > it
> > > > > > > > >> > >>> will recreate a new VR, after which you should be
> able
> > > to
> > > > > > start
> > > > > > > > the
> > > > > > > > >> VM.
> > > > > > > > >> > >>>
> > > > > > > > >> > >>> Andrija
> > > > > > > > >> > >>>
> > > > > > > > >> > >>> andrija.pa...@shapeblue.com
> > > > > > > > >> > >>> www.shapeblue.com
> > > > > > > > >> > >>> Amadeus House, Floral Street, London WC2E 9DPUK
> > > > > > > > >> > >>> @shapeblue
> > > > > > > > >> > >>>
> > > > > > > > >> > >>>
> > > > > > > > >> > >>>
> > > > > > > > >> > >>>
> > > > > > > > >> > >>> -----Original Message-----
> > > > > > > > >> > >>> From: Jevgeni Zolotarjov <j.zolotar...@gmail.com>
> > > > > > > > >> > >>> Sent: 19 March 2019 15:10
> > > > > > > > >> > >>> To: users@cloudstack.apache.org
> > > > > > > > >> > >>> Subject: Re: Disaster after maintenance
> > > > > > > > >> > >>>
> > > > > > > > >> > >>> I mean I cannot delete network: In the management
> > server
> > > > > log I
> > > > > > > see
> > > > > > > > >> > >>>
> > > > > > > > >> > >>>
> > > ==========================================================
> > > > > > > > >> > >>> 019-03-19 14:06:36,316 DEBUG
> > > > > [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > > > >> > >>> (API-Job-Executor-1:ctx-1c0fd4dc job-5090)
> > > > (logid:c734edfc)
> > > > > > > > >> Executing
> > > > > > > > >> > >>> AsyncJobVO {id:5090, userId: 2, accountId: 2,
> > > > instanceType:
> > > > > > > > Network,
> > > > > > > > >> > >>> instanceId: 204, cmd:
> > > > > > > > >> > >>>
> > > > > > org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd,
> > > > > > > > >> > cmdInfo:
> > > > > > > > >> > >>>
> > > > > > > > >> > >>>
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"2641","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\"interface
> > > > > > > > >> > >>>
> > > > > > > > >> > >>>
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}","ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmdEventType":"NETWORK.DELETE","_":"1553004396247"},
> > > > > > > > >> > >>> cmdVersion: 0, status: IN_PROGRESS, processStatus:
> 0,
> > > > > > > resultCode:
> > > > > > > > 0,
> > > > > > > > >> > >>> result: null, initMsid: 264216221068220,
> completeMsid:
> > > > null,
> > > > > > > > >> > lastUpdated:
> > > > > > > > >> > >>> null, lastPolled: null, created: null}
> > > > > > > > >> > >>> 2019-03-19 14:06:36,351 WARN
> > > > > [o.a.c.e.o.NetworkOrchestrator]
> > > > > > > > >> > >>> (API-Job-Executor-1:ctx-1c0fd4dc job-5090
> > ctx-134954fa)
> > > > > > > > >> > (logid:c734edfc)
> > > > > > > > >> > >>> Can't delete the network, not all user vms are
> > expunged.
> > > > Vm
> > > > > > > > >> > >>> VM[User|i-2-11-VM] is in Stopped state
> > > > > > > > >> > >>> 2019-03-19 14:06:36,356 DEBUG
> > > > > > [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > > > >> > >>> (API-Job-Executor-1:ctx-1c0fd4dc job-5090)
> > > > (logid:c734edfc)
> > > > > > > > Complete
> > > > > > > > >> > async
> > > > > > > > >> > >>> job-5090, jobStatus: FAILED, resultCode: 530,
> result:
> > > > > > > > >> > >>>
> > > > > > > > >> > >>>
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
> > > > > > > > >> > >>> to delete network"}
> > > > > > > > >> > >>>
> > > ==========================================================
> > > > > > > > >> > >>>
> > > > > > > > >> > >>>
> > > > > > > > >> > >>> I deleted a router, expecting it to be recreated on
> > > > deleting
> > > > > > > > >> network.
> > > > > > > > >> > But
> > > > > > > > >> > >>> I am unable to delete network because of above error
> > > > > > > > >> > >>>
> > > > > > > > >> > >>> On Tue, Mar 19, 2019 at 3:58 PM Jevgeni Zolotarjov <
> > > > > > > > >> > >>> j.zolotar...@gmail.com>
> > > > > > > > >> > >>> wrote:
> > > > > > > > >> > >>>
> > > > > > > > >> > >>>> I've managed to make libvirtd running
> > > > > > > > >> > >>>> Now cloudstack console shows both hosts - running
> > > > > > > > >> > >>>>
> > > > > > > > >> > >>>> But now as I have removed network, VMs are unable
> to
> > > > start.
> > > > > > > > >> > >>>>
> > > > > > > > >> > >>>> How can I recreate the network now?
> > > > > > > > >> > >>>>
> > > > > > > > >> > >>>> On Tue, Mar 19, 2019 at 3:14 PM Ivan Kudryavtsev
> > > > > > > > >> > >>>> <kudryavtsev...@bw-sw.com>
> > > > > > > > >> > >>>> wrote:
> > > > > > > > >> > >>>>
> > > > > > > > >> > >>>>> Jevgeniy, it may be a documentation bug. Take s
> > look:
> > > > > > > > >> > >>>>>
> > > > > > > >
> > https://github.com/apache/cloudstack-documentation/pull/27/files
> > > > > > > > >> > >>>>>
> > > > > > > > >> > >>>>> вт, 19 мар. 2019 г., 9:09 Jevgeni Zolotarjov <
> > > > > > > > >> j.zolotar...@gmail.com
> > > > > > > > >> > >:
> > > > > > > > >> > >>>>>
> > > > > > > > >> > >>>>>> That's it - libvirtd failed to start on second
> > host.
> > > > > > > > >> > >>>>>> Tried restarting, but it does not start.
> > > > > > > > >> > >>>>>>
> > > > > > > > >> > >>>>>>
> > > > > > > > >> > >>>>>>>> Do you have some NUMA constraints or anything
> > which
> > > > > > > requires
> > > > > > > > >> > >>>>> particular
> > > > > > > > >> > >>>>>> RAM configuration?
> > > > > > > > >> > >>>>>> No
> > > > > > > > >> > >>>>>>
> > > > > > > > >> > >>>>>> libvirtd.service - Virtualization daemon
> > > > > > > > >> > >>>>>> Loaded: loaded
> > > > > (/usr/lib/systemd/system/libvirtd.service;
> > > > > > > > >> > >>>>>> enabled; vendor preset: enabled)
> > > > > > > > >> > >>>>>> Active: failed (Result: start-limit) since Tue
> > > > > 2019-03-19
> > > > > > > > >> > >>>>>> 13:03:07
> > > > > > > > >> > >>>>> GMT;
> > > > > > > > >> > >>>>>> 12s ago
> > > > > > > > >> > >>>>>> Docs: man:libvirtd(8)
> > > > > > > > >> > >>>>>> https://libvirt.org
> > > > > > > > >> > >>>>>> Process: 892 ExecStart=/usr/sbin/libvirtd
> > > > $LIBVIRTD_ARGS
> > > > > > > > >> > >>>>>> (code=exited,
> > > > > > > > >> > >>>>>> status=1/FAILURE)
> > > > > > > > >> > >>>>>> Main PID: 892 (code=exited, status=1/FAILURE)
> > > > > > > > >> > >>>>>> Tasks: 19 (limit: 32768)
> > > > > > > > >> > >>>>>> CGroup: /system.slice/libvirtd.service
> > > > > > > > >> > >>>>>> ├─11338 /usr/sbin/libvirtd -d -l
> > > > > > > > >> > >>>>>> ├─11909 /usr/sbin/dnsmasq
> > > > > > > > >> > >>>>>> --conf-file=/var/lib/libvirt/dnsmasq/default.conf
> > > > > > > > --leasefile-ro
> > > > > > > > >> > >>>>>> --dhcp-script=/usr/libexec/libvirt_leaseshelper
> > > > > > > > >> > >>>>>> └─11910 /usr/sbin/dnsmasq
> > > > > > > > >> > >>>>>> --conf-file=/var/lib/libvirt/dnsmasq/default.conf
> > > > > > > > --leasefile-ro
> > > > > > > > >> > >>>>>> --dhcp-script=/usr/libexec/libvirt_leaseshelper
> > > > > > > > >> > >>>>>>
> > > > > > > > >> > >>>>>> Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt
> > > > systemd[1]:
> > > > > > > Failed
> > > > > > > > >> to
> > > > > > > > >> > >>>>>> start Virtualization daemon.
> > > > > > > > >> > >>>>>> Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt
> > > > systemd[1]:
> > > > > > Unit
> > > > > > > > >> > >>>>>> libvirtd.service entered failed state.
> > > > > > > > >> > >>>>>> Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt
> > > > systemd[1]:
> > > > > > > > >> > >>>>> libvirtd.service
> > > > > > > > >> > >>>>>> failed.
> > > > > > > > >> > >>>>>> Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt
> > > > systemd[1]:
> > > > > > > > >> > >>>>> libvirtd.service
> > > > > > > > >> > >>>>>> holdoff time over, scheduling restart.
> > > > > > > > >> > >>>>>> Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt
> > > > systemd[1]:
> > > > > > > > Stopped
> > > > > > > > >> > >>>>>> Virtualization daemon.
> > > > > > > > >> > >>>>>> Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt
> > > > systemd[1]:
> > > > > > > start
> > > > > > > > >> > >>>>>> request repeated too quickly for libvirtd.service
> > Mar
> > > > 19
> > > > > > > > 13:03:07
> > > > > > > > >> > >>>>>> mtl1-apphst04.mt.pbt.com.mt systemd[1]: Failed
> to
> > > > start
> > > > > > > > >> > >>>>>> Virtualization daemon.
> > > > > > > > >> > >>>>>> Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt
> > > > systemd[1]:
> > > > > > Unit
> > > > > > > > >> > >>>>>> libvirtd.service entered failed state.
> > > > > > > > >> > >>>>>> Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt
> > > > systemd[1]:
> > > > > > > > >> > >>>>> libvirtd.service
> > > > > > > > >> > >>>>>> failed.
> > > > > > > > >> > >>>>>>
> > > > > > > > >> > >>>>>>
> > > > > > > > >> > >>>>>> On Tue, Mar 19, 2019 at 3:04 PM Paul Angus
> > > > > > > > >> > >>>>>> <paul.an...@shapeblue.com>
> > > > > > > > >> > >>>>>> wrote:
> > > > > > > > >> > >>>>>>
> > > > > > > > >> > >>>>>>> Can you check that the cloudstack agent is
> running
> > > on
> > > > > the
> > > > > > > host
> > > > > > > > >> > >>>>>>> and the agent logs (usual logs directory) Also
> > worth
> > > > > > > checking
> > > > > > > > >> > >>>>>>> that libvirt has started ok. Do you have some
> > > > > > > > >> > >>>>> NUMA
> > > > > > > > >> > >>>>>>> constraints or anything which requires
> particular
> > > RAM
> > > > > > > > >> > >>> configuration?
> > > > > > > > >> > >>>>>>>
> > > > > > > > >> > >>>>>>> paul.an...@shapeblue.com
> > > > > > > > >> > >>>>>>> www.shapeblue.com
> > > > > > > > >> > >>>>>>> Amadeus House, Floral Street, London WC2E 9DPUK
> > > > > > @shapeblue
> > > > > > > > >> > >>>>>>>
> > > > > > > > >> > >>>>>>>
> > > > > > > > >> > >>>>>>>
> > > > > > > > >> > >>>>>>>
> > > > > > > > >> > >>>>>>> -----Original Message-----
> > > > > > > > >> > >>>>>>> From: Jevgeni Zolotarjov <
> j.zolotar...@gmail.com>
> > > > > > > > >> > >>>>>>> Sent: 19 March 2019 14:49
> > > > > > > > >> > >>>>>>> To: users@cloudstack.apache.org
> > > > > > > > >> > >>>>>>> Subject: Re: Disaster after maintenance
> > > > > > > > >> > >>>>>>>
> > > > > > > > >> > >>>>>>> Can you try migrating a VM to the server that
> you
> > > > > changed
> > > > > > > the
> > > > > > > > >> RAM
> > > > > > > > >> > >>>>> amount?
> > > > > > > > >> > >>>>>>>
> > > > > > > > >> > >>>>>>> Also:
> > > > > > > > >> > >>>>>>> What is the hypervisor version?
> > > > > > > > >> > >>>>>>> KVM
> > > > > > > > >> > >>>>>>> QEMU Version : 2.0.0
> > > > > > > > >> > >>>>>>> Release : 1.el7.6
> > > > > > > > >> > >>>>>>>
> > > > > > > > >> > >>>>>>>
> > > > > > > > >> > >>>>>>> Host status in ACS?
> > > > > > > > >> > >>>>>>> 1st server: Unsecure
> > > > > > > > >> > >>>>>>> 2nd server: Disconnected
> > > > > > > > >> > >>>>>>>
> > > > > > > > >> > >>>>>>> Did you try to force a VM to start/deploy in
> this
> > > > server
> > > > > > > where
> > > > > > > > >> you
> > > > > > > > >> > >>>>>> changed
> > > > > > > > >> > >>>>>>> the RAM?
> > > > > > > > >> > >>>>>>> Host status became disconnected. I don't know
> how
> > to
> > > > > make
> > > > > > it
> > > > > > > > >> > >>>>> "connected"
> > > > > > > > >> > >>>>>>> again
> > > > > > > > >> > >>>>>>>
> > > > > > > > >> > >>>>>>>
> > > > > > > > >> > >>>>>>>
> > > > > > > > >> > >>>>>>> On Tue, Mar 19, 2019 at 2:42 PM Rafael
> > Weingärtner <
> > > > > > > > >> > >>>>>>> rafaelweingart...@gmail.com> wrote:
> > > > > > > > >> > >>>>>>>
> > > > > > > > >> > >>>>>>>> Can you try migrating a VM to the server that
> you
> > > > > changed
> > > > > > > the
> > > > > > > > >> RAM
> > > > > > > > >> > >>>>>> amount?
> > > > > > > > >> > >>>>>>>>
> > > > > > > > >> > >>>>>>>> Also:
> > > > > > > > >> > >>>>>>>> What is the hypervisor version?
> > > > > > > > >> > >>>>>>>> Host status in ACS?
> > > > > > > > >> > >>>>>>>> Did you try to force a VM to start/deploy in
> this
> > > > > server
> > > > > > > > where
> > > > > > > > >> > >>> you
> > > > > > > > >> > >>>>>>>> changed the RAM?
> > > > > > > > >> > >>>>>>>>
> > > > > > > > >> > >>>>>>>>
> > > > > > > > >> > >>>>>>>> On Tue, Mar 19, 2019 at 9:39 AM Jevgeni
> > Zolotarjov
> > > > > > > > >> > >>>>>>>> <j.zolotar...@gmail.com
> > > > > > > > >> > >>>>>>>>>
> > > > > > > > >> > >>>>>>>> wrote:
> > > > > > > > >> > >>>>>>>>
> > > > > > > > >> > >>>>>>>>> We have Cloudstack 4.11.2 setup running fine
> for
> > > few
> > > > > > > months
> > > > > > > > >> > >>> (>4)
> > > > > > > > >> > >>>>> The
> > > > > > > > >> > >>>>>>>>> setup is very simple: 2 hosts We decided to
> do a
> > > > > > > maintenance
> > > > > > > > >> to
> > > > > > > > >> > >>>>>>>>> increase RAM on both servers
> > > > > > > > >> > >>>>>>>>>
> > > > > > > > >> > >>>>>>>>> For this we put first server to maintenance.
> All
> > > VMS
> > > > > > moved
> > > > > > > > to
> > > > > > > > >> > >>>>> second
> > > > > > > > >> > >>>>>>>>> host after a while.
> > > > > > > > >> > >>>>>>>>>
> > > > > > > > >> > >>>>>>>>> Then first server was shutdown, RAM increased,
> > > > server
> > > > > > > turned
> > > > > > > > >> > >>> ON.
> > > > > > > > >> > >>>>>>>>> Now nothing starts on first server.
> > > > > > > > >> > >>>>>>>>>
> > > > > > > > >> > >>>>>>>>>
> > > > > > > > >> > >>>>>>>>> Tried to delete network, but this fails as
> well
> > > > > > > > >> > >>>>>>>>>
> > > > > > > > >> > >>>>>>>>> Please help !
> > > > > > > > >> > >>>>>>>>>
> > > > > > > > >> > >>>>>>>>> Here is extract from log:
> > > > > > > > >> > >>>>>>>>> ======================================
> > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,064 DEBUG
> > > > > > > > >> > >>>>> [o.a.c.s.SecondaryStorageManagerImpl]
> > > > > > > > >> > >>>>>>>>> (secstorage-1:ctx-16d6c797) (logid:7e3160ce)
> > Zone
> > > 1
> > > > is
> > > > > > > ready
> > > > > > > > >> to
> > > > > > > > >> > >>>>>>>>> launch secondary storage VM
> > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,125 DEBUG
> > > > > > > > [c.c.c.ConsoleProxyManagerImpl]
> > > > > > > > >> > >>>>>>>>> (consoleproxy-1:ctx-cbd034b9) (logid:0a8c8bf4)
> > > Zone
> > > > 1
> > > > > is
> > > > > > > > ready
> > > > > > > > >> > >>> to
> > > > > > > > >> > >>>>>>>>> launch console proxy
> > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,181 DEBUG
> [c.c.a.ApiServlet]
> > > > > > > > >> > >>>>>>>>> (qtp510113906-285:ctx-6c5e11c3)
> (logid:cd8e30be)
> > > > > > > ===START===
> > > > > > > > >> > >>>>>>>> 192.168.5.140
> > > > > > > > >> > >>>>>>>>> -- GET
> > > > > > > > >> > >>>>>>>>>
> > > > > > > > >> > >>>>>>>>>
> > > > > > > > >> > >>>>>>>>
> > > > > > > > >> > >>>>>
> > > > > > > > >> >
> > > > > > >
> > > >
> command=deleteNetwork&id=4ba834ed-48f3-468f-b667-9bb2d2c258f1&response
> > > > > > > > >> > >>>>>>>> =json&_=1552998473154
> > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,186 DEBUG
> [c.c.a.ApiServer]
> > > > > > > > >> > >>>>>>>>> (qtp510113906-285:ctx-6c5e11c3 ctx-0cc34dc6)
> > > > > > > > (logid:cd8e30be)
> > > > > > > > >> > >>>>> CIDRs
> > > > > > > > >> > >>>>>>>>> from which account
> > > > > > > > >> > >>>>>>>>>
> > 'Acct[15863393-8e8d-11e7-8f52-f04da2002bbe-admin]'
> > > > is
> > > > > > > > >> > >>>>>>>> allowed
> > > > > > > > >> > >>>>>>>>> to perform API calls: 0.0.0.0/0,::/0
> > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,208 INFO
> > > > > > > [o.a.c.f.j.i.AsyncJobMonitor]
> > > > > > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081)
> > > > > > > (logid:f6751fa7)
> > > > > > > > >> Add
> > > > > > > > >> > >>>>>>>>> job-5081 into job monitoring
> > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,209 DEBUG
> > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > > > >> > >>>>>>>>> (qtp510113906-285:ctx-6c5e11c3 ctx-0cc34dc6)
> > > > > > > > (logid:cd8e30be)
> > > > > > > > >> > >>>>> submit
> > > > > > > > >> > >>>>>>>> async
> > > > > > > > >> > >>>>>>>>> job-5081, details: AsyncJobVO {id:5081,
> userId:
> > 2,
> > > > > > > > accountId:
> > > > > > > > >> > >>> 2,
> > > > > > > > >> > >>>>>>>>> instanceType: Network, instanceId: 204, cmd:
> > > > > > > > >> > >>>>>>>>>
> > > > > > > > >> > >>>
> > > > > > org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd,
> > > > > > > > >> > >>>>>>> cmdInfo:
> > > > > > > > >> > >>>>>>>>>
> > > > > > > > >> > >>>>>>>>>
> > > > > > > > >> > >>>>>>>>
> > > > > > > > >> > >>>>>
> > > > > > > > >> >
> > > > > > >
> > > >
> {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId
> > > > > > > > >> > >>>>>>>>
> > > > > > > > >> > >>>>>
> > > > > > > > >> >
> > > > > > >
> > > >
> ":"2615","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\"
> > > > > > > > >> > >>>>>>>> interface
> > > > > > > > >> > >>>>>>>>>
> > > > > > > > >> > >>>>>>>>>
> > > > > > > > >> > >>>>>>>>
> > > > > > > > >> > >>>>>
> > > > > > > > >> >
> > > > > > >
> > > >
> com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}"
> > > > > > > > >> > >>>>>>>>
> > > > > > > > >> > >>>>>
> > > > > > > > >> >
> > > > > > >
> > > >
> ,"ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmd
> > > > > > > > >> > >>>>>>>>
> EventType":"NETWORK.DELETE","_":"1552998473154"},
> > > > > > > > >> > >>>>>>>>> cmdVersion: 0, status: IN_PROGRESS,
> > processStatus:
> > > > 0,
> > > > > > > > >> > >>> resultCode:
> > > > > > > > >> > >>>>> 0,
> > > > > > > > >> > >>>>>>>>> result: null, initMsid: 264216221068220,
> > > > completeMsid:
> > > > > > > null,
> > > > > > > > >> > >>>>>>> lastUpdated:
> > > > > > > > >> > >>>>>>>>> null, lastPolled: null, created: null}
> > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,211 DEBUG
> [c.c.a.ApiServlet]
> > > > > > > > >> > >>>>>>>>> (qtp510113906-285:ctx-6c5e11c3 ctx-0cc34dc6)
> > > > > > > > (logid:cd8e30be)
> > > > > > > > >> > >>>>>>>>> ===END===
> > > > > > > > >> > >>>>>>>>> 192.168.5.140 -- GET
> > > > > > > > >> > >>>>>>>>>
> > > > > > > > >> > >>>>>>>>>
> > > > > > > > >> > >>>>>>>>
> > > > > > > > >> > >>>>>
> > > > > > > > >> >
> > > > > > >
> > > >
> command=deleteNetwork&id=4ba834ed-48f3-468f-b667-9bb2d2c258f1&response
> > > > > > > > >> > >>>>>>>> =json&_=1552998473154
> > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,212 DEBUG
> > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081)
> > > > > > > (logid:16897ea6)
> > > > > > > > >> > >>>>>>>>> Executing AsyncJobVO {id:5081, userId: 2,
> > > accountId:
> > > > > 2,
> > > > > > > > >> > >>>>>>>>> instanceType: Network,
> > > > > > > > >> > >>>>>>>>> instanceId: 204, cmd:
> > > > > > > > >> > >>>>>>>>>
> > > > > > > > >> > >>>
> > > > > > org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd,
> > > > > > > > >> > >>>>>>> cmdInfo:
> > > > > > > > >> > >>>>>>>>>
> > > > > > > > >> > >>>>>>>>>
> > > > > > > > >> > >>>>>>>>
> > > > > > > > >> > >>>>>
> > > > > > > > >> >
> > > > > > >
> > > >
> {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId
> > > > > > > > >> > >>>>>>>>
> > > > > > > > >> > >>>>>
> > > > > > > > >> >
> > > > > > >
> > > >
> ":"2615","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\"
> > > > > > > > >> > >>>>>>>> interface
> > > > > > > > >> > >>>>>>>>>
> > > > > > > > >> > >>>>>>>>>
> > > > > > > > >> > >>>>>>>>
> > > > > > > > >> > >>>>>
> > > > > > > > >> >
> > > > > > >
> > > >
> com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}"
> > > > > > > > >> > >>>>>>>>
> > > > > > > > >> > >>>>>
> > > > > > > > >> >
> > > > > > >
> > > >
> ,"ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmd
> > > > > > > > >> > >>>>>>>>
> EventType":"NETWORK.DELETE","_":"1552998473154"},
> > > > > > > > >> > >>>>>>>>> cmdVersion: 0, status: IN_PROGRESS,
> > processStatus:
> > > > 0,
> > > > > > > > >> > >>> resultCode:
> > > > > > > > >> > >>>>> 0,
> > > > > > > > >> > >>>>>>>>> result: null, initMsid: 264216221068220,
> > > > completeMsid:
> > > > > > > null,
> > > > > > > > >> > >>>>>>> lastUpdated:
> > > > > > > > >> > >>>>>>>>> null, lastPolled: null, created: null}
> > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,257 WARN
> > > > > > > > [o.a.c.e.o.NetworkOrchestrator]
> > > > > > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081
> > > > > ctx-d5de7979)
> > > > > > > > >> > >>>>>>>>> (logid:16897ea6) Can't delete the network, not
> > all
> > > > > user
> > > > > > > vms
> > > > > > > > >> are
> > > > > > > > >> > >>>>>>>>> expunged. Vm VM[User|i-2-11-VM] is in Stopped
> > > state
> > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,263 DEBUG
> > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081)
> > > > > > > (logid:16897ea6)
> > > > > > > > >> > >>>>> Complete
> > > > > > > > >> > >>>>>>>> async
> > > > > > > > >> > >>>>>>>>> job-5081, jobStatus: FAILED, resultCode: 530,
> > > > result:
> > > > > > > > >> > >>>>>>>>>
> > > > > > > > >> > >>>>>>>>>
> > > > > > > > >> > >>>>>>>>
> > > > > > > > >> > >>>>>
> > > > > > > > >> >
> > > > > > >
> > > >
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":
> > > > > > > > >> > >>>>>>>> [],"errorcode":530,"errortext":"Failed
> > > > > > > > >> > >>>>>>>>> to delete network"}
> > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,264 DEBUG
> > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081)
> > > > > > > (logid:16897ea6)
> > > > > > > > >> > >>>>> Publish
> > > > > > > > >> > >>>>>>>>> async
> > > > > > > > >> > >>>>>>>>> job-5081 complete on message bus
> > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,264 DEBUG
> > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081)
> > > > > > > (logid:16897ea6)
> > > > > > > > >> > >>> Wake
> > > > > > > > >> > >>>>> up
> > > > > > > > >> > >>>>>>>>> jobs related to job-5081
> > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,264 DEBUG
> > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081)
> > > > > > > (logid:16897ea6)
> > > > > > > > >> > >>> Update
> > > > > > > > >> > >>>>>>>>> db status for job-5081
> > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,265 DEBUG
> > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081)
> > > > > > > (logid:16897ea6)
> > > > > > > > >> > >>> Wake
> > > > > > > > >> > >>>>> up
> > > > > > > > >> > >>>>>>>>> jobs joined with job-5081 and disjoin all
> > subjobs
> > > > > > created
> > > > > > > > from
> > > > > > > > >> > >>>>> job-
> > > > > > > > >> > >>>>>>>>> 5081
> > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,267 DEBUG
> > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081)
> > > > > > > (logid:16897ea6)
> > > > > > > > >> > >>> Done
> > > > > > > > >> > >>>>>>>> executing
> > > > > > > > >> > >>>>>>>>>
> > > > > > > > >>
> > > org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd
> > > > > > > > >> > >>>>> for
> > > > > > > > >> > >>>>>>>>> job-5081
> > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,267 INFO
> > > > > > > [o.a.c.f.j.i.AsyncJobMonitor]
> > > > > > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081)
> > > > > > > (logid:16897ea6)
> > > > > > > > >> > >>> Remove
> > > > > > > > >> > >>>>>>>> job-5081
> > > > > > > > >> > >>>>>>>>> from job monitoring
> > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:56,230 DEBUG
> [c.c.a.ApiServlet]
> > > > > > > > >> > >>>>>>>>> (qtp510113906-28:ctx-e6c5bc85)
> (logid:7fe68f75)
> > > > > > > ===START===
> > > > > > > > >> > >>>>>>>> 192.168.5.140
> > > > > > > > >> > >>>>>>>>> -- GET
> > > > > > > > >> > >>>>>>>>>
> > > > > > > > >> > >>>>>>>>>
> > > > > > > > >> > >>>>>>>>
> > > > > > > > >> > >>>>>
> > > > > > > > >> >
> > > > > > >
> > > >
> command=queryAsyncJobResult&jobId=16897ea6-27c3-45b9-a0df-ab217bb5393c
> > > > > > > > >> > >>>>>>>> &response=json&_=1552998476202
> > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:56,238 DEBUG
> [c.c.a.ApiServer]
> > > > > > > > >> > >>>>>>>>> (qtp510113906-28:ctx-e6c5bc85 ctx-da1f4cbd)
> > > > > > > (logid:7fe68f75)
> > > > > > > > >> > >>> CIDRs
> > > > > > > > >> > >>>>>>>>> from which account
> > > > > > > > >> > >>>>>>>>>
> > 'Acct[15863393-8e8d-11e7-8f52-f04da2002bbe-admin]'
> > > > is
> > > > > > > > >> > >>>>>>>> allowed
> > > > > > > > >> > >>>>>>>>> to perform API calls: 0.0.0.0/0,::/0
> > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:56,260 DEBUG
> [c.c.a.ApiServlet]
> > > > > > > > >> > >>>>>>>>> (qtp510113906-28:ctx-e6c5bc85 ctx-da1f4cbd)
> > > > > > > (logid:7fe68f75)
> > > > > > > > >> > >>>>>>>>> ===END===
> > > > > > > > >> > >>>>>>>>> 192.168.5.140 -- GET
> > > > > > > > >> > >>>>>>>>>
> > > > > > > > >> > >>>>>>>>>
> > > > > > > > >> > >>>>>>>>
> > > > > > > > >> > >>>>>
> > > > > > > > >> >
> > > > > > >
> > > >
> command=queryAsyncJobResult&jobId=16897ea6-27c3-45b9-a0df-ab217bb5393c
> > > > > > > > >> > >>>>>>>> &response=json&_=1552998476202
> > > > > > > > >> > >>>>>>>>> 2019-03-19 12:28:00,946 INFO
> > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > > > >> > >>>>>>>>> (AsyncJobMgr-Heartbeat-1:ctx-9b43d1fd)
> > > > > (logid:a605267a)
> > > > > > > > Begin
> > > > > > > > >> > >>>>>>>>> cleanup expired async-jobs
> > > > > > > > >> > >>>>>>>>> 2019-03-19 12:28:00,951 INFO
> > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > > > >> > >>>>>>>>> (AsyncJobMgr-Heartbeat-1:ctx-9b43d1fd)
> > > > > (logid:a605267a)
> > > > > > > End
> > > > > > > > >> > >>>>> cleanup
> > > > > > > > >> > >>>>>>>> expired
> > > > > > > > >> > >>>>>>>>> async-jobs
> > > > > > > > >> > >>>>>>>>> 2019-03-19 12:28:01,142 DEBUG
> > > > > > > > >> > >>>>>>>> [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> > > > > > > > >> > >>>>>>>>> (RouterStatusMonitor-1:ctx-ad6bbe7e)
> > > > (logid:04e4c72b)
> > > > > > > Found
> > > > > > > > 0
> > > > > > > > >> > >>>>>>>>> routers to update status.
> > > > > > > > >> > >>>>>>>>>
> > > > > > > > >> > >>>>>>>>
> > > > > > > > >> > >>>>>>>>
> > > > > > > > >> > >>>>>>>> --
> > > > > > > > >> > >>>>>>>> Rafael Weingärtner
> > > > > > > > >> > >>>>>>>>
> > > > > > > > >> > >>>>>>>
> > > > > > > > >> > >>>>>>
> > > > > > > > >> > >>>>>
> > > > > > > > >> > >>>>
> > > > > > > > >> > >>>
> > > > > > > > >> > >>
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
--
Andrija Panić
