Network offering needs to be change to non-redundant. Most likely old bug is
resurfaced.
https://issues.apache.org/jira/browse/CLOUDSTACK-9024
On 3/19/19, 2:19 PM, "Jevgeni Zolotarjov" <j.zolotar...@gmail.com> wrote:
Hello
I did exactly like you suggested.
After UPDATE on db, I can see the router in cloudstack console.
But attempt to restart network fails.
I get error:
Resource [DataCenter:1] is unreachable: Can't find all necessary running
routers!
I rechecked agents on both servers. They look running OK.
On Tue, Mar 19, 2019 at 10:00 PM Andrija Panic <andrija.pa...@gmail.com>
wrote:
> Ok, so:
>
> 1.BACKUP YOUR DB - in case there is issues, you will want to restore
> 2. I tried to reproduce your case on 4.11.2 (though didn't do any
> maintenance etc.) - and could not - I have artificially marked existing VR
> as removed in DB - and tried to restart network and it worked just fine.
>
> Let's try to "undelete" the VR for that network - though I can't be sure
if
> this will work or not.
>
> Find VR based on the network UUID:
> SELECT * FROM vm_instance WHERE id IN (SELECT instance_id from nics
> where network_id IN (SELECT id FROM networks WHERE
> UUID="65ca9a05-ff96-4563-ab9c-ffb610dc8b73"));
> Obviously, replace this UUID with your network UUID.
> SQL above will return all VMs (user VMs, VRs, etc, delete or not...) that
> have NIC inside the network with that UUID - look for the r-XX-VM being a
> router.
>
> Now that you have the name of the VR (i.e. r-4-VM) - you want to set 2
> fields to some other value than what it might be already, as below:
>
> Need to set field "state" to Stopped:
> UPDATE cloud.vm_instance SET state='Stopped' WHERE name="r-4-VM";
> (what state is set currently ???)
>
> Need to set "removed" to NULL value:
> UPDATE cloud.vm_instance SET removed = NULL WHERE NAME="r-4-VM"; (is
> there a removed date already set ???)
>
> Obviously make sure that this VR is not running on any hypervisor and if
> running, virsh destroy it...
>
> AFTER above has been done (VR is considered to exist but stopped, from
> CloudStack point of view) - try to RESTART the network (don't bother
> deleteing it, since there are VMs in that network)
>
> I'm not 100% positive this will fix your issue, but doesn't hurt to try
>
> If above doesn't work - I would still take a look into agents and if they
> are still connected - optionally, restart agents on both hosts once more
> and confirm they are connected and up.
>
> Let us know how it goes (and please backup DB once more before any actions
> !)
>
> Andrija
>
>
> On Tue, 19 Mar 2019 at 20:41, Jevgeni Zolotarjov <j.zolotar...@gmail.com>
> wrote:
>
> > Yes. Just a single network.
> >
> > On Tue, 19 Mar 2019, 21:39 Andrija Panic, <andrija.pa...@gmail.com>
> wrote:
> >
> > > Just one more clarification - this is Isolate single network (not
> Shared
> > > Network, not VPC) ?
> > >
> > >
> > >
> > > On Tue, Mar 19, 2019, 19:36 Jevgeni Zolotarjov <j.zolotar...@gmail.com
> >
> > > wrote:
> > >
> > > > Name defaultGuestNetwork
> > > > ID 4ba834ed-48f3-468f-b667-9bb2d2c258f1
> > > > Zone PBT zone 1
> > > > Description defaultGuestNetwork
> > > > Type Shared
> > > > State Setup
> > > > VPC ID N/A
> > > > Persistent No
> > > >
> > > > On Tue, Mar 19, 2019 at 8:29 PM Andrija Panic <
> andrija.pa...@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Share the network id and name as seen from GUI...
> > > > >
> > > > > On Tue, Mar 19, 2019, 19:27 Jevgeni Zolotarjov <
> > j.zolotar...@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > >>> 1. Confirm please rolling restart is set to false please -
> > double
> > > > > check
> > > > > > Double checked - It is set to false
> > > > > >
> > > > > > >>>>2. If so - do you know the name of VR which you deleted ? Is
> it
> > > > last
> > > > > > one
> > > > > > >>> ever created - if so we can find it easily...
> > > > > > I dont know the name
> > > > > > Is there a way to fetch it from DB?
> > > > > >
> > > > > > On Tue, Mar 19, 2019 at 7:58 PM Andrija Panic <
> > > andrija.pa...@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Right...still complaining on missing running routers.
> > > > > > >
> > > > > > > 1. Confirm please rolling restart is set to false please -
> double
> > > > check
> > > > > > > 2. If so - do you know the name of VR which you deleted ? Is
it
> > > last
> > > > > one
> > > > > > > ever created - if so we can find it easily...
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Mar 19, 2019, 18:40 Jevgeni Zolotarjov <
> > > > j.zolotar...@gmail.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Here is management server log
> > > > > > > >
> > > https://drive.google.com/open?id=1H2jI0roeiWxtzReB8qV6QxDkNpaki99A
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Mar 19, 2019 at 7:29 PM Andrija Panic <
> > > > > andrija.pa...@gmail.com
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I would also try to "undelete" VR in DB, but let's keep
> this
> > as
> > > > > last
> > > > > > > > step.
> > > > > > > > >
> > > > > > > > > On Tue, Mar 19, 2019, 18:24 Andrija Panic <
> > > > andrija.pa...@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Disclaimer: from mobile device.
> > > > > > > > > >
> > > > > > > > > > I don't see any special failure in agent log, except
some
> > > long
> > > > > > > running
> > > > > > > > > > migration, timeout for graceful VM shutdown etc (and
> agent
> > > > > restart)
> > > > > > > > > >
> > > > > > > > > > You did not send mgmt log after last network restart
> > failure
> > > ?
> > > > > > > > > >
> > > > > > > > > > On Tue, Mar 19, 2019, 17:29 Jevgeni Zolotarjov <
> > > > > > > j.zolotar...@gmail.com
> > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> Under
> > > > > > > > > >> Infrastructure / Hosts:
> > > > > > > > > >> both hosts are enabled and green (Unsecure though)
> > > > > > > > > >>
> > > > > > > > > >> agent.log -
> > > > > > > > > >>
> > > > > >
> https://drive.google.com/open?id=1rATxHKqgNKo2kD23BtlrZy_9gFXC-Bq-
> > > > > > > > > >>
> > > > > > > > > >> I set network.rolling.restart to false now. Restarted
> > > > management
> > > > > > > > server
> > > > > > > > > -
> > > > > > > > > >> same problem - cannot restart not delete network
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >> On Tue, Mar 19, 2019 at 5:48 PM Boris Stoyanov <
> > > > > > > > > >> boris.stoya...@shapeblue.com>
> > > > > > > > > >> wrote:
> > > > > > > > > >>
> > > > > > > > > >> > Hi, you shouldn’t be worried about data as long as
you
> > > use a
> > > > > > > > separate
> > > > > > > > > >> > shared storage.
> > > > > > > > > >> >
> > > > > > > > > >> > What does the cloudstack console say about your host?
> Is
> > > it
> > > > > up?
> > > > > > If
> > > > > > > > > it’s
> > > > > > > > > >> up
> > > > > > > > > >> > then you should be able to deploy a VM on it.
> > > > > > > > > >> >
> > > > > > > > > >> > If not, you’ll need to investigate the reason
> > > > cloudstack-agent
> > > > > > is
> > > > > > > > not
> > > > > > > > > >> > connected, is the agent running? If not just start it
> > and
> > > > see
> > > > > if
> > > > > > > it
> > > > > > > > > >> > connects.
> > > > > > > > > >> >
> > > > > > > > > >> > If it’s running but still not connected please share
> the
> > > > > > > agent.log.
> > > > > > > > > >> >
> > > > > > > > > >> > If nothing helps at all you can remove the host and
> add
> > it
> > > > > > again,
> > > > > > > > but
> > > > > > > > > >> > let’s try to troubleshoot first.
> > > > > > > > > >> >
> > > > > > > > > >> > Bobby.
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> > boris.stoya...@shapeblue.com
> > > > > > > > > >> > www.shapeblue.com
> > > > > > > > > >> > Amadeus House, Floral Street, London WC2E 9DPUK
> > > > > > > > > >> > @shapeblue
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> > > On 19 Mar 2019, at 17:29, Jevgeni Zolotarjov <
> > > > > > > > > j.zolotar...@gmail.com>
> > > > > > > > > >> > wrote:
> > > > > > > > > >> > >
> > > > > > > > > >> > > Guys, please help with it.
> > > > > > > > > >> > > What can be done here?
> > > > > > > > > >> > > There is too much valuable data.
> > > > > > > > > >> > >
> > > > > > > > > >> > > On Tue, Mar 19, 2019 at 4:21 PM Jevgeni Zolotarjov
<
> > > > > > > > > >> > j.zolotar...@gmail.com>
> > > > > > > > > >> > > wrote:
> > > > > > > > > >> > >
> > > > > > > > > >> > >> Tried that just now and got error:
> > > > > > > > > >> > >> Resource [DataCenter:1] is unreachable: Can't find
> > all
> > > > > > > necessary
> > > > > > > > > >> running
> > > > > > > > > >> > >> routers!
> > > > > > > > > >> > >>
> > > > > > > > > >> > >> In the log I see:
> > > > > > > > > >> > >> =============================================
> > > > > > > > > >> > >>
> > > > > > > > > >> > >> 2019-03-19 14:20:39,644 DEBUG
> > > > > [o.a.c.e.o.NetworkOrchestrator]
> > > > > > > > > >> > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093
> > ctx-9be30648)
> > > > > > > > > >> (logid:265a6099)
> > > > > > > > > >> > >> Restarting network 204...
> > > > > > > > > >> > >> 2019-03-19 14:20:39,645 DEBUG
> > > > > [o.a.c.e.o.NetworkOrchestrator]
> > > > > > > > > >> > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093
> > ctx-9be30648)
> > > > > > > > > >> (logid:265a6099)
> > > > > > > > > >> > >> Performing rolling restart of routers of network
> > > > > > > > Ntwk[204|Guest|6]
> > > > > > > > > >> > >> 2019-03-19 14:20:39,649 DEBUG
> > > > > [o.a.c.e.o.NetworkOrchestrator]
> > > > > > > > > >> > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093
> > ctx-9be30648)
> > > > > > > > > >> (logid:265a6099)
> > > > > > > > > >> > >> Asking VirtualRouter to implemenet
> Ntwk[204|Guest|6]
> > > > > > > > > >> > >> 2019-03-19 14:20:39,658 ERROR
> > > > [c.c.a.ApiAsyncJobDispatcher]
> > > > > > > > > >> > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093)
> > > > (logid:265a6099)
> > > > > > > > > >> Unexpected
> > > > > > > > > >> > >> exception while executing
> > > > > > > > > >> > >>
> > > > > > >
> org.apache.cloudstack.api.command.user.network.RestartNetworkCmd
> > > > > > > > > >> > >> com.cloud.exception.ResourceUnavailableException:
> > > > Resource
> > > > > > > > > >> > [DataCenter:1]
> > > > > > > > > >> > >> is unreachable: Can't find all necessary running
> > > routers!
> > > > > > > > > >> > >> at
> > > > > > > > > >> > >>
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:243)
> > > > > > > > > >> > >> at
> > > > > > > > > >> > >>
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElements(NetworkOrchestrator.java:1203)
> > > > > > > > > >> > >> at
> > > > > > > > > >> > >>
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.rollingRestartRouters(NetworkOrchestrator.java:2948)
> > > > > > > > > >> > >> at
> > > > > > > > > >> > >>
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:2853)
> > > > > > > > > >> > >> at
> > > > > > > > > >> > >>
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1883)
> > > > > > > > > >> > >> at
> > > > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> > > > > > > > > Method)
> > > > > > > > > >> > >> at
> > > > > sun.reflect.NativeMethodAccessorImpl.invoke(Unknown
> > > > > > > > > Source)
> > > > > > > > > >> > >> at
> > > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
> > > > > > > > > >> > Source)
> > > > > > > > > >> > >> at java.lang.reflect.Method.invoke(Unknown
> > > Source)
> > > > > > > > > >> > >> at
> > > > > > > > > >> > >>
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
> > > > > > > > > >> > >> at
> > > > > > > > > >> > >>
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
> > > > > > > > > >> > >> at
> > > > > > > > > >> > >>
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> > > > > > > > > >> > >> at
> > > > > > > > > >> > >>
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
> > > > > > > > > >> > >> at
> > > > > > > > > >> > >>
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> > > > > > > > > >> > >> at
> > > > > > > > > >> > >>
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
> > > > > > > > > >> > >> at
> > > > > > > > > >> > >>
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> > > > > > > > > >> > >> at
> > > > > > > > > >> > >>
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
> > > > > > > > > >> > >> at
> > > > > > > > > >> > >>
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
> > > > > > > > > >> > >> at
> > > > > > > > > >> > >>
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
> > > > > > > > > >> > >> at
> > > com.sun.proxy.$Proxy229.restartNetwork(Unknown
> > > > > > > Source)
> > > > > > > > > >> > >> at
> > > > > > > > > >> > >>
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:99)
> > > > > > > > > >> > >> at
> > > > > > > > > >>
> > com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
> > > > > > > > > >> > >> at
> > > > > > > > > >> > >>
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
> > > > > > > > > >> > >> at
> > > > > > > > > >> > >>
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
> > > > > > > > > >> > >> at
> > > > > > > > > >> > >>
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> > > > > > > > > >> > >> at
> > > > > > > > > >> > >>
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> > > > > > > > > >> > >> at
> > > > > > > > > >> > >>
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> > > > > > > > > >> > >> at
> > > > > > > > > >> > >>
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> > > > > > > > > >> > >> at
> > > > > > > > > >> > >>
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> > > > > > > > > >> > >> at
> > > > > > > > > >> > >>
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
> > > > > > > > > >> > >> at
> > > > > > > > >
java.util.concurrent.Executors$RunnableAdapter.call(Unknown
> > > > > > > > > >> > >> Source)
> > > > > > > > > >> > >> at
> java.util.concurrent.FutureTask.run(Unknown
> > > > > Source)
> > > > > > > > > >> > >> at
> > > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> > > > > > > > > >> > >> Source)
> > > > > > > > > >> > >> at
> > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> > > > > > > > > >> > >> Source)
> > > > > > > > > >> > >> at java.lang.Thread.run(Unknown Source)
> > > > > > > > > >> > >> 2019-03-19 14:20:39,659 DEBUG
> > > > > > [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > > > > >> > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093)
> > > > (logid:265a6099)
> > > > > > > > > Complete
> > > > > > > > > >> > async
> > > > > > > > > >> > >> job-5093, jobStatus: FAILED, resultCode: 530,
> result:
> > > > > > > > > >> > >>
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Resource
> > > > > > > > > >> > >> [DataCenter:1] is un
> > > > > > > > > >> > >>
> > > > > > > > > >> > >> On Tue, Mar 19, 2019 at 4:19 PM Andrija Panic <
> > > > > > > > > >> > andrija.pa...@shapeblue.com>
> > > > > > > > > >> > >> wrote:
> > > > > > > > > >> > >>
> > > > > > > > > >> > >>>
> > > > > > > > > >> > >>> Your network can't be deleted due to "Can't
delete
> > the
> > > > > > > network,
> > > > > > > > > not
> > > > > > > > > >> all
> > > > > > > > > >> > >>> user vms are expunged. Vm
> > > > > > > > > >> > >>> VM[User|i-2-11-VM] is in Stopped state" - which
is
> > > fine.
> > > > > > > > > >> > >>>
> > > > > > > > > >> > >>> You should be able to just start the user VM -
but
> > if
> > > > you
> > > > > > have
> > > > > > > > > >> actually
> > > > > > > > > >> > >>> delete the VR itself, then just do Network
restart
> > > with
> > > > > > > > "cleanup"
> > > > > > > > > >> and
> > > > > > > > > >> > it
> > > > > > > > > >> > >>> will recreate a new VR, after which you should be
> > able
> > > > to
> > > > > > > start
> > > > > > > > > the
> > > > > > > > > >> VM.
> > > > > > > > > >> > >>>
> > > > > > > > > >> > >>> Andrija
> > > > > > > > > >> > >>>
> > > > > > > > > >> > >>> andrija.pa...@shapeblue.com
> > > > > > > > > >> > >>> www.shapeblue.com
> > > > > > > > > >> > >>> Amadeus House, Floral Street, London WC2E 9DPUK
> > > > > > > > > >> > >>> @shapeblue
> > > > > > > > > >> > >>>
> > > > > > > > > >> > >>>
> > > > > > > > > >> > >>>
> > > > > > > > > >> > >>>
> > > > > > > > > >> > >>> -----Original Message-----
> > > > > > > > > >> > >>> From: Jevgeni Zolotarjov <j.zolotar...@gmail.com>
> > > > > > > > > >> > >>> Sent: 19 March 2019 15:10
> > > > > > > > > >> > >>> To: users@cloudstack.apache.org
> > > > > > > > > >> > >>> Subject: Re: Disaster after maintenance
> > > > > > > > > >> > >>>
> > > > > > > > > >> > >>> I mean I cannot delete network: In the management
> > > server
> > > > > > log I
> > > > > > > > see
> > > > > > > > > >> > >>>
> > > > > > > > > >> > >>>
> > > > ==========================================================
> > > > > > > > > >> > >>> 019-03-19 14:06:36,316 DEBUG
> > > > > > [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > > > > >> > >>> (API-Job-Executor-1:ctx-1c0fd4dc job-5090)
> > > > > (logid:c734edfc)
> > > > > > > > > >> Executing
> > > > > > > > > >> > >>> AsyncJobVO {id:5090, userId: 2, accountId: 2,
> > > > > instanceType:
> > > > > > > > > Network,
> > > > > > > > > >> > >>> instanceId: 204, cmd:
> > > > > > > > > >> > >>>
> > > > > > >
> org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd,
> > > > > > > > > >> > cmdInfo:
> > > > > > > > > >> > >>>
> > > > > > > > > >> > >>>
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
{"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"2641","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\"interface
> > > > > > > > > >> > >>>
> > > > > > > > > >> > >>>
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}","ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmdEventType":"NETWORK.DELETE","_":"1553004396247"},
> > > > > > > > > >> > >>> cmdVersion: 0, status: IN_PROGRESS,
processStatus:
> > 0,
> > > > > > > > resultCode:
> > > > > > > > > 0,
> > > > > > > > > >> > >>> result: null, initMsid: 264216221068220,
> > completeMsid:
> > > > > null,
> > > > > > > > > >> > lastUpdated:
> > > > > > > > > >> > >>> null, lastPolled: null, created: null}
> > > > > > > > > >> > >>> 2019-03-19 14:06:36,351 WARN
> > > > > > [o.a.c.e.o.NetworkOrchestrator]
> > > > > > > > > >> > >>> (API-Job-Executor-1:ctx-1c0fd4dc job-5090
> > > ctx-134954fa)
> > > > > > > > > >> > (logid:c734edfc)
> > > > > > > > > >> > >>> Can't delete the network, not all user vms are
> > > expunged.
> > > > > Vm
> > > > > > > > > >> > >>> VM[User|i-2-11-VM] is in Stopped state
> > > > > > > > > >> > >>> 2019-03-19 14:06:36,356 DEBUG
> > > > > > > [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > > > > >> > >>> (API-Job-Executor-1:ctx-1c0fd4dc job-5090)
> > > > > (logid:c734edfc)
> > > > > > > > > Complete
> > > > > > > > > >> > async
> > > > > > > > > >> > >>> job-5090, jobStatus: FAILED, resultCode: 530,
> > result:
> > > > > > > > > >> > >>>
> > > > > > > > > >> > >>>
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
> > > > > > > > > >> > >>> to delete network"}
> > > > > > > > > >> > >>>
> > > > ==========================================================
> > > > > > > > > >> > >>>
> > > > > > > > > >> > >>>
> > > > > > > > > >> > >>> I deleted a router, expecting it to be recreated
> on
> > > > > deleting
> > > > > > > > > >> network.
> > > > > > > > > >> > But
> > > > > > > > > >> > >>> I am unable to delete network because of above
> error
> > > > > > > > > >> > >>>
> > > > > > > > > >> > >>> On Tue, Mar 19, 2019 at 3:58 PM Jevgeni
> Zolotarjov <
> > > > > > > > > >> > >>> j.zolotar...@gmail.com>
> > > > > > > > > >> > >>> wrote:
> > > > > > > > > >> > >>>
> > > > > > > > > >> > >>>> I've managed to make libvirtd running
> > > > > > > > > >> > >>>> Now cloudstack console shows both hosts -
running
> > > > > > > > > >> > >>>>
> > > > > > > > > >> > >>>> But now as I have removed network, VMs are
unable
> > to
> > > > > start.
> > > > > > > > > >> > >>>>
> > > > > > > > > >> > >>>> How can I recreate the network now?
> > > > > > > > > >> > >>>>
> > > > > > > > > >> > >>>> On Tue, Mar 19, 2019 at 3:14 PM Ivan Kudryavtsev
> > > > > > > > > >> > >>>> <kudryavtsev...@bw-sw.com>
> > > > > > > > > >> > >>>> wrote:
> > > > > > > > > >> > >>>>
> > > > > > > > > >> > >>>>> Jevgeniy, it may be a documentation bug. Take s
> > > look:
> > > > > > > > > >> > >>>>>
> > > > > > > > >
> > > https://github.com/apache/cloudstack-documentation/pull/27/files
> > > > > > > > > >> > >>>>>
> > > > > > > > > >> > >>>>> вт, 19 мар. 2019 г., 9:09 Jevgeni Zolotarjov <
> > > > > > > > > >> j.zolotar...@gmail.com
> > > > > > > > > >> > >:
> > > > > > > > > >> > >>>>>
> > > > > > > > > >> > >>>>>> That's it - libvirtd failed to start on second
> > > host.
> > > > > > > > > >> > >>>>>> Tried restarting, but it does not start.
> > > > > > > > > >> > >>>>>>
> > > > > > > > > >> > >>>>>>
> > > > > > > > > >> > >>>>>>>> Do you have some NUMA constraints or
anything
> > > which
> > > > > > > > requires
> > > > > > > > > >> > >>>>> particular
> > > > > > > > > >> > >>>>>> RAM configuration?
> > > > > > > > > >> > >>>>>> No
> > > > > > > > > >> > >>>>>>
> > > > > > > > > >> > >>>>>> libvirtd.service - Virtualization daemon
> > > > > > > > > >> > >>>>>> Loaded: loaded
> > > > > > (/usr/lib/systemd/system/libvirtd.service;
> > > > > > > > > >> > >>>>>> enabled; vendor preset: enabled)
> > > > > > > > > >> > >>>>>> Active: failed (Result: start-limit) since
> Tue
> > > > > > 2019-03-19
> > > > > > > > > >> > >>>>>> 13:03:07
> > > > > > > > > >> > >>>>> GMT;
> > > > > > > > > >> > >>>>>> 12s ago
> > > > > > > > > >> > >>>>>> Docs: man:libvirtd(8)
> > > > > > > > > >> > >>>>>> https://libvirt.org
> > > > > > > > > >> > >>>>>> Process: 892 ExecStart=/usr/sbin/libvirtd
> > > > > $LIBVIRTD_ARGS
> > > > > > > > > >> > >>>>>> (code=exited,
> > > > > > > > > >> > >>>>>> status=1/FAILURE)
> > > > > > > > > >> > >>>>>> Main PID: 892 (code=exited, status=1/FAILURE)
> > > > > > > > > >> > >>>>>> Tasks: 19 (limit: 32768)
> > > > > > > > > >> > >>>>>> CGroup: /system.slice/libvirtd.service
> > > > > > > > > >> > >>>>>> ├─11338 /usr/sbin/libvirtd -d -l
> > > > > > > > > >> > >>>>>> ├─11909 /usr/sbin/dnsmasq
> > > > > > > > > >> > >>>>>>
> --conf-file=/var/lib/libvirt/dnsmasq/default.conf
> > > > > > > > > --leasefile-ro
> > > > > > > > > >> > >>>>>>
--dhcp-script=/usr/libexec/libvirt_leaseshelper
> > > > > > > > > >> > >>>>>> └─11910 /usr/sbin/dnsmasq
> > > > > > > > > >> > >>>>>>
> --conf-file=/var/lib/libvirt/dnsmasq/default.conf
> > > > > > > > > --leasefile-ro
> > > > > > > > > >> > >>>>>>
--dhcp-script=/usr/libexec/libvirt_leaseshelper
> > > > > > > > > >> > >>>>>>
> > > > > > > > > >> > >>>>>> Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt
> > > > > systemd[1]:
> > > > > > > > Failed
> > > > > > > > > >> to
> > > > > > > > > >> > >>>>>> start Virtualization daemon.
> > > > > > > > > >> > >>>>>> Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt
> > > > > systemd[1]:
> > > > > > > Unit
> > > > > > > > > >> > >>>>>> libvirtd.service entered failed state.
> > > > > > > > > >> > >>>>>> Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt
> > > > > systemd[1]:
> > > > > > > > > >> > >>>>> libvirtd.service
> > > > > > > > > >> > >>>>>> failed.
> > > > > > > > > >> > >>>>>> Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt
> > > > > systemd[1]:
> > > > > > > > > >> > >>>>> libvirtd.service
> > > > > > > > > >> > >>>>>> holdoff time over, scheduling restart.
> > > > > > > > > >> > >>>>>> Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt
> > > > > systemd[1]:
> > > > > > > > > Stopped
> > > > > > > > > >> > >>>>>> Virtualization daemon.
> > > > > > > > > >> > >>>>>> Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt
> > > > > systemd[1]:
> > > > > > > > start
> > > > > > > > > >> > >>>>>> request repeated too quickly for
> libvirtd.service
> > > Mar
> > > > > 19
> > > > > > > > > 13:03:07
> > > > > > > > > >> > >>>>>> mtl1-apphst04.mt.pbt.com.mt systemd[1]: Failed
> > to
> > > > > start
> > > > > > > > > >> > >>>>>> Virtualization daemon.
> > > > > > > > > >> > >>>>>> Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt
> > > > > systemd[1]:
> > > > > > > Unit
> > > > > > > > > >> > >>>>>> libvirtd.service entered failed state.
> > > > > > > > > >> > >>>>>> Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt
> > > > > systemd[1]:
> > > > > > > > > >> > >>>>> libvirtd.service
> > > > > > > > > >> > >>>>>> failed.
> > > > > > > > > >> > >>>>>>
> > > > > > > > > >> > >>>>>>
> > > > > > > > > >> > >>>>>> On Tue, Mar 19, 2019 at 3:04 PM Paul Angus
> > > > > > > > > >> > >>>>>> <paul.an...@shapeblue.com>
> > > > > > > > > >> > >>>>>> wrote:
> > > > > > > > > >> > >>>>>>
> > > > > > > > > >> > >>>>>>> Can you check that the cloudstack agent is
> > running
> > > > on
> > > > > > the
> > > > > > > > host
> > > > > > > > > >> > >>>>>>> and the agent logs (usual logs directory)
Also
> > > worth
> > > > > > > > checking
> > > > > > > > > >> > >>>>>>> that libvirt has started ok. Do you have
some
> > > > > > > > > >> > >>>>> NUMA
> > > > > > > > > >> > >>>>>>> constraints or anything which requires
> > particular
> > > > RAM
> > > > > > > > > >> > >>> configuration?
> > > > > > > > > >> > >>>>>>>
> > > > > > > > > >> > >>>>>>> paul.an...@shapeblue.com
> > > > > > > > > >> > >>>>>>> www.shapeblue.com
> > > > > > > > > >> > >>>>>>> Amadeus House, Floral Street, London WC2E
> 9DPUK
> > > > > > > @shapeblue
> > > > > > > > > >> > >>>>>>>
> > > > > > > > > >> > >>>>>>>
> > > > > > > > > >> > >>>>>>>
> > > > > > > > > >> > >>>>>>>
> > > > > > > > > >> > >>>>>>> -----Original Message-----
> > > > > > > > > >> > >>>>>>> From: Jevgeni Zolotarjov <
> > j.zolotar...@gmail.com>
> > > > > > > > > >> > >>>>>>> Sent: 19 March 2019 14:49
> > > > > > > > > >> > >>>>>>> To: users@cloudstack.apache.org
> > > > > > > > > >> > >>>>>>> Subject: Re: Disaster after maintenance
> > > > > > > > > >> > >>>>>>>
> > > > > > > > > >> > >>>>>>> Can you try migrating a VM to the server that
> > you
> > > > > > changed
> > > > > > > > the
> > > > > > > > > >> RAM
> > > > > > > > > >> > >>>>> amount?
> > > > > > > > > >> > >>>>>>>
> > > > > > > > > >> > >>>>>>> Also:
> > > > > > > > > >> > >>>>>>> What is the hypervisor version?
> > > > > > > > > >> > >>>>>>> KVM
> > > > > > > > > >> > >>>>>>> QEMU Version : 2.0.0
> > > > > > > > > >> > >>>>>>> Release : 1.el7.6
> > > > > > > > > >> > >>>>>>>
> > > > > > > > > >> > >>>>>>>
> > > > > > > > > >> > >>>>>>> Host status in ACS?
> > > > > > > > > >> > >>>>>>> 1st server: Unsecure
> > > > > > > > > >> > >>>>>>> 2nd server: Disconnected
> > > > > > > > > >> > >>>>>>>
> > > > > > > > > >> > >>>>>>> Did you try to force a VM to start/deploy in
> > this
> > > > > server
> > > > > > > > where
> > > > > > > > > >> you
> > > > > > > > > >> > >>>>>> changed
> > > > > > > > > >> > >>>>>>> the RAM?
> > > > > > > > > >> > >>>>>>> Host status became disconnected. I don't know
> > how
> > > to
> > > > > > make
> > > > > > > it
> > > > > > > > > >> > >>>>> "connected"
> > > > > > > > > >> > >>>>>>> again
> > > > > > > > > >> > >>>>>>>
> > > > > > > > > >> > >>>>>>>
> > > > > > > > > >> > >>>>>>>
> > > > > > > > > >> > >>>>>>> On Tue, Mar 19, 2019 at 2:42 PM Rafael
> > > Weingärtner <
> > > > > > > > > >> > >>>>>>> rafaelweingart...@gmail.com> wrote:
> > > > > > > > > >> > >>>>>>>
> > > > > > > > > >> > >>>>>>>> Can you try migrating a VM to the server
that
> > you
> > > > > > changed
> > > > > > > > the
> > > > > > > > > >> RAM
> > > > > > > > > >> > >>>>>> amount?
> > > > > > > > > >> > >>>>>>>>
> > > > > > > > > >> > >>>>>>>> Also:
> > > > > > > > > >> > >>>>>>>> What is the hypervisor version?
> > > > > > > > > >> > >>>>>>>> Host status in ACS?
> > > > > > > > > >> > >>>>>>>> Did you try to force a VM to start/deploy in
> > this
> > > > > > server
> > > > > > > > > where
> > > > > > > > > >> > >>> you
> > > > > > > > > >> > >>>>>>>> changed the RAM?
> > > > > > > > > >> > >>>>>>>>
> > > > > > > > > >> > >>>>>>>>
> > > > > > > > > >> > >>>>>>>> On Tue, Mar 19, 2019 at 9:39 AM Jevgeni
> > > Zolotarjov
> > > > > > > > > >> > >>>>>>>> <j.zolotar...@gmail.com
> > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > >> > >>>>>>>> wrote:
> > > > > > > > > >> > >>>>>>>>
> > > > > > > > > >> > >>>>>>>>> We have Cloudstack 4.11.2 setup running
fine
> > for
> > > > few
> > > > > > > > months
> > > > > > > > > >> > >>> (>4)
> > > > > > > > > >> > >>>>> The
> > > > > > > > > >> > >>>>>>>>> setup is very simple: 2 hosts We decided to
> > do a
> > > > > > > > maintenance
> > > > > > > > > >> to
> > > > > > > > > >> > >>>>>>>>> increase RAM on both servers
> > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > >> > >>>>>>>>> For this we put first server to
maintenance.
> > All
> > > > VMS
> > > > > > > moved
> > > > > > > > > to
> > > > > > > > > >> > >>>>> second
> > > > > > > > > >> > >>>>>>>>> host after a while.
> > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > >> > >>>>>>>>> Then first server was shutdown, RAM
> increased,
> > > > > server
> > > > > > > > turned
> > > > > > > > > >> > >>> ON.
> > > > > > > > > >> > >>>>>>>>> Now nothing starts on first server.
> > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > >> > >>>>>>>>> Tried to delete network, but this fails as
> > well
> > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > >> > >>>>>>>>> Please help !
> > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > >> > >>>>>>>>> Here is extract from log:
> > > > > > > > > >> > >>>>>>>>> ======================================
> > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,064 DEBUG
> > > > > > > > > >> > >>>>> [o.a.c.s.SecondaryStorageManagerImpl]
> > > > > > > > > >> > >>>>>>>>> (secstorage-1:ctx-16d6c797)
(logid:7e3160ce)
> > > Zone
> > > > 1
> > > > > is
> > > > > > > > ready
> > > > > > > > > >> to
> > > > > > > > > >> > >>>>>>>>> launch secondary storage VM
> > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,125 DEBUG
> > > > > > > > > [c.c.c.ConsoleProxyManagerImpl]
> > > > > > > > > >> > >>>>>>>>> (consoleproxy-1:ctx-cbd034b9)
> (logid:0a8c8bf4)
> > > > Zone
> > > > > 1
> > > > > > is
> > > > > > > > > ready
> > > > > > > > > >> > >>> to
> > > > > > > > > >> > >>>>>>>>> launch console proxy
> > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,181 DEBUG
> > [c.c.a.ApiServlet]
> > > > > > > > > >> > >>>>>>>>> (qtp510113906-285:ctx-6c5e11c3)
> > (logid:cd8e30be)
> > > > > > > > ===START===
> > > > > > > > > >> > >>>>>>>> 192.168.5.140
> > > > > > > > > >> > >>>>>>>>> -- GET
> > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > >> > >>>>>>>>
> > > > > > > > > >> > >>>>>
> > > > > > > > > >> >
> > > > > > > >
> > > > >
> > command=deleteNetwork&id=4ba834ed-48f3-468f-b667-9bb2d2c258f1&response
> > > > > > > > > >> > >>>>>>>> =json&_=1552998473154
> > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,186 DEBUG
> > [c.c.a.ApiServer]
> > > > > > > > > >> > >>>>>>>>> (qtp510113906-285:ctx-6c5e11c3
ctx-0cc34dc6)
> > > > > > > > > (logid:cd8e30be)
> > > > > > > > > >> > >>>>> CIDRs
> > > > > > > > > >> > >>>>>>>>> from which account
> > > > > > > > > >> > >>>>>>>>>
> > > 'Acct[15863393-8e8d-11e7-8f52-f04da2002bbe-admin]'
> > > > > is
> > > > > > > > > >> > >>>>>>>> allowed
> > > > > > > > > >> > >>>>>>>>> to perform API calls: 0.0.0.0/0,::/0
> > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,208 INFO
> > > > > > > > [o.a.c.f.j.i.AsyncJobMonitor]
> > > > > > > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081)
> > > > > > > > (logid:f6751fa7)
> > > > > > > > > >> Add
> > > > > > > > > >> > >>>>>>>>> job-5081 into job monitoring
> > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,209 DEBUG
> > > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > > > > >> > >>>>>>>>> (qtp510113906-285:ctx-6c5e11c3
ctx-0cc34dc6)
> > > > > > > > > (logid:cd8e30be)
> > > > > > > > > >> > >>>>> submit
> > > > > > > > > >> > >>>>>>>> async
> > > > > > > > > >> > >>>>>>>>> job-5081, details: AsyncJobVO {id:5081,
> > userId:
> > > 2,
> > > > > > > > > accountId:
> > > > > > > > > >> > >>> 2,
> > > > > > > > > >> > >>>>>>>>> instanceType: Network, instanceId: 204,
cmd:
> > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > >> > >>>
> > > > > > >
> org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd,
> > > > > > > > > >> > >>>>>>> cmdInfo:
> > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > >> > >>>>>>>>
> > > > > > > > > >> > >>>>>
> > > > > > > > > >> >
> > > > > > > >
> > > > >
> > {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId
> > > > > > > > > >> > >>>>>>>>
> > > > > > > > > >> > >>>>>
> > > > > > > > > >> >
> > > > > > > >
> > > > >
> > ":"2615","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\"
> > > > > > > > > >> > >>>>>>>> interface
> > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > >> > >>>>>>>>
> > > > > > > > > >> > >>>>>
> > > > > > > > > >> >
> > > > > > > >
> > > > >
> > com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}"
> > > > > > > > > >> > >>>>>>>>
> > > > > > > > > >> > >>>>>
> > > > > > > > > >> >
> > > > > > > >
> > > > >
> > ,"ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmd
> > > > > > > > > >> > >>>>>>>>
> > EventType":"NETWORK.DELETE","_":"1552998473154"},
> > > > > > > > > >> > >>>>>>>>> cmdVersion: 0, status: IN_PROGRESS,
> > > processStatus:
> > > > > 0,
> > > > > > > > > >> > >>> resultCode:
> > > > > > > > > >> > >>>>> 0,
> > > > > > > > > >> > >>>>>>>>> result: null, initMsid: 264216221068220,
> > > > > completeMsid:
> > > > > > > > null,
> > > > > > > > > >> > >>>>>>> lastUpdated:
> > > > > > > > > >> > >>>>>>>>> null, lastPolled: null, created: null}
> > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,211 DEBUG
> > [c.c.a.ApiServlet]
> > > > > > > > > >> > >>>>>>>>> (qtp510113906-285:ctx-6c5e11c3
ctx-0cc34dc6)
> > > > > > > > > (logid:cd8e30be)
> > > > > > > > > >> > >>>>>>>>> ===END===
> > > > > > > > > >> > >>>>>>>>> 192.168.5.140 -- GET
> > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > >> > >>>>>>>>
> > > > > > > > > >> > >>>>>
> > > > > > > > > >> >
> > > > > > > >
> > > > >
> > command=deleteNetwork&id=4ba834ed-48f3-468f-b667-9bb2d2c258f1&response
> > > > > > > > > >> > >>>>>>>> =json&_=1552998473154
> > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,212 DEBUG
> > > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081)
> > > > > > > > (logid:16897ea6)
> > > > > > > > > >> > >>>>>>>>> Executing AsyncJobVO {id:5081, userId: 2,
> > > > accountId:
> > > > > > 2,
> > > > > > > > > >> > >>>>>>>>> instanceType: Network,
> > > > > > > > > >> > >>>>>>>>> instanceId: 204, cmd:
> > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > >> > >>>
> > > > > > >
> org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd,
> > > > > > > > > >> > >>>>>>> cmdInfo:
> > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > >> > >>>>>>>>
> > > > > > > > > >> > >>>>>
> > > > > > > > > >> >
> > > > > > > >
> > > > >
> > {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId
> > > > > > > > > >> > >>>>>>>>
> > > > > > > > > >> > >>>>>
> > > > > > > > > >> >
> > > > > > > >
> > > > >
> > ":"2615","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\"
> > > > > > > > > >> > >>>>>>>> interface
> > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > >> > >>>>>>>>
> > > > > > > > > >> > >>>>>
> > > > > > > > > >> >
> > > > > > > >
> > > > >
> > com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}"
> > > > > > > > > >> > >>>>>>>>
> > > > > > > > > >> > >>>>>
> > > > > > > > > >> >
> > > > > > > >
> > > > >
> > ,"ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmd
> > > > > > > > > >> > >>>>>>>>
> > EventType":"NETWORK.DELETE","_":"1552998473154"},
> > > > > > > > > >> > >>>>>>>>> cmdVersion: 0, status: IN_PROGRESS,
> > > processStatus:
> > > > > 0,
> > > > > > > > > >> > >>> resultCode:
> > > > > > > > > >> > >>>>> 0,
> > > > > > > > > >> > >>>>>>>>> result: null, initMsid: 264216221068220,
> > > > > completeMsid:
> > > > > > > > null,
> > > > > > > > > >> > >>>>>>> lastUpdated:
> > > > > > > > > >> > >>>>>>>>> null, lastPolled: null, created: null}
> > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,257 WARN
> > > > > > > > > [o.a.c.e.o.NetworkOrchestrator]
> > > > > > > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081
> > > > > > ctx-d5de7979)
> > > > > > > > > >> > >>>>>>>>> (logid:16897ea6) Can't delete the network,
> not
> > > all
> > > > > > user
> > > > > > > > vms
> > > > > > > > > >> are
> > > > > > > > > >> > >>>>>>>>> expunged. Vm VM[User|i-2-11-VM] is in
> Stopped
> > > > state
> > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,263 DEBUG
> > > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081)
> > > > > > > > (logid:16897ea6)
> > > > > > > > > >> > >>>>> Complete
> > > > > > > > > >> > >>>>>>>> async
> > > > > > > > > >> > >>>>>>>>> job-5081, jobStatus: FAILED, resultCode:
> 530,
> > > > > result:
> > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > >> > >>>>>>>>
> > > > > > > > > >> > >>>>>
> > > > > > > > > >> >
> > > > > > > >
> > > > >
> > org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":
> > > > > > > > > >> > >>>>>>>> [],"errorcode":530,"errortext":"Failed
> > > > > > > > > >> > >>>>>>>>> to delete network"}
> > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,264 DEBUG
> > > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081)
> > > > > > > > (logid:16897ea6)
> > > > > > > > > >> > >>>>> Publish
> > > > > > > > > >> > >>>>>>>>> async
> > > > > > > > > >> > >>>>>>>>> job-5081 complete on message bus
> > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,264 DEBUG
> > > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081)
> > > > > > > > (logid:16897ea6)
> > > > > > > > > >> > >>> Wake
> > > > > > > > > >> > >>>>> up
> > > > > > > > > >> > >>>>>>>>> jobs related to job-5081
> > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,264 DEBUG
> > > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081)
> > > > > > > > (logid:16897ea6)
> > > > > > > > > >> > >>> Update
> > > > > > > > > >> > >>>>>>>>> db status for job-5081
> > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,265 DEBUG
> > > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081)
> > > > > > > > (logid:16897ea6)
> > > > > > > > > >> > >>> Wake
> > > > > > > > > >> > >>>>> up
> > > > > > > > > >> > >>>>>>>>> jobs joined with job-5081 and disjoin all
> > > subjobs
> > > > > > > created
> > > > > > > > > from
> > > > > > > > > >> > >>>>> job-
> > > > > > > > > >> > >>>>>>>>> 5081
> > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,267 DEBUG
> > > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081)
> > > > > > > > (logid:16897ea6)
> > > > > > > > > >> > >>> Done
> > > > > > > > > >> > >>>>>>>> executing
> > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > >>
> > > > org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd
> > > > > > > > > >> > >>>>> for
> > > > > > > > > >> > >>>>>>>>> job-5081
> > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,267 INFO
> > > > > > > > [o.a.c.f.j.i.AsyncJobMonitor]
> > > > > > > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081)
> > > > > > > > (logid:16897ea6)
> > > > > > > > > >> > >>> Remove
> > > > > > > > > >> > >>>>>>>> job-5081
> > > > > > > > > >> > >>>>>>>>> from job monitoring
> > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:56,230 DEBUG
> > [c.c.a.ApiServlet]
> > > > > > > > > >> > >>>>>>>>> (qtp510113906-28:ctx-e6c5bc85)
> > (logid:7fe68f75)
> > > > > > > > ===START===
> > > > > > > > > >> > >>>>>>>> 192.168.5.140
> > > > > > > > > >> > >>>>>>>>> -- GET
> > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > >> > >>>>>>>>
> > > > > > > > > >> > >>>>>
> > > > > > > > > >> >
> > > > > > > >
> > > > >
> > command=queryAsyncJobResult&jobId=16897ea6-27c3-45b9-a0df-ab217bb5393c
> > > > > > > > > >> > >>>>>>>> &response=json&_=1552998476202
> > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:56,238 DEBUG
> > [c.c.a.ApiServer]
> > > > > > > > > >> > >>>>>>>>> (qtp510113906-28:ctx-e6c5bc85 ctx-da1f4cbd)
> > > > > > > > (logid:7fe68f75)
> > > > > > > > > >> > >>> CIDRs
> > > > > > > > > >> > >>>>>>>>> from which account
> > > > > > > > > >> > >>>>>>>>>
> > > 'Acct[15863393-8e8d-11e7-8f52-f04da2002bbe-admin]'
> > > > > is
> > > > > > > > > >> > >>>>>>>> allowed
> > > > > > > > > >> > >>>>>>>>> to perform API calls: 0.0.0.0/0,::/0
> > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:56,260 DEBUG
> > [c.c.a.ApiServlet]
> > > > > > > > > >> > >>>>>>>>> (qtp510113906-28:ctx-e6c5bc85 ctx-da1f4cbd)
> > > > > > > > (logid:7fe68f75)
> > > > > > > > > >> > >>>>>>>>> ===END===
> > > > > > > > > >> > >>>>>>>>> 192.168.5.140 -- GET
> > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > >> > >>>>>>>>
> > > > > > > > > >> > >>>>>
> > > > > > > > > >> >
> > > > > > > >
> > > > >
> > command=queryAsyncJobResult&jobId=16897ea6-27c3-45b9-a0df-ab217bb5393c
> > > > > > > > > >> > >>>>>>>> &response=json&_=1552998476202
> > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:28:00,946 INFO
> > > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > > > > >> > >>>>>>>>> (AsyncJobMgr-Heartbeat-1:ctx-9b43d1fd)
> > > > > > (logid:a605267a)
> > > > > > > > > Begin
> > > > > > > > > >> > >>>>>>>>> cleanup expired async-jobs
> > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:28:00,951 INFO
> > > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > > > > >> > >>>>>>>>> (AsyncJobMgr-Heartbeat-1:ctx-9b43d1fd)
> > > > > > (logid:a605267a)
> > > > > > > > End
> > > > > > > > > >> > >>>>> cleanup
> > > > > > > > > >> > >>>>>>>> expired
> > > > > > > > > >> > >>>>>>>>> async-jobs
> > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:28:01,142 DEBUG
> > > > > > > > > >> > >>>>>>>> [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> > > > > > > > > >> > >>>>>>>>> (RouterStatusMonitor-1:ctx-ad6bbe7e)
> > > > > (logid:04e4c72b)
> > > > > > > > Found
> > > > > > > > > 0
> > > > > > > > > >> > >>>>>>>>> routers to update status.
> > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > >> > >>>>>>>>
> > > > > > > > > >> > >>>>>>>>
> > > > > > > > > >> > >>>>>>>> --
> > > > > > > > > >> > >>>>>>>> Rafael Weingärtner
> > > > > > > > > >> > >>>>>>>>
> > > > > > > > > >> > >>>>>>>
> > > > > > > > > >> > >>>>>>
> > > > > > > > > >> > >>>>>
> > > > > > > > > >> > >>>>
> > > > > > > > > >> > >>>
> > > > > > > > > >> > >>
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
> --
>
> Andrija Panić
>
