Check network_offering table for value in column redundant_router_service for
the network offering you use.
Can you also run the following:
select name, state, removed from host where name like 'r%'
select * from domain_router;
select * from router_network_ref;
Cloudstack is supposed to recreate you VR. If it is not happening there is
something fundamentally wrong. I would advise to destroy your VR again. Stop
you management server. Rotate management server log and start it again. If your
VR doesn't start in few min, post your complete management server log and
agent log again.
On 3/19/19, 2:56 PM, "Jevgeni Zolotarjov" <j.zolotar...@gmail.com> wrote:
>>>Network offering needs to be change to non-redundant
How do I do that?
On Tue, Mar 19, 2019 at 11:47 PM Sergey Levitskiy <serg...@hotmail.com>
wrote:
> Network offering needs to be change to non-redundant. Most likely old bug
> is resurfaced.
> https://issues.apache.org/jira/browse/CLOUDSTACK-9024
>
>
>
> On 3/19/19, 2:19 PM, "Jevgeni Zolotarjov" <j.zolotar...@gmail.com> wrote:
>
> Hello
>
> I did exactly like you suggested.
>
> After UPDATE on db, I can see the router in cloudstack console.
> But attempt to restart network fails.
> I get error:
> Resource [DataCenter:1] is unreachable: Can't find all necessary
> running
> routers!
>
> I rechecked agents on both servers. They look running OK.
>
> On Tue, Mar 19, 2019 at 10:00 PM Andrija Panic <
> andrija.pa...@gmail.com>
> wrote:
>
> > Ok, so:
> >
> > 1.BACKUP YOUR DB - in case there is issues, you will want to restore
> > 2. I tried to reproduce your case on 4.11.2 (though didn't do any
> > maintenance etc.) - and could not - I have artificially marked
> existing VR
> > as removed in DB - and tried to restart network and it worked just
> fine.
> >
> > Let's try to "undelete" the VR for that network - though I can't be
> sure if
> > this will work or not.
> >
> > Find VR based on the network UUID:
> > SELECT * FROM vm_instance WHERE id IN (SELECT instance_id from
> nics
> > where network_id IN (SELECT id FROM networks WHERE
> > UUID="65ca9a05-ff96-4563-ab9c-ffb610dc8b73"));
> > Obviously, replace this UUID with your network UUID.
> > SQL above will return all VMs (user VMs, VRs, etc, delete or not...)
> that
> > have NIC inside the network with that UUID - look for the r-XX-VM
> being a
> > router.
> >
> > Now that you have the name of the VR (i.e. r-4-VM) - you want to set
> 2
> > fields to some other value than what it might be already, as below:
> >
> > Need to set field "state" to Stopped:
> > UPDATE cloud.vm_instance SET state='Stopped' WHERE
> name="r-4-VM";
> > (what state is set currently ???)
> >
> > Need to set "removed" to NULL value:
> > UPDATE cloud.vm_instance SET removed = NULL WHERE
> NAME="r-4-VM"; (is
> > there a removed date already set ???)
> >
> > Obviously make sure that this VR is not running on any hypervisor
> and if
> > running, virsh destroy it...
> >
> > AFTER above has been done (VR is considered to exist but stopped,
> from
> > CloudStack point of view) - try to RESTART the network (don't bother
> > deleteing it, since there are VMs in that network)
> >
> > I'm not 100% positive this will fix your issue, but doesn't hurt to
> try
> >
> > If above doesn't work - I would still take a look into agents and if
> they
> > are still connected - optionally, restart agents on both hosts once
> more
> > and confirm they are connected and up.
> >
> > Let us know how it goes (and please backup DB once more before any
> actions
> > !)
> >
> > Andrija
> >
> >
> > On Tue, 19 Mar 2019 at 20:41, Jevgeni Zolotarjov <
> j.zolotar...@gmail.com>
> > wrote:
> >
> > > Yes. Just a single network.
> > >
> > > On Tue, 19 Mar 2019, 21:39 Andrija Panic, <andrija.pa...@gmail.com
> >
> > wrote:
> > >
> > > > Just one more clarification - this is Isolate single network
(not
> > Shared
> > > > Network, not VPC) ?
> > > >
> > > >
> > > >
> > > > On Tue, Mar 19, 2019, 19:36 Jevgeni Zolotarjov <
> j.zolotar...@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Name defaultGuestNetwork
> > > > > ID 4ba834ed-48f3-468f-b667-9bb2d2c258f1
> > > > > Zone PBT zone 1
> > > > > Description defaultGuestNetwork
> > > > > Type Shared
> > > > > State Setup
> > > > > VPC ID N/A
> > > > > Persistent No
> > > > >
> > > > > On Tue, Mar 19, 2019 at 8:29 PM Andrija Panic <
> > andrija.pa...@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Share the network id and name as seen from GUI...
> > > > > >
> > > > > > On Tue, Mar 19, 2019, 19:27 Jevgeni Zolotarjov <
> > > j.zolotar...@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > >>> 1. Confirm please rolling restart is set to false
> please -
> > > double
> > > > > > check
> > > > > > > Double checked - It is set to false
> > > > > > >
> > > > > > > >>>>2. If so - do you know the name of VR which you
> deleted ? Is
> > it
> > > > > last
> > > > > > > one
> > > > > > > >>> ever created - if so we can find it easily...
> > > > > > > I dont know the name
> > > > > > > Is there a way to fetch it from DB?
> > > > > > >
> > > > > > > On Tue, Mar 19, 2019 at 7:58 PM Andrija Panic <
> > > > andrija.pa...@gmail.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Right...still complaining on missing running routers.
> > > > > > > >
> > > > > > > > 1. Confirm please rolling restart is set to false please
> -
> > double
> > > > > check
> > > > > > > > 2. If so - do you know the name of VR which you deleted
> ? Is it
> > > > last
> > > > > > one
> > > > > > > > ever created - if so we can find it easily...
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Mar 19, 2019, 18:40 Jevgeni Zolotarjov <
> > > > > j.zolotar...@gmail.com
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Here is management server log
> > > > > > > > >
> > > >
> https://drive.google.com/open?id=1H2jI0roeiWxtzReB8qV6QxDkNpaki99A
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Mar 19, 2019 at 7:29 PM Andrija Panic <
> > > > > > andrija.pa...@gmail.com
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > I would also try to "undelete" VR in DB, but let's
> keep
> > this
> > > as
> > > > > > last
> > > > > > > > > step.
> > > > > > > > > >
> > > > > > > > > > On Tue, Mar 19, 2019, 18:24 Andrija Panic <
> > > > > andrija.pa...@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Disclaimer: from mobile device.
> > > > > > > > > > >
> > > > > > > > > > > I don't see any special failure in agent log,
> except some
> > > > long
> > > > > > > > running
> > > > > > > > > > > migration, timeout for graceful VM shutdown etc
> (and
> > agent
> > > > > > restart)
> > > > > > > > > > >
> > > > > > > > > > > You did not send mgmt log after last network
> restart
> > > failure
> > > > ?
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Mar 19, 2019, 17:29 Jevgeni Zolotarjov <
> > > > > > > > j.zolotar...@gmail.com
> > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > >> Under
> > > > > > > > > > >> Infrastructure / Hosts:
> > > > > > > > > > >> both hosts are enabled and green (Unsecure
though)
> > > > > > > > > > >>
> > > > > > > > > > >> agent.log -
> > > > > > > > > > >>
> > > > > > >
> > https://drive.google.com/open?id=1rATxHKqgNKo2kD23BtlrZy_9gFXC-Bq-
> > > > > > > > > > >>
> > > > > > > > > > >> I set network.rolling.restart to false now.
> Restarted
> > > > > management
> > > > > > > > > server
> > > > > > > > > > -
> > > > > > > > > > >> same problem - cannot restart not delete network
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >> On Tue, Mar 19, 2019 at 5:48 PM Boris Stoyanov <
> > > > > > > > > > >> boris.stoya...@shapeblue.com>
> > > > > > > > > > >> wrote:
> > > > > > > > > > >>
> > > > > > > > > > >> > Hi, you shouldn’t be worried about data as long
> as you
> > > > use a
> > > > > > > > > separate
> > > > > > > > > > >> > shared storage.
> > > > > > > > > > >> >
> > > > > > > > > > >> > What does the cloudstack console say about your
> host?
> > Is
> > > > it
> > > > > > up?
> > > > > > > If
> > > > > > > > > > it’s
> > > > > > > > > > >> up
> > > > > > > > > > >> > then you should be able to deploy a VM on it.
> > > > > > > > > > >> >
> > > > > > > > > > >> > If not, you’ll need to investigate the reason
> > > > > cloudstack-agent
> > > > > > > is
> > > > > > > > > not
> > > > > > > > > > >> > connected, is the agent running? If not just
> start it
> > > and
> > > > > see
> > > > > > if
> > > > > > > > it
> > > > > > > > > > >> > connects.
> > > > > > > > > > >> >
> > > > > > > > > > >> > If it’s running but still not connected please
> share
> > the
> > > > > > > > agent.log.
> > > > > > > > > > >> >
> > > > > > > > > > >> > If nothing helps at all you can remove the host
> and
> > add
> > > it
> > > > > > > again,
> > > > > > > > > but
> > > > > > > > > > >> > let’s try to troubleshoot first.
> > > > > > > > > > >> >
> > > > > > > > > > >> > Bobby.
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> > boris.stoya...@shapeblue.com
> > > > > > > > > > >> > www.shapeblue.com
> > > > > > > > > > >> > Amadeus House, Floral Street, London WC2E
9DPUK
> > > > > > > > > > >> > @shapeblue
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> > > On 19 Mar 2019, at 17:29, Jevgeni Zolotarjov
<
> > > > > > > > > > j.zolotar...@gmail.com>
> > > > > > > > > > >> > wrote:
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > Guys, please help with it.
> > > > > > > > > > >> > > What can be done here?
> > > > > > > > > > >> > > There is too much valuable data.
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > On Tue, Mar 19, 2019 at 4:21 PM Jevgeni
> Zolotarjov <
> > > > > > > > > > >> > j.zolotar...@gmail.com>
> > > > > > > > > > >> > > wrote:
> > > > > > > > > > >> > >
> > > > > > > > > > >> > >> Tried that just now and got error:
> > > > > > > > > > >> > >> Resource [DataCenter:1] is unreachable:
> Can't find
> > > all
> > > > > > > > necessary
> > > > > > > > > > >> running
> > > > > > > > > > >> > >> routers!
> > > > > > > > > > >> > >>
> > > > > > > > > > >> > >> In the log I see:
> > > > > > > > > > >> > >>
=============================================
> > > > > > > > > > >> > >>
> > > > > > > > > > >> > >> 2019-03-19 14:20:39,644 DEBUG
> > > > > > [o.a.c.e.o.NetworkOrchestrator]
> > > > > > > > > > >> > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093
> > > ctx-9be30648)
> > > > > > > > > > >> (logid:265a6099)
> > > > > > > > > > >> > >> Restarting network 204...
> > > > > > > > > > >> > >> 2019-03-19 14:20:39,645 DEBUG
> > > > > > [o.a.c.e.o.NetworkOrchestrator]
> > > > > > > > > > >> > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093
> > > ctx-9be30648)
> > > > > > > > > > >> (logid:265a6099)
> > > > > > > > > > >> > >> Performing rolling restart of routers of
> network
> > > > > > > > > Ntwk[204|Guest|6]
> > > > > > > > > > >> > >> 2019-03-19 14:20:39,649 DEBUG
> > > > > > [o.a.c.e.o.NetworkOrchestrator]
> > > > > > > > > > >> > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093
> > > ctx-9be30648)
> > > > > > > > > > >> (logid:265a6099)
> > > > > > > > > > >> > >> Asking VirtualRouter to implemenet
> > Ntwk[204|Guest|6]
> > > > > > > > > > >> > >> 2019-03-19 14:20:39,658 ERROR
> > > > > [c.c.a.ApiAsyncJobDispatcher]
> > > > > > > > > > >> > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093)
> > > > > (logid:265a6099)
> > > > > > > > > > >> Unexpected
> > > > > > > > > > >> > >> exception while executing
> > > > > > > > > > >> > >>
> > > > > > > >
> > org.apache.cloudstack.api.command.user.network.RestartNetworkCmd
> > > > > > > > > > >> > >>
> com.cloud.exception.ResourceUnavailableException:
> > > > > Resource
> > > > > > > > > > >> > [DataCenter:1]
> > > > > > > > > > >> > >> is unreachable: Can't find all necessary
> running
> > > > routers!
> > > > > > > > > > >> > >> at
> > > > > > > > > > >> > >>
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:243)
> > > > > > > > > > >> > >> at
> > > > > > > > > > >> > >>
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElements(NetworkOrchestrator.java:1203)
> > > > > > > > > > >> > >> at
> > > > > > > > > > >> > >>
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.rollingRestartRouters(NetworkOrchestrator.java:2948)
> > > > > > > > > > >> > >> at
> > > > > > > > > > >> > >>
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:2853)
> > > > > > > > > > >> > >> at
> > > > > > > > > > >> > >>
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1883)
> > > > > > > > > > >> > >> at
> > > > > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> > > > > > > > > > Method)
> > > > > > > > > > >> > >> at
> > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(Unknown
> > > > > > > > > > Source)
> > > > > > > > > > >> > >> at
> > > > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
> > > > > > > > > > >> > Source)
> > > > > > > > > > >> > >> at
> java.lang.reflect.Method.invoke(Unknown
> > > > Source)
> > > > > > > > > > >> > >> at
> > > > > > > > > > >> > >>
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
> > > > > > > > > > >> > >> at
> > > > > > > > > > >> > >>
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
> > > > > > > > > > >> > >> at
> > > > > > > > > > >> > >>
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> > > > > > > > > > >> > >> at
> > > > > > > > > > >> > >>
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
> > > > > > > > > > >> > >> at
> > > > > > > > > > >> > >>
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> > > > > > > > > > >> > >> at
> > > > > > > > > > >> > >>
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
> > > > > > > > > > >> > >> at
> > > > > > > > > > >> > >>
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> > > > > > > > > > >> > >> at
> > > > > > > > > > >> > >>
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
> > > > > > > > > > >> > >> at
> > > > > > > > > > >> > >>
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
> > > > > > > > > > >> > >> at
> > > > > > > > > > >> > >>
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
> > > > > > > > > > >> > >> at
> > > > com.sun.proxy.$Proxy229.restartNetwork(Unknown
> > > > > > > > Source)
> > > > > > > > > > >> > >> at
> > > > > > > > > > >> > >>
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:99)
> > > > > > > > > > >> > >> at
> > > > > > > > > > >>
> > > com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
> > > > > > > > > > >> > >> at
> > > > > > > > > > >> > >>
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
> > > > > > > > > > >> > >> at
> > > > > > > > > > >> > >>
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
> > > > > > > > > > >> > >> at
> > > > > > > > > > >> > >>
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> > > > > > > > > > >> > >> at
> > > > > > > > > > >> > >>
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> > > > > > > > > > >> > >> at
> > > > > > > > > > >> > >>
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> > > > > > > > > > >> > >> at
> > > > > > > > > > >> > >>
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> > > > > > > > > > >> > >> at
> > > > > > > > > > >> > >>
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> > > > > > > > > > >> > >> at
> > > > > > > > > > >> > >>
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
> > > > > > > > > > >> > >> at
> > > > > > > > > >
> java.util.concurrent.Executors$RunnableAdapter.call(Unknown
> > > > > > > > > > >> > >> Source)
> > > > > > > > > > >> > >> at
> > java.util.concurrent.FutureTask.run(Unknown
> > > > > > Source)
> > > > > > > > > > >> > >> at
> > > > > > > > >
> java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> > > > > > > > > > >> > >> Source)
> > > > > > > > > > >> > >> at
> > > > > > > > > >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> > > > > > > > > > >> > >> Source)
> > > > > > > > > > >> > >> at java.lang.Thread.run(Unknown
> Source)
> > > > > > > > > > >> > >> 2019-03-19 14:20:39,659 DEBUG
> > > > > > > [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > > > > > >> > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093)
> > > > > (logid:265a6099)
> > > > > > > > > > Complete
> > > > > > > > > > >> > async
> > > > > > > > > > >> > >> job-5093, jobStatus: FAILED, resultCode:
530,
> > result:
> > > > > > > > > > >> > >>
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Resource
> > > > > > > > > > >> > >> [DataCenter:1] is un
> > > > > > > > > > >> > >>
> > > > > > > > > > >> > >> On Tue, Mar 19, 2019 at 4:19 PM Andrija
> Panic <
> > > > > > > > > > >> > andrija.pa...@shapeblue.com>
> > > > > > > > > > >> > >> wrote:
> > > > > > > > > > >> > >>
> > > > > > > > > > >> > >>>
> > > > > > > > > > >> > >>> Your network can't be deleted due to "Can't
> delete
> > > the
> > > > > > > > network,
> > > > > > > > > > not
> > > > > > > > > > >> all
> > > > > > > > > > >> > >>> user vms are expunged. Vm
> > > > > > > > > > >> > >>> VM[User|i-2-11-VM] is in Stopped state" -
> which is
> > > > fine.
> > > > > > > > > > >> > >>>
> > > > > > > > > > >> > >>> You should be able to just start the user
> VM - but
> > > if
> > > > > you
> > > > > > > have
> > > > > > > > > > >> actually
> > > > > > > > > > >> > >>> delete the VR itself, then just do Network
> restart
> > > > with
> > > > > > > > > "cleanup"
> > > > > > > > > > >> and
> > > > > > > > > > >> > it
> > > > > > > > > > >> > >>> will recreate a new VR, after which you
> should be
> > > able
> > > > > to
> > > > > > > > start
> > > > > > > > > > the
> > > > > > > > > > >> VM.
> > > > > > > > > > >> > >>>
> > > > > > > > > > >> > >>> Andrija
> > > > > > > > > > >> > >>>
> > > > > > > > > > >> > >>> andrija.pa...@shapeblue.com
> > > > > > > > > > >> > >>> www.shapeblue.com
> > > > > > > > > > >> > >>> Amadeus House, Floral Street, London WC2E
> 9DPUK
> > > > > > > > > > >> > >>> @shapeblue
> > > > > > > > > > >> > >>>
> > > > > > > > > > >> > >>>
> > > > > > > > > > >> > >>>
> > > > > > > > > > >> > >>>
> > > > > > > > > > >> > >>> -----Original Message-----
> > > > > > > > > > >> > >>> From: Jevgeni Zolotarjov <
> j.zolotar...@gmail.com>
> > > > > > > > > > >> > >>> Sent: 19 March 2019 15:10
> > > > > > > > > > >> > >>> To: users@cloudstack.apache.org
> > > > > > > > > > >> > >>> Subject: Re: Disaster after maintenance
> > > > > > > > > > >> > >>>
> > > > > > > > > > >> > >>> I mean I cannot delete network: In the
> management
> > > > server
> > > > > > > log I
> > > > > > > > > see
> > > > > > > > > > >> > >>>
> > > > > > > > > > >> > >>>
> > > > > ==========================================================
> > > > > > > > > > >> > >>> 019-03-19 14:06:36,316 DEBUG
> > > > > > > [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > > > > > >> > >>> (API-Job-Executor-1:ctx-1c0fd4dc job-5090)
> > > > > > (logid:c734edfc)
> > > > > > > > > > >> Executing
> > > > > > > > > > >> > >>> AsyncJobVO {id:5090, userId: 2, accountId:
> 2,
> > > > > > instanceType:
> > > > > > > > > > Network,
> > > > > > > > > > >> > >>> instanceId: 204, cmd:
> > > > > > > > > > >> > >>>
> > > > > > > >
> > org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd,
> > > > > > > > > > >> > cmdInfo:
> > > > > > > > > > >> > >>>
> > > > > > > > > > >> > >>>
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
{"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"2641","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\"interface
> > > > > > > > > > >> > >>>
> > > > > > > > > > >> > >>>
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}","ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmdEventType":"NETWORK.DELETE","_":"1553004396247"},
> > > > > > > > > > >> > >>> cmdVersion: 0, status: IN_PROGRESS,
> processStatus:
> > > 0,
> > > > > > > > > resultCode:
> > > > > > > > > > 0,
> > > > > > > > > > >> > >>> result: null, initMsid: 264216221068220,
> > > completeMsid:
> > > > > > null,
> > > > > > > > > > >> > lastUpdated:
> > > > > > > > > > >> > >>> null, lastPolled: null, created: null}
> > > > > > > > > > >> > >>> 2019-03-19 14:06:36,351 WARN
> > > > > > > [o.a.c.e.o.NetworkOrchestrator]
> > > > > > > > > > >> > >>> (API-Job-Executor-1:ctx-1c0fd4dc job-5090
> > > > ctx-134954fa)
> > > > > > > > > > >> > (logid:c734edfc)
> > > > > > > > > > >> > >>> Can't delete the network, not all user vms
> are
> > > > expunged.
> > > > > > Vm
> > > > > > > > > > >> > >>> VM[User|i-2-11-VM] is in Stopped state
> > > > > > > > > > >> > >>> 2019-03-19 14:06:36,356 DEBUG
> > > > > > > > [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > > > > > >> > >>> (API-Job-Executor-1:ctx-1c0fd4dc job-5090)
> > > > > > (logid:c734edfc)
> > > > > > > > > > Complete
> > > > > > > > > > >> > async
> > > > > > > > > > >> > >>> job-5090, jobStatus: FAILED, resultCode:
> 530,
> > > result:
> > > > > > > > > > >> > >>>
> > > > > > > > > > >> > >>>
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
> > > > > > > > > > >> > >>> to delete network"}
> > > > > > > > > > >> > >>>
> > > > > ==========================================================
> > > > > > > > > > >> > >>>
> > > > > > > > > > >> > >>>
> > > > > > > > > > >> > >>> I deleted a router, expecting it to be
> recreated
> > on
> > > > > > deleting
> > > > > > > > > > >> network.
> > > > > > > > > > >> > But
> > > > > > > > > > >> > >>> I am unable to delete network because of
> above
> > error
> > > > > > > > > > >> > >>>
> > > > > > > > > > >> > >>> On Tue, Mar 19, 2019 at 3:58 PM Jevgeni
> > Zolotarjov <
> > > > > > > > > > >> > >>> j.zolotar...@gmail.com>
> > > > > > > > > > >> > >>> wrote:
> > > > > > > > > > >> > >>>
> > > > > > > > > > >> > >>>> I've managed to make libvirtd running
> > > > > > > > > > >> > >>>> Now cloudstack console shows both hosts -
> running
> > > > > > > > > > >> > >>>>
> > > > > > > > > > >> > >>>> But now as I have removed network, VMs are
> unable
> > > to
> > > > > > start.
> > > > > > > > > > >> > >>>>
> > > > > > > > > > >> > >>>> How can I recreate the network now?
> > > > > > > > > > >> > >>>>
> > > > > > > > > > >> > >>>> On Tue, Mar 19, 2019 at 3:14 PM Ivan
> Kudryavtsev
> > > > > > > > > > >> > >>>> <kudryavtsev...@bw-sw.com>
> > > > > > > > > > >> > >>>> wrote:
> > > > > > > > > > >> > >>>>
> > > > > > > > > > >> > >>>>> Jevgeniy, it may be a documentation bug.
> Take s
> > > > look:
> > > > > > > > > > >> > >>>>>
> > > > > > > > > >
> > > > https://github.com/apache/cloudstack-documentation/pull/27/files
> > > > > > > > > > >> > >>>>>
> > > > > > > > > > >> > >>>>> вт, 19 мар. 2019 г., 9:09 Jevgeni
> Zolotarjov <
> > > > > > > > > > >> j.zolotar...@gmail.com
> > > > > > > > > > >> > >:
> > > > > > > > > > >> > >>>>>
> > > > > > > > > > >> > >>>>>> That's it - libvirtd failed to start on
> second
> > > > host.
> > > > > > > > > > >> > >>>>>> Tried restarting, but it does not start.
> > > > > > > > > > >> > >>>>>>
> > > > > > > > > > >> > >>>>>>
> > > > > > > > > > >> > >>>>>>>> Do you have some NUMA constraints or
> anything
> > > > which
> > > > > > > > > requires
> > > > > > > > > > >> > >>>>> particular
> > > > > > > > > > >> > >>>>>> RAM configuration?
> > > > > > > > > > >> > >>>>>> No
> > > > > > > > > > >> > >>>>>>
> > > > > > > > > > >> > >>>>>> libvirtd.service - Virtualization daemon
> > > > > > > > > > >> > >>>>>> Loaded: loaded
> > > > > > > (/usr/lib/systemd/system/libvirtd.service;
> > > > > > > > > > >> > >>>>>> enabled; vendor preset: enabled)
> > > > > > > > > > >> > >>>>>> Active: failed (Result: start-limit)
> since
> > Tue
> > > > > > > 2019-03-19
> > > > > > > > > > >> > >>>>>> 13:03:07
> > > > > > > > > > >> > >>>>> GMT;
> > > > > > > > > > >> > >>>>>> 12s ago
> > > > > > > > > > >> > >>>>>> Docs: man:libvirtd(8)
> > > > > > > > > > >> > >>>>>> https://libvirt.org
> > > > > > > > > > >> > >>>>>> Process: 892
> ExecStart=/usr/sbin/libvirtd
> > > > > > $LIBVIRTD_ARGS
> > > > > > > > > > >> > >>>>>> (code=exited,
> > > > > > > > > > >> > >>>>>> status=1/FAILURE)
> > > > > > > > > > >> > >>>>>> Main PID: 892 (code=exited,
> status=1/FAILURE)
> > > > > > > > > > >> > >>>>>> Tasks: 19 (limit: 32768)
> > > > > > > > > > >> > >>>>>> CGroup: /system.slice/libvirtd.service
> > > > > > > > > > >> > >>>>>> ├─11338 /usr/sbin/libvirtd -d
> -l
> > > > > > > > > > >> > >>>>>> ├─11909 /usr/sbin/dnsmasq
> > > > > > > > > > >> > >>>>>>
> > --conf-file=/var/lib/libvirt/dnsmasq/default.conf
> > > > > > > > > > --leasefile-ro
> > > > > > > > > > >> > >>>>>>
> --dhcp-script=/usr/libexec/libvirt_leaseshelper
> > > > > > > > > > >> > >>>>>> └─11910 /usr/sbin/dnsmasq
> > > > > > > > > > >> > >>>>>>
> > --conf-file=/var/lib/libvirt/dnsmasq/default.conf
> > > > > > > > > > --leasefile-ro
> > > > > > > > > > >> > >>>>>>
> --dhcp-script=/usr/libexec/libvirt_leaseshelper
> > > > > > > > > > >> > >>>>>>
> > > > > > > > > > >> > >>>>>> Mar 19 13:03:07
> mtl1-apphst04.mt.pbt.com.mt
> > > > > > systemd[1]:
> > > > > > > > > Failed
> > > > > > > > > > >> to
> > > > > > > > > > >> > >>>>>> start Virtualization daemon.
> > > > > > > > > > >> > >>>>>> Mar 19 13:03:07
> mtl1-apphst04.mt.pbt.com.mt
> > > > > > systemd[1]:
> > > > > > > > Unit
> > > > > > > > > > >> > >>>>>> libvirtd.service entered failed state.
> > > > > > > > > > >> > >>>>>> Mar 19 13:03:07
> mtl1-apphst04.mt.pbt.com.mt
> > > > > > systemd[1]:
> > > > > > > > > > >> > >>>>> libvirtd.service
> > > > > > > > > > >> > >>>>>> failed.
> > > > > > > > > > >> > >>>>>> Mar 19 13:03:07
> mtl1-apphst04.mt.pbt.com.mt
> > > > > > systemd[1]:
> > > > > > > > > > >> > >>>>> libvirtd.service
> > > > > > > > > > >> > >>>>>> holdoff time over, scheduling restart.
> > > > > > > > > > >> > >>>>>> Mar 19 13:03:07
> mtl1-apphst04.mt.pbt.com.mt
> > > > > > systemd[1]:
> > > > > > > > > > Stopped
> > > > > > > > > > >> > >>>>>> Virtualization daemon.
> > > > > > > > > > >> > >>>>>> Mar 19 13:03:07
> mtl1-apphst04.mt.pbt.com.mt
> > > > > > systemd[1]:
> > > > > > > > > start
> > > > > > > > > > >> > >>>>>> request repeated too quickly for
> > libvirtd.service
> > > > Mar
> > > > > > 19
> > > > > > > > > > 13:03:07
> > > > > > > > > > >> > >>>>>> mtl1-apphst04.mt.pbt.com.mt systemd[1]:
> Failed
> > > to
> > > > > > start
> > > > > > > > > > >> > >>>>>> Virtualization daemon.
> > > > > > > > > > >> > >>>>>> Mar 19 13:03:07
> mtl1-apphst04.mt.pbt.com.mt
> > > > > > systemd[1]:
> > > > > > > > Unit
> > > > > > > > > > >> > >>>>>> libvirtd.service entered failed state.
> > > > > > > > > > >> > >>>>>> Mar 19 13:03:07
> mtl1-apphst04.mt.pbt.com.mt
> > > > > > systemd[1]:
> > > > > > > > > > >> > >>>>> libvirtd.service
> > > > > > > > > > >> > >>>>>> failed.
> > > > > > > > > > >> > >>>>>>
> > > > > > > > > > >> > >>>>>>
> > > > > > > > > > >> > >>>>>> On Tue, Mar 19, 2019 at 3:04 PM Paul
> Angus
> > > > > > > > > > >> > >>>>>> <paul.an...@shapeblue.com>
> > > > > > > > > > >> > >>>>>> wrote:
> > > > > > > > > > >> > >>>>>>
> > > > > > > > > > >> > >>>>>>> Can you check that the cloudstack agent
> is
> > > running
> > > > > on
> > > > > > > the
> > > > > > > > > host
> > > > > > > > > > >> > >>>>>>> and the agent logs (usual logs
> directory) Also
> > > > worth
> > > > > > > > > checking
> > > > > > > > > > >> > >>>>>>> that libvirt has started ok. Do you
> have some
> > > > > > > > > > >> > >>>>> NUMA
> > > > > > > > > > >> > >>>>>>> constraints or anything which requires
> > > particular
> > > > > RAM
> > > > > > > > > > >> > >>> configuration?
> > > > > > > > > > >> > >>>>>>>
> > > > > > > > > > >> > >>>>>>> paul.an...@shapeblue.com
> > > > > > > > > > >> > >>>>>>> www.shapeblue.com
> > > > > > > > > > >> > >>>>>>> Amadeus House, Floral Street, London
> WC2E
> > 9DPUK
> > > > > > > > @shapeblue
> > > > > > > > > > >> > >>>>>>>
> > > > > > > > > > >> > >>>>>>>
> > > > > > > > > > >> > >>>>>>>
> > > > > > > > > > >> > >>>>>>>
> > > > > > > > > > >> > >>>>>>> -----Original Message-----
> > > > > > > > > > >> > >>>>>>> From: Jevgeni Zolotarjov <
> > > j.zolotar...@gmail.com>
> > > > > > > > > > >> > >>>>>>> Sent: 19 March 2019 14:49
> > > > > > > > > > >> > >>>>>>> To: users@cloudstack.apache.org
> > > > > > > > > > >> > >>>>>>> Subject: Re: Disaster after maintenance
> > > > > > > > > > >> > >>>>>>>
> > > > > > > > > > >> > >>>>>>> Can you try migrating a VM to the
> server that
> > > you
> > > > > > > changed
> > > > > > > > > the
> > > > > > > > > > >> RAM
> > > > > > > > > > >> > >>>>> amount?
> > > > > > > > > > >> > >>>>>>>
> > > > > > > > > > >> > >>>>>>> Also:
> > > > > > > > > > >> > >>>>>>> What is the hypervisor version?
> > > > > > > > > > >> > >>>>>>> KVM
> > > > > > > > > > >> > >>>>>>> QEMU Version : 2.0.0
> > > > > > > > > > >> > >>>>>>> Release : 1.el7.6
> > > > > > > > > > >> > >>>>>>>
> > > > > > > > > > >> > >>>>>>>
> > > > > > > > > > >> > >>>>>>> Host status in ACS?
> > > > > > > > > > >> > >>>>>>> 1st server: Unsecure
> > > > > > > > > > >> > >>>>>>> 2nd server: Disconnected
> > > > > > > > > > >> > >>>>>>>
> > > > > > > > > > >> > >>>>>>> Did you try to force a VM to
> start/deploy in
> > > this
> > > > > > server
> > > > > > > > > where
> > > > > > > > > > >> you
> > > > > > > > > > >> > >>>>>> changed
> > > > > > > > > > >> > >>>>>>> the RAM?
> > > > > > > > > > >> > >>>>>>> Host status became disconnected. I
> don't know
> > > how
> > > > to
> > > > > > > make
> > > > > > > > it
> > > > > > > > > > >> > >>>>> "connected"
> > > > > > > > > > >> > >>>>>>> again
> > > > > > > > > > >> > >>>>>>>
> > > > > > > > > > >> > >>>>>>>
> > > > > > > > > > >> > >>>>>>>
> > > > > > > > > > >> > >>>>>>> On Tue, Mar 19, 2019 at 2:42 PM Rafael
> > > > Weingärtner <
> > > > > > > > > > >> > >>>>>>> rafaelweingart...@gmail.com> wrote:
> > > > > > > > > > >> > >>>>>>>
> > > > > > > > > > >> > >>>>>>>> Can you try migrating a VM to the
> server that
> > > you
> > > > > > > changed
> > > > > > > > > the
> > > > > > > > > > >> RAM
> > > > > > > > > > >> > >>>>>> amount?
> > > > > > > > > > >> > >>>>>>>>
> > > > > > > > > > >> > >>>>>>>> Also:
> > > > > > > > > > >> > >>>>>>>> What is the hypervisor version?
> > > > > > > > > > >> > >>>>>>>> Host status in ACS?
> > > > > > > > > > >> > >>>>>>>> Did you try to force a VM to
> start/deploy in
> > > this
> > > > > > > server
> > > > > > > > > > where
> > > > > > > > > > >> > >>> you
> > > > > > > > > > >> > >>>>>>>> changed the RAM?
> > > > > > > > > > >> > >>>>>>>>
> > > > > > > > > > >> > >>>>>>>>
> > > > > > > > > > >> > >>>>>>>> On Tue, Mar 19, 2019 at 9:39 AM
Jevgeni
> > > > Zolotarjov
> > > > > > > > > > >> > >>>>>>>> <j.zolotar...@gmail.com
> > > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > > >> > >>>>>>>> wrote:
> > > > > > > > > > >> > >>>>>>>>
> > > > > > > > > > >> > >>>>>>>>> We have Cloudstack 4.11.2 setup
> running fine
> > > for
> > > > > few
> > > > > > > > > months
> > > > > > > > > > >> > >>> (>4)
> > > > > > > > > > >> > >>>>> The
> > > > > > > > > > >> > >>>>>>>>> setup is very simple: 2 hosts We
> decided to
> > > do a
> > > > > > > > > maintenance
> > > > > > > > > > >> to
> > > > > > > > > > >> > >>>>>>>>> increase RAM on both servers
> > > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > > >> > >>>>>>>>> For this we put first server to
> maintenance.
> > > All
> > > > > VMS
> > > > > > > > moved
> > > > > > > > > > to
> > > > > > > > > > >> > >>>>> second
> > > > > > > > > > >> > >>>>>>>>> host after a while.
> > > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > > >> > >>>>>>>>> Then first server was shutdown, RAM
> > increased,
> > > > > > server
> > > > > > > > > turned
> > > > > > > > > > >> > >>> ON.
> > > > > > > > > > >> > >>>>>>>>> Now nothing starts on first server.
> > > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > > >> > >>>>>>>>> Tried to delete network, but this
> fails as
> > > well
> > > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > > >> > >>>>>>>>> Please help !
> > > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > > >> > >>>>>>>>> Here is extract from log:
> > > > > > > > > > >> > >>>>>>>>>
======================================
> > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,064 DEBUG
> > > > > > > > > > >> > >>>>> [o.a.c.s.SecondaryStorageManagerImpl]
> > > > > > > > > > >> > >>>>>>>>> (secstorage-1:ctx-16d6c797)
> (logid:7e3160ce)
> > > > Zone
> > > > > 1
> > > > > > is
> > > > > > > > > ready
> > > > > > > > > > >> to
> > > > > > > > > > >> > >>>>>>>>> launch secondary storage VM
> > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,125 DEBUG
> > > > > > > > > > [c.c.c.ConsoleProxyManagerImpl]
> > > > > > > > > > >> > >>>>>>>>> (consoleproxy-1:ctx-cbd034b9)
> > (logid:0a8c8bf4)
> > > > > Zone
> > > > > > 1
> > > > > > > is
> > > > > > > > > > ready
> > > > > > > > > > >> > >>> to
> > > > > > > > > > >> > >>>>>>>>> launch console proxy
> > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,181 DEBUG
> > > [c.c.a.ApiServlet]
> > > > > > > > > > >> > >>>>>>>>> (qtp510113906-285:ctx-6c5e11c3)
> > > (logid:cd8e30be)
> > > > > > > > > ===START===
> > > > > > > > > > >> > >>>>>>>> 192.168.5.140
> > > > > > > > > > >> > >>>>>>>>> -- GET
> > > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > > >> > >>>>>>>>
> > > > > > > > > > >> > >>>>>
> > > > > > > > > > >> >
> > > > > > > > >
> > > > > >
> > >
> command=deleteNetwork&id=4ba834ed-48f3-468f-b667-9bb2d2c258f1&response
> > > > > > > > > > >> > >>>>>>>> =json&_=1552998473154
> > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,186 DEBUG
> > > [c.c.a.ApiServer]
> > > > > > > > > > >> > >>>>>>>>> (qtp510113906-285:ctx-6c5e11c3
> ctx-0cc34dc6)
> > > > > > > > > > (logid:cd8e30be)
> > > > > > > > > > >> > >>>>> CIDRs
> > > > > > > > > > >> > >>>>>>>>> from which account
> > > > > > > > > > >> > >>>>>>>>>
> > > > 'Acct[15863393-8e8d-11e7-8f52-f04da2002bbe-admin]'
> > > > > > is
> > > > > > > > > > >> > >>>>>>>> allowed
> > > > > > > > > > >> > >>>>>>>>> to perform API calls: 0.0.0.0/0,::/0
> > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,208 INFO
> > > > > > > > > [o.a.c.f.j.i.AsyncJobMonitor]
> > > > > > > > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19
> job-5081)
> > > > > > > > > (logid:f6751fa7)
> > > > > > > > > > >> Add
> > > > > > > > > > >> > >>>>>>>>> job-5081 into job monitoring
> > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,209 DEBUG
> > > > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > > > > > >> > >>>>>>>>> (qtp510113906-285:ctx-6c5e11c3
> ctx-0cc34dc6)
> > > > > > > > > > (logid:cd8e30be)
> > > > > > > > > > >> > >>>>> submit
> > > > > > > > > > >> > >>>>>>>> async
> > > > > > > > > > >> > >>>>>>>>> job-5081, details: AsyncJobVO
> {id:5081,
> > > userId:
> > > > 2,
> > > > > > > > > > accountId:
> > > > > > > > > > >> > >>> 2,
> > > > > > > > > > >> > >>>>>>>>> instanceType: Network, instanceId:
> 204, cmd:
> > > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > > >> > >>>
> > > > > > > >
> > org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd,
> > > > > > > > > > >> > >>>>>>> cmdInfo:
> > > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > > >> > >>>>>>>>
> > > > > > > > > > >> > >>>>>
> > > > > > > > > > >> >
> > > > > > > > >
> > > > > >
> > >
> {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId
> > > > > > > > > > >> > >>>>>>>>
> > > > > > > > > > >> > >>>>>
> > > > > > > > > > >> >
> > > > > > > > >
> > > > > >
> > >
> ":"2615","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\"
> > > > > > > > > > >> > >>>>>>>> interface
> > > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > > >> > >>>>>>>>
> > > > > > > > > > >> > >>>>>
> > > > > > > > > > >> >
> > > > > > > > >
> > > > > >
> > >
> com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}"
> > > > > > > > > > >> > >>>>>>>>
> > > > > > > > > > >> > >>>>>
> > > > > > > > > > >> >
> > > > > > > > >
> > > > > >
> > >
> ,"ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmd
> > > > > > > > > > >> > >>>>>>>>
> > > EventType":"NETWORK.DELETE","_":"1552998473154"},
> > > > > > > > > > >> > >>>>>>>>> cmdVersion: 0, status: IN_PROGRESS,
> > > > processStatus:
> > > > > > 0,
> > > > > > > > > > >> > >>> resultCode:
> > > > > > > > > > >> > >>>>> 0,
> > > > > > > > > > >> > >>>>>>>>> result: null, initMsid:
> 264216221068220,
> > > > > > completeMsid:
> > > > > > > > > null,
> > > > > > > > > > >> > >>>>>>> lastUpdated:
> > > > > > > > > > >> > >>>>>>>>> null, lastPolled: null, created:
null}
> > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,211 DEBUG
> > > [c.c.a.ApiServlet]
> > > > > > > > > > >> > >>>>>>>>> (qtp510113906-285:ctx-6c5e11c3
> ctx-0cc34dc6)
> > > > > > > > > > (logid:cd8e30be)
> > > > > > > > > > >> > >>>>>>>>> ===END===
> > > > > > > > > > >> > >>>>>>>>> 192.168.5.140 -- GET
> > > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > > >> > >>>>>>>>
> > > > > > > > > > >> > >>>>>
> > > > > > > > > > >> >
> > > > > > > > >
> > > > > >
> > >
> command=deleteNetwork&id=4ba834ed-48f3-468f-b667-9bb2d2c258f1&response
> > > > > > > > > > >> > >>>>>>>> =json&_=1552998473154
> > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,212 DEBUG
> > > > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > > > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19
> job-5081)
> > > > > > > > > (logid:16897ea6)
> > > > > > > > > > >> > >>>>>>>>> Executing AsyncJobVO {id:5081,
> userId: 2,
> > > > > accountId:
> > > > > > > 2,
> > > > > > > > > > >> > >>>>>>>>> instanceType: Network,
> > > > > > > > > > >> > >>>>>>>>> instanceId: 204, cmd:
> > > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > > >> > >>>
> > > > > > > >
> > org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd,
> > > > > > > > > > >> > >>>>>>> cmdInfo:
> > > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > > >> > >>>>>>>>
> > > > > > > > > > >> > >>>>>
> > > > > > > > > > >> >
> > > > > > > > >
> > > > > >
> > >
> {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId
> > > > > > > > > > >> > >>>>>>>>
> > > > > > > > > > >> > >>>>>
> > > > > > > > > > >> >
> > > > > > > > >
> > > > > >
> > >
> ":"2615","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\"
> > > > > > > > > > >> > >>>>>>>> interface
> > > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > > >> > >>>>>>>>
> > > > > > > > > > >> > >>>>>
> > > > > > > > > > >> >
> > > > > > > > >
> > > > > >
> > >
> com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}"
> > > > > > > > > > >> > >>>>>>>>
> > > > > > > > > > >> > >>>>>
> > > > > > > > > > >> >
> > > > > > > > >
> > > > > >
> > >
> ,"ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmd
> > > > > > > > > > >> > >>>>>>>>
> > > EventType":"NETWORK.DELETE","_":"1552998473154"},
> > > > > > > > > > >> > >>>>>>>>> cmdVersion: 0, status: IN_PROGRESS,
> > > > processStatus:
> > > > > > 0,
> > > > > > > > > > >> > >>> resultCode:
> > > > > > > > > > >> > >>>>> 0,
> > > > > > > > > > >> > >>>>>>>>> result: null, initMsid:
> 264216221068220,
> > > > > > completeMsid:
> > > > > > > > > null,
> > > > > > > > > > >> > >>>>>>> lastUpdated:
> > > > > > > > > > >> > >>>>>>>>> null, lastPolled: null, created:
null}
> > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,257 WARN
> > > > > > > > > > [o.a.c.e.o.NetworkOrchestrator]
> > > > > > > > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19
> job-5081
> > > > > > > ctx-d5de7979)
> > > > > > > > > > >> > >>>>>>>>> (logid:16897ea6) Can't delete the
> network,
> > not
> > > > all
> > > > > > > user
> > > > > > > > > vms
> > > > > > > > > > >> are
> > > > > > > > > > >> > >>>>>>>>> expunged. Vm VM[User|i-2-11-VM] is in
> > Stopped
> > > > > state
> > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,263 DEBUG
> > > > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > > > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19
> job-5081)
> > > > > > > > > (logid:16897ea6)
> > > > > > > > > > >> > >>>>> Complete
> > > > > > > > > > >> > >>>>>>>> async
> > > > > > > > > > >> > >>>>>>>>> job-5081, jobStatus: FAILED,
> resultCode:
> > 530,
> > > > > > result:
> > > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > > >> > >>>>>>>>
> > > > > > > > > > >> > >>>>>
> > > > > > > > > > >> >
> > > > > > > > >
> > > > > >
> > >
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":
> > > > > > > > > > >> > >>>>>>>> [],"errorcode":530,"errortext":"Failed
> > > > > > > > > > >> > >>>>>>>>> to delete network"}
> > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,264 DEBUG
> > > > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > > > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19
> job-5081)
> > > > > > > > > (logid:16897ea6)
> > > > > > > > > > >> > >>>>> Publish
> > > > > > > > > > >> > >>>>>>>>> async
> > > > > > > > > > >> > >>>>>>>>> job-5081 complete on message bus
> > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,264 DEBUG
> > > > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > > > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19
> job-5081)
> > > > > > > > > (logid:16897ea6)
> > > > > > > > > > >> > >>> Wake
> > > > > > > > > > >> > >>>>> up
> > > > > > > > > > >> > >>>>>>>>> jobs related to job-5081
> > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,264 DEBUG
> > > > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > > > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19
> job-5081)
> > > > > > > > > (logid:16897ea6)
> > > > > > > > > > >> > >>> Update
> > > > > > > > > > >> > >>>>>>>>> db status for job-5081
> > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,265 DEBUG
> > > > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > > > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19
> job-5081)
> > > > > > > > > (logid:16897ea6)
> > > > > > > > > > >> > >>> Wake
> > > > > > > > > > >> > >>>>> up
> > > > > > > > > > >> > >>>>>>>>> jobs joined with job-5081 and disjoin
> all
> > > > subjobs
> > > > > > > > created
> > > > > > > > > > from
> > > > > > > > > > >> > >>>>> job-
> > > > > > > > > > >> > >>>>>>>>> 5081
> > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,267 DEBUG
> > > > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > > > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19
> job-5081)
> > > > > > > > > (logid:16897ea6)
> > > > > > > > > > >> > >>> Done
> > > > > > > > > > >> > >>>>>>>> executing
> > > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > > >>
> > > > >
org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd
> > > > > > > > > > >> > >>>>> for
> > > > > > > > > > >> > >>>>>>>>> job-5081
> > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,267 INFO
> > > > > > > > > [o.a.c.f.j.i.AsyncJobMonitor]
> > > > > > > > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19
> job-5081)
> > > > > > > > > (logid:16897ea6)
> > > > > > > > > > >> > >>> Remove
> > > > > > > > > > >> > >>>>>>>> job-5081
> > > > > > > > > > >> > >>>>>>>>> from job monitoring
> > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:56,230 DEBUG
> > > [c.c.a.ApiServlet]
> > > > > > > > > > >> > >>>>>>>>> (qtp510113906-28:ctx-e6c5bc85)
> > > (logid:7fe68f75)
> > > > > > > > > ===START===
> > > > > > > > > > >> > >>>>>>>> 192.168.5.140
> > > > > > > > > > >> > >>>>>>>>> -- GET
> > > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > > >> > >>>>>>>>
> > > > > > > > > > >> > >>>>>
> > > > > > > > > > >> >
> > > > > > > > >
> > > > > >
> > >
> command=queryAsyncJobResult&jobId=16897ea6-27c3-45b9-a0df-ab217bb5393c
> > > > > > > > > > >> > >>>>>>>> &response=json&_=1552998476202
> > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:56,238 DEBUG
> > > [c.c.a.ApiServer]
> > > > > > > > > > >> > >>>>>>>>> (qtp510113906-28:ctx-e6c5bc85
> ctx-da1f4cbd)
> > > > > > > > > (logid:7fe68f75)
> > > > > > > > > > >> > >>> CIDRs
> > > > > > > > > > >> > >>>>>>>>> from which account
> > > > > > > > > > >> > >>>>>>>>>
> > > > 'Acct[15863393-8e8d-11e7-8f52-f04da2002bbe-admin]'
> > > > > > is
> > > > > > > > > > >> > >>>>>>>> allowed
> > > > > > > > > > >> > >>>>>>>>> to perform API calls: 0.0.0.0/0,::/0
> > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:56,260 DEBUG
> > > [c.c.a.ApiServlet]
> > > > > > > > > > >> > >>>>>>>>> (qtp510113906-28:ctx-e6c5bc85
> ctx-da1f4cbd)
> > > > > > > > > (logid:7fe68f75)
> > > > > > > > > > >> > >>>>>>>>> ===END===
> > > > > > > > > > >> > >>>>>>>>> 192.168.5.140 -- GET
> > > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > > >> > >>>>>>>>
> > > > > > > > > > >> > >>>>>
> > > > > > > > > > >> >
> > > > > > > > >
> > > > > >
> > >
> command=queryAsyncJobResult&jobId=16897ea6-27c3-45b9-a0df-ab217bb5393c
> > > > > > > > > > >> > >>>>>>>> &response=json&_=1552998476202
> > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:28:00,946 INFO
> > > > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > > > > > >> > >>>>>>>>>
(AsyncJobMgr-Heartbeat-1:ctx-9b43d1fd)
> > > > > > > (logid:a605267a)
> > > > > > > > > > Begin
> > > > > > > > > > >> > >>>>>>>>> cleanup expired async-jobs
> > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:28:00,951 INFO
> > > > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > > > > > > > >> > >>>>>>>>>
(AsyncJobMgr-Heartbeat-1:ctx-9b43d1fd)
> > > > > > > (logid:a605267a)
> > > > > > > > > End
> > > > > > > > > > >> > >>>>> cleanup
> > > > > > > > > > >> > >>>>>>>> expired
> > > > > > > > > > >> > >>>>>>>>> async-jobs
> > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:28:01,142 DEBUG
> > > > > > > > > > >> > >>>>>>>>
> [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> > > > > > > > > > >> > >>>>>>>>> (RouterStatusMonitor-1:ctx-ad6bbe7e)
> > > > > > (logid:04e4c72b)
> > > > > > > > > Found
> > > > > > > > > > 0
> > > > > > > > > > >> > >>>>>>>>> routers to update status.
> > > > > > > > > > >> > >>>>>>>>>
> > > > > > > > > > >> > >>>>>>>>
> > > > > > > > > > >> > >>>>>>>>
> > > > > > > > > > >> > >>>>>>>> --
> > > > > > > > > > >> > >>>>>>>> Rafael Weingärtner
> > > > > > > > > > >> > >>>>>>>>
> > > > > > > > > > >> > >>>>>>>
> > > > > > > > > > >> > >>>>>>
> > > > > > > > > > >> > >>>>>
> > > > > > > > > > >> > >>>>
> > > > > > > > > > >> > >>>
> > > > > > > > > > >> > >>
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> >
> > Andrija Panić
> >
>
>
>
