It started with 4.10 and then gradually upgraded with all stops, when new
releases were available.


>>> Why do you have 3 zones in this installation - what is the setup ?
>>> SSVM and CPVM (for whatever zone) are failing to be created...
Its a result of attempts to create new zone and somehow move VMs to this
new zone. These all are unsuccessful attempts.
Before problem started there was 1 zone and There should be just 1 zone in
reality.


>>> yes, the VR can't be started, it get's timeout - in AGENT logs, I see
that
>>> it attemps to create a volume on primary storage...
I guess this is the root cause. I checked, and primary storage is
accessible via NFS share on both hosts. How to troubleshoot it?


On Wed, Mar 20, 2019 at 6:29 PM Andrija Panic <andrija.pa...@gmail.com>
wrote:

> Hi,
>
> 2019-03-20 06:41:50,446 INFO  [c.c.u.DatabaseUpgradeChecker] (main:null)
> (logid:) DB version = 4.10.0.0 Code Version = 4.11.2.0
> 2019-03-20 06:41:50,447 DEBUG [c.c.u.DatabaseUpgradeChecker] (main:null)
> (logid:) Running upgrade Upgrade41000to41100 to upgrade from
> 4.10.0.0-4.11.0.0 to 4.11.0.0
> fails due to
> java.sql.SQLException: Error on rename of './cloud/ldap_trust_map' to
> './cloud/#sql2-2f01-13d' (errno: 152)
>
> Then later...
>
> com.cloud.exception.InsufficientServerCapacityException: Unable to create a
> deployment for VM[SecondaryStorageVm|s-734-VM]Scope=interface
> com.cloud.dc.DataCenter; id=3
> com.cloud.exception.InsufficientServerCapacityException: Unable to create a
> deployment for VM[ConsoleProxy|v-733-VM]Scope=interface
> com.cloud.dc.DataCenter; id=3
>
> 2019-03-20 15:02:39,113 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
> (secstorage-1:ctx-059f87f3) (logid:cf6cf89a) Zone 1 is ready to launch
> secondary storage VM
> 2019-03-20 15:02:39,117 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
> (secstorage-1:ctx-059f87f3) (logid:cf6cf89a) Zone 2 is not ready to launch
> secondary storage VM yet
> 2019-03-20 15:02:39,122 DEBUG [o.a.c.s.SecondaryStorageManagerImpl]
> (secstorage-1:ctx-059f87f3) (logid:cf6cf89a) Zone 3 is ready to launch
> secondary storage VM
>
> so did you start with clean 4.11.2 install, or was it upgraded one - I see
> in logs an upgrade from DB version 4.10 to 4.11 was tried and failed...
> Why do you have 3 zones in this installation - what is the setup ?
> SSVM and CPVM (for whatever zone) are failing to be created...
>
> yes, the VR can't be started, it get's timeout - in AGENT logs, I see that
> it attemps to create a volume on primary storage...
>
>
> Also, for SSVM I got this one...
> 2019-03-20 14:38:09,227 DEBUG [c.c.d.FirstFitPlanner]
> (Work-Job-Executor-96:ctx-04c5c9f2 job-5120/job-6960 ctx-fde3d4d7)
> (logid:49483c7a) No clusters found having a host with enough capacity,
> returning.
>
> Andrija
>
> On Wed, 20 Mar 2019 at 16:39, Jevgeni Zolotarjov <j.zolotar...@gmail.com>
> wrote:
>
> > Basic Zone - Yes
> >
> > router has been actually started/created on KVM side - not created, not
> > started. Thats the main problem, I guess
> >
> > agent.log
> > https://drive.google.com/open?id=1rATxHKqgNKo2kD23BtlrZy_9gFXC-Bq-
> >
> > management log
> > https://drive.google.com/open?id=1H2jI0roeiWxtzReB8qV6QxDkNpaki99A
> >
> > >> Can you confirm your zone/pod/cluster/hosts are all in Enabled state,
> > i.e.
> > YES, all green
> >
> > >> Can you connect your both KVM hosts can access/mount both Primary and
> > Secondary Storage
> > YES. Double checked
> >
> > >>>Can you also explain your infrastructure - you said you have two hosts
> > only, where does CloudStack management run?
> > 2 hosts:
> > host1: 192.168.1.14
> > host2: 192.168.1.5
> >
> > Servers are standing next to each other - connected to the same switch
> > Management server runs on the same physical server with host1
> >
> > I noticed, that Virtual router gets created after I try to start any of
> the
> > existing guest VM
> > Here are logs
> > management:
> > https://drive.google.com/open?id=1H2jI0roeiWxtzReB8qV6QxDkNpaki99A
> >
> > agent on host1:
> > https://drive.google.com/open?id=1u8YHYIuyU2MA2UKY7G5z7q8p5XxU1zsy
> >
> > agent on host2:
> > https://drive.google.com/open?id=1YzkCL-FmTgPva-QHHp5vTM5Nb3qAXxz4
> >
> > But this virtual router stays in Starting state forever and hence VMs do
> > not start either.
> >
> > On Wed, Mar 20, 2019 at 2:49 PM Andrija Panic <andrija.pa...@gmail.com>
> > wrote:
> >
> > > Just to confirm, you are using Basic Zone in CloudStack, right ?
> > >
> > > Can you confirm that router has been actually started/created on KVM
> > side,
> > > again, as requested please post logs (mgmt and agent - and note the
> time
> > > around which you tried to start VR last time it partially succeeded) -
> we
> > > can't guess what went wrong without logs.
> > >
> > > I would push more effort solving this one, instead of reinstalling -
> you
> > > might hit the issue again and then it's no good.
> > >
> > > Can you confirm your zone/pod/cluster/hosts are all in Enabled state,
> > i.e.
> > > not disabled and hosts connected AND both SSVM and CPVM are
> > > connectedUP/green
> > > Is your dashboard in GUI all green - no issues there ?
> > > Can you connect your both KVM hosts can access/mount both Primary and
> > > Secondary Storage
> > >
> > > On Wed, 20 Mar 2019 at 13:15, Jevgeni Zolotarjov <
> j.zolotar...@gmail.com
> > >
> > > wrote:
> > >
> > > > After dozen of attempts, the Virtual Router could finally be
> recreated.
> > > But
> > > > its in eternal Starting status, and console prompts it required
> upgrade
> > > and
> > > > Version is UNKNOWN
> > > >
> > > > It does not resolve the problem, I cannot move further form this
> point.
> > > > Any hints?
> > > >
> > > > Or I am condemned to do reinstall cloudstack from scratch?
> > > >
> > > > On Wed, Mar 20, 2019 at 11:08 AM Jevgeni Zolotarjov <
> > > > j.zolotar...@gmail.com>
> > > > wrote:
> > > >
> > > > > Under this defaultGuestNetwork, I go to Virtual Appliances. There
> is
> > no
> > > > > VMS - "no data to show"
> > > > >
> > > > > I dont have any network, other than this single default one.
> > > > >
> > > > > I've tried adding new network - Add guest network. But I am not
> able
> > to
> > > > do
> > > > > so, cause in the wizard popup, it offers empty dropdown with Zones
> > > > > selection. And this wizard doesnt not allow to go further without
> > > > selecting
> > > > > Zone
> > > > >
> > > > > On Wed, Mar 20, 2019 at 10:28 AM Andrija Panic <
> > > andrija.pa...@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> you need to delete/remove all VMs inside this network (tick the
> > > > "Expunge"
> > > > >> button during VM deletion - if you want to really delete the VMs)
> in
> > > > order
> > > > >> to be able to delete the network - OR simply attach this VM to
> > another
> > > > >> network, make this new network a DEFAULT one (NIC that is...), and
> > > then
> > > > >> detach from old network - and then effectively your VM was
> "removed"
> > > > from
> > > > >> old network - after this you should be able to delete the old
> > > network. I
> > > > >> assume some DB incosistencies perhaps, being the reason you can
> not
> > > > >> restart
> > > > >> the network.
> > > > >>
> > > > >> Did you try restarting some other Network - or deploying a new
> > > network,
> > > > >> spin a VM in it, then again try to restart this new network - does
> > it
> > > > >> work ?
> > > > >>
> > > > >> Andrija
> > > > >>
> > > > >> On Wed, 20 Mar 2019 at 08:58, Jevgeni Zolotarjov <
> > > > j.zolotar...@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >> > >>>Stop mgmt,
> > > > >> > >>>Stop all agents
> > > > >> > >>>Restart libvirtd (and check libvirt logs afterwards)
> > > > >> > >>>Start agents
> > > > >> > >>>Start mgmt.
> > > > >> >
> > > > >> > I did that numerous time. Nothing really suspicious
> > > > >> > I can see that systems VMs are running - both in cloudstack
> > console
> > > > and
> > > > >> > with virsh list -all
> > > > >> >
> > > > >> > It is apparently problem with network.
> > > > >> > Is there a way to force recreation of defaultGuestNetwork? or
> > force
> > > > >> > recreation of Virtual Router.
> > > > >> > I am unable to delete network, which is supposed to rebuild
> > network
> > > > with
> > > > >> > its router. Thats the issue
> > > > >> >
> > > > >> > The issue with libvirtd was, that eventually at some point it
> was
> > > > >> updated
> > > > >> > during 4 months of running, and not rebooted. It still worked.
> We
> > > had
> > > > to
> > > > >> > add listen_tcp = 1 for libvirtd to start working again.
> > > > >> >
> > > > >> > On Wed, Mar 20, 2019 at 9:49 AM Andrija Panic <
> > > > andrija.pa...@gmail.com>
> > > > >> > wrote:
> > > > >> >
> > > > >> > > As Sergey suggested... but i would also verify no libvirt
> issues
> > > or
> > > > >> > storage
> > > > >> > > pool issues - so perhaps:
> > > > >> > >
> > > > >> > > Stop mgmt,
> > > > >> > > Stop all agents
> > > > >> > > Restart libvirtd (and check libvirt logs afterwards)
> > > > >> > > Start agents
> > > > >> > > Start mgmt.
> > > > >> > >
> > > > >> > > What was originally issue with libvirtd ?
> > > > >> > > That sounds fishy to me...
> > > > >> > >
> > > > >> > > Andrija
> > > > >> > >
> > > > >> > > On Wed, Mar 20, 2019, 02:15 Sergey Levitskiy <
> > serg...@hotmail.com
> > > >
> > > > >> > wrote:
> > > > >> > >
> > > > >> > > > select * from networks where removed is null;
> > > > >> > > > select * from vm_instance where id=87;
> > > > >> > > > select id,name from vm_instance where name like 'r%' and
> > removed
> > > > is
> > > > >> > null;
> > > > >> > > >
> > > > >> > > > Basically since the network offering is not redundant this
> > error
> > > > is
> > > > >> > only
> > > > >> > > > thrown when there is no router associated with your network.
> > > > Usually
> > > > >> > > > management server restart tries to implement network again.
> > > Please
> > > > >> > > restart
> > > > >> > > > management server, save and share management server log.
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > On 3/19/19, 3:31 PM, "Jevgeni Zolotarjov" <
> > > > j.zolotar...@gmail.com>
> > > > >> > > wrote:
> > > > >> > > >
> > > > >> > > >     >>>>Check network_offering table for  value in column
> > > > >> > > >     redundant_router_service  for the network offering you
> > use.
> > > > >> > > >     in table network_offering_table all records have
> > > > >> > > > redundant_router_service =
> > > > >> > > >     0
> > > > >> > > >
> > > > >> > > >     Can you also run the following:
> > > > >> > > >     >>>select name, state, removed  from host where name
> like
> > > 'r%'
> > > > >> > > >     returns zero rows - nothing
> > > > >> > > >
> > > > >> > > >     >>>select * from domain_router;
> > > > >> > > >     # id, element_id, public_mac_address, public_ip_address,
> > > > >> > > > public_netmask,
> > > > >> > > >     guest_netmask, guest_ip_address, is_redundant_router,
> > > > priority,
> > > > >> > > >     redundant_state, stop_pending, role, template_version,
> > > > >> > > scripts_version,
> > > > >> > > >     vpc_id, update_state
> > > > >> > > >     '4', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL,
> > > 'UNKNOWN',
> > > > >> '0',
> > > > >> > > >     'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14
> > > > 15:37:28
> > > > >> > UTC
> > > > >> > > > 2018',
> > > > >> > > >     '57db7bd8118977a5f2cd3ef1c7503633\n', NULL, NULL
> > > > >> > > >     '49', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL,
> > > 'UNKNOWN',
> > > > >> '0',
> > > > >> > > >     'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14
> > > > 15:37:28
> > > > >> > UTC
> > > > >> > > > 2018',
> > > > >> > > >     'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL
> > > > >> > > >     '73', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL,
> > > 'UNKNOWN',
> > > > >> '0',
> > > > >> > > >     'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14
> > > > 15:37:28
> > > > >> > UTC
> > > > >> > > > 2018',
> > > > >> > > >     'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL
> > > > >> > > >     '74', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL,
> > > 'UNKNOWN',
> > > > >> '0',
> > > > >> > > >     'VIRTUAL_ROUTER', NULL, NULL, NULL, NULL
> > > > >> > > >     '75', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL,
> > > 'UNKNOWN',
> > > > >> '0',
> > > > >> > > >     'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14
> > > > 15:37:28
> > > > >> > UTC
> > > > >> > > > 2018',
> > > > >> > > >     'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL
> > > > >> > > >     '76', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL,
> > > 'UNKNOWN',
> > > > >> '0',
> > > > >> > > >     'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14
> > > > 15:37:28
> > > > >> > UTC
> > > > >> > > > 2018',
> > > > >> > > >     'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL
> > > > >> > > >     '77', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL,
> > > 'UNKNOWN',
> > > > >> '0',
> > > > >> > > >     'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.1 Fri Jun 22
> > > > 07:52:17
> > > > >> > UTC
> > > > >> > > > 2018',
> > > > >> > > >     'c03a474302d89fa82d345e10fe4cb751\n', NULL,
> > 'UPDATE_FAILED'
> > > > >> > > >     '80', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL,
> > > 'UNKNOWN',
> > > > >> '0',
> > > > >> > > >     'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.1 Fri Jun 22
> > > > 07:52:17
> > > > >> > UTC
> > > > >> > > > 2018',
> > > > >> > > >     'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL
> > > > >> > > >     '85', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL,
> > > 'UNKNOWN',
> > > > >> '0',
> > > > >> > > >     'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.1 Fri Jun 22
> > > > 07:52:17
> > > > >> > UTC
> > > > >> > > > 2018',
> > > > >> > > >     'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL
> > > > >> > > >     '86', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL,
> > > 'UNKNOWN',
> > > > >> '0',
> > > > >> > > >     'VIRTUAL_ROUTER', NULL, NULL, NULL, NULL
> > > > >> > > >     '87', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL,
> > > 'UNKNOWN',
> > > > >> '0',
> > > > >> > > >     'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.2.0 Mon Nov
> 12
> > > > >> 15:06:49
> > > > >> > > UTC
> > > > >> > > >     2018', '873057731ff2cba4a1f3b2411765407c\n', NULL, NULL
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >     >>>select * from router_network_ref;
> > > > >> > > >     # id, router_id, network_id, guest_type
> > > > >> > > >     '1', '4', '204', 'Shared'
> > > > >> > > >     '2', '49', '204', 'Shared'
> > > > >> > > >     '3', '73', '204', 'Shared'
> > > > >> > > >     '4', '75', '204', 'Shared'
> > > > >> > > >     '5', '76', '204', 'Shared'
> > > > >> > > >     '6', '77', '204', 'Shared'
> > > > >> > > >     '7', '80', '204', 'Shared'
> > > > >> > > >     '8', '85', '204', 'Shared'
> > > > >> > > >     '9', '86', '204', 'Shared'
> > > > >> > > >     '10', '87', '204', 'Shared'
> > > > >> > > >
> > > > >> > > >
> > > > >> > > >     On Wed, Mar 20, 2019 at 12:18 AM Sergey Levitskiy <
> > > > >> > > serg...@hotmail.com
> > > > >> > > > >
> > > > >> > > >     wrote:
> > > > >> > > >
> > > > >> > > >     > Check network_offering table for  value in column
> > > > >> > > >     > redundant_router_service  for the network offering you
> > > use.
> > > > >> > > >     > Can you also run the following:
> > > > >> > > >     > select name, state, removed  from host where name like
> > > 'r%'
> > > > >> > > >     > select * from domain_router;
> > > > >> > > >     > select * from router_network_ref;
> > > > >> > > >     >
> > > > >> > > >     > Cloudstack is supposed to recreate you VR. If it is
> not
> > > > >> happening
> > > > >> > > > there is
> > > > >> > > >     > something fundamentally wrong. I would advise to
> destroy
> > > > your
> > > > >> VR
> > > > >> > > > again.
> > > > >> > > >     > Stop you management server. Rotate management server
> log
> > > and
> > > > >> > start
> > > > >> > > it
> > > > >> > > >     > again. If your VR doesn't start in few min, post your
> > > > complete
> > > > >> > > > management
> > > > >> > > >     > server log  and agent log again.
> > > > >> > > >     >
> > > > >> > > >     >
> > > > >> > > >     >
> > > > >> > > >     >
> > > > >> > > >     > On 3/19/19, 2:56 PM, "Jevgeni Zolotarjov" <
> > > > >> > j.zolotar...@gmail.com>
> > > > >> > > > wrote:
> > > > >> > > >     >
> > > > >> > > >     >     >>>Network offering needs to be change to
> > > non-redundant
> > > > >> > > >     >     How do I do that?
> > > > >> > > >     >
> > > > >> > > >     >     On Tue, Mar 19, 2019 at 11:47 PM Sergey Levitskiy
> <
> > > > >> > > > serg...@hotmail.com
> > > > >> > > >     > >
> > > > >> > > >     >     wrote:
> > > > >> > > >     >
> > > > >> > > >     >     > Network offering needs to be change to
> > > non-redundant.
> > > > >> Most
> > > > >> > > > likely
> > > > >> > > >     > old bug
> > > > >> > > >     >     > is resurfaced.
> > > > >> > > >     >     >
> > > https://issues.apache.org/jira/browse/CLOUDSTACK-9024
> > > > >> > > >     >     >
> > > > >> > > >     >     >
> > > > >> > > >     >     >
> > > > >> > > >     >     > On 3/19/19, 2:19 PM, "Jevgeni Zolotarjov" <
> > > > >> > > > j.zolotar...@gmail.com>
> > > > >> > > >     > wrote:
> > > > >> > > >     >     >
> > > > >> > > >     >     >     Hello
> > > > >> > > >     >     >
> > > > >> > > >     >     >     I did exactly like you suggested.
> > > > >> > > >     >     >
> > > > >> > > >     >     >     After UPDATE on db, I can see the router in
> > > > >> cloudstack
> > > > >> > > > console.
> > > > >> > > >     >     >     But attempt to restart network fails.
> > > > >> > > >     >     >     I get error:
> > > > >> > > >     >     >     Resource [DataCenter:1] is unreachable:
> Can't
> > > find
> > > > >> all
> > > > >> > > > necessary
> > > > >> > > >     >     > running
> > > > >> > > >     >     >     routers!
> > > > >> > > >     >     >
> > > > >> > > >     >     >     I rechecked agents on both servers. They
> look
> > > > >> running
> > > > >> > OK.
> > > > >> > > >     >     >
> > > > >> > > >     >     >     On Tue, Mar 19, 2019 at 10:00 PM Andrija
> > Panic <
> > > > >> > > >     >     > andrija.pa...@gmail.com>
> > > > >> > > >     >     >     wrote:
> > > > >> > > >     >     >
> > > > >> > > >     >     >     > Ok, so:
> > > > >> > > >     >     >     >
> > > > >> > > >     >     >     > 1.BACKUP YOUR DB - in case there is
> issues,
> > > you
> > > > >> will
> > > > >> > > > want to
> > > > >> > > >     > restore
> > > > >> > > >     >     >     > 2. I tried to reproduce your case on
> 4.11.2
> > > > >> (though
> > > > >> > > > didn't do
> > > > >> > > >     > any
> > > > >> > > >     >     >     > maintenance etc.) - and could not - I have
> > > > >> > artificially
> > > > >> > > > marked
> > > > >> > > >     >     > existing VR
> > > > >> > > >     >     >     > as removed in DB - and tried to restart
> > > network
> > > > >> and
> > > > >> > it
> > > > >> > > > worked
> > > > >> > > >     > just
> > > > >> > > >     >     > fine.
> > > > >> > > >     >     >     >
> > > > >> > > >     >     >     > Let's try to "undelete" the VR for that
> > > network
> > > > -
> > > > >> > > though
> > > > >> > > > I
> > > > >> > > >     > can't be
> > > > >> > > >     >     > sure if
> > > > >> > > >     >     >     > this will work or not.
> > > > >> > > >     >     >     >
> > > > >> > > >     >     >     > Find VR based on the network UUID:
> > > > >> > > >     >     >     >       SELECT * FROM vm_instance WHERE id
> IN
> > > > >> (SELECT
> > > > >> > > >     > instance_id from
> > > > >> > > >     >     > nics
> > > > >> > > >     >     >     > where network_id IN (SELECT id FROM
> networks
> > > > WHERE
> > > > >> > > >     >     >     >
> > UUID="65ca9a05-ff96-4563-ab9c-ffb610dc8b73"));
> > > > >> > > >     >     >     > Obviously, replace this UUID with your
> > network
> > > > >> UUID.
> > > > >> > > >     >     >     > SQL above will return all VMs (user VMs,
> > VRs,
> > > > etc,
> > > > >> > > > delete or
> > > > >> > > >     > not...)
> > > > >> > > >     >     > that
> > > > >> > > >     >     >     > have NIC inside the network with that
> UUID -
> > > > look
> > > > >> for
> > > > >> > > the
> > > > >> > > >     > r-XX-VM
> > > > >> > > >     >     > being a
> > > > >> > > >     >     >     > router.
> > > > >> > > >     >     >     >
> > > > >> > > >     >     >     > Now that you have the name of the VR (i.e.
> > > > >> r-4-VM) -
> > > > >> > > you
> > > > >> > > > want
> > > > >> > > >     > to set
> > > > >> > > >     >     > 2
> > > > >> > > >     >     >     > fields to some other value than what it
> > might
> > > be
> > > > >> > > > already, as
> > > > >> > > >     > below:
> > > > >> > > >     >     >     >
> > > > >> > > >     >     >     > Need to set field "state" to Stopped:
> > > > >> > > >     >     >     >       UPDATE cloud.vm_instance SET
> > > > state='Stopped'
> > > > >> > > WHERE
> > > > >> > > >     >     > name="r-4-VM";
> > > > >> > > >     >     >     >  (what state is set currently ???)
> > > > >> > > >     >     >     >
> > > > >> > > >     >     >     > Need to set "removed" to NULL value:
> > > > >> > > >     >     >     >       UPDATE cloud.vm_instance SET
> removed =
> > > > NULL
> > > > >> > WHERE
> > > > >> > > >     >     > NAME="r-4-VM"; (is
> > > > >> > > >     >     >     > there a removed date already set ???)
> > > > >> > > >     >     >     >
> > > > >> > > >     >     >     > Obviously make sure that this VR is not
> > > running
> > > > on
> > > > >> > any
> > > > >> > > >     > hypervisor
> > > > >> > > >     >     > and if
> > > > >> > > >     >     >     > running, virsh destroy it...
> > > > >> > > >     >     >     >
> > > > >> > > >     >     >     > AFTER above has been done (VR is
> considered
> > to
> > > > >> exist
> > > > >> > > but
> > > > >> > > >     > stopped,
> > > > >> > > >     >     > from
> > > > >> > > >     >     >     > CloudStack point of view) - try to RESTART
> > the
> > > > >> > network
> > > > >> > > > (don't
> > > > >> > > >     > bother
> > > > >> > > >     >     >     > deleteing it, since there are VMs in that
> > > > network)
> > > > >> > > >     >     >     >
> > > > >> > > >     >     >     > I'm not 100% positive this will fix your
> > > issue,
> > > > >> but
> > > > >> > > > doesn't
> > > > >> > > >     > hurt to
> > > > >> > > >     >     > try
> > > > >> > > >     >     >     >
> > > > >> > > >     >     >     > If above doesn't work - I would still
> take a
> > > > look
> > > > >> > into
> > > > >> > > > agents
> > > > >> > > >     > and if
> > > > >> > > >     >     > they
> > > > >> > > >     >     >     > are still connected - optionally, restart
> > > agents
> > > > >> on
> > > > >> > > both
> > > > >> > > > hosts
> > > > >> > > >     > once
> > > > >> > > >     >     > more
> > > > >> > > >     >     >     > and confirm they are connected and up.
> > > > >> > > >     >     >     >
> > > > >> > > >     >     >     > Let us know how it goes (and please backup
> > DB
> > > > once
> > > > >> > more
> > > > >> > > > before
> > > > >> > > >     > any
> > > > >> > > >     >     > actions
> > > > >> > > >     >     >     > !)
> > > > >> > > >     >     >     >
> > > > >> > > >     >     >     > Andrija
> > > > >> > > >     >     >     >
> > > > >> > > >     >     >     >
> > > > >> > > >     >     >     > On Tue, 19 Mar 2019 at 20:41, Jevgeni
> > > > Zolotarjov <
> > > > >> > > >     >     > j.zolotar...@gmail.com>
> > > > >> > > >     >     >     > wrote:
> > > > >> > > >     >     >     >
> > > > >> > > >     >     >     > > Yes.  Just a single network.
> > > > >> > > >     >     >     > >
> > > > >> > > >     >     >     > > On Tue, 19 Mar 2019, 21:39 Andrija
> Panic,
> > <
> > > > >> > > >     > andrija.pa...@gmail.com
> > > > >> > > >     >     > >
> > > > >> > > >     >     >     > wrote:
> > > > >> > > >     >     >     > >
> > > > >> > > >     >     >     > > > Just one more clarification - this is
> > > > Isolate
> > > > >> > > single
> > > > >> > > >     > network (not
> > > > >> > > >     >     >     > Shared
> > > > >> > > >     >     >     > > > Network, not VPC) ?
> > > > >> > > >     >     >     > > >
> > > > >> > > >     >     >     > > >
> > > > >> > > >     >     >     > > >
> > > > >> > > >     >     >     > > > On Tue, Mar 19, 2019, 19:36 Jevgeni
> > > > >> Zolotarjov <
> > > > >> > > >     >     > j.zolotar...@gmail.com
> > > > >> > > >     >     >     > >
> > > > >> > > >     >     >     > > > wrote:
> > > > >> > > >     >     >     > > >
> > > > >> > > >     >     >     > > > > Name defaultGuestNetwork
> > > > >> > > >     >     >     > > > > ID
> > 4ba834ed-48f3-468f-b667-9bb2d2c258f1
> > > > >> > > >     >     >     > > > > Zone PBT zone 1
> > > > >> > > >     >     >     > > > > Description defaultGuestNetwork
> > > > >> > > >     >     >     > > > > Type Shared
> > > > >> > > >     >     >     > > > > State Setup
> > > > >> > > >     >     >     > > > > VPC ID N/A
> > > > >> > > >     >     >     > > > > Persistent No
> > > > >> > > >     >     >     > > > >
> > > > >> > > >     >     >     > > > > On Tue, Mar 19, 2019 at 8:29 PM
> > Andrija
> > > > >> Panic <
> > > > >> > > >     >     >     > andrija.pa...@gmail.com
> > > > >> > > >     >     >     > > >
> > > > >> > > >     >     >     > > > > wrote:
> > > > >> > > >     >     >     > > > >
> > > > >> > > >     >     >     > > > > > Share the network id and name as
> > seen
> > > > from
> > > > >> > > GUI...
> > > > >> > > >     >     >     > > > > >
> > > > >> > > >     >     >     > > > > > On Tue, Mar 19, 2019, 19:27
> Jevgeni
> > > > >> > Zolotarjov
> > > > >> > > <
> > > > >> > > >     >     >     > > j.zolotar...@gmail.com
> > > > >> > > >     >     >     > > > >
> > > > >> > > >     >     >     > > > > > wrote:
> > > > >> > > >     >     >     > > > > >
> > > > >> > > >     >     >     > > > > > > >>> 1. Confirm please rolling
> > > restart
> > > > is
> > > > >> > set
> > > > >> > > > to false
> > > > >> > > >     >     > please -
> > > > >> > > >     >     >     > > double
> > > > >> > > >     >     >     > > > > > check
> > > > >> > > >     >     >     > > > > > > Double checked - It is set to
> > false
> > > > >> > > >     >     >     > > > > > >
> > > > >> > > >     >     >     > > > > > > >>>>2. If so - do you know the
> > name
> > > of
> > > > >> VR
> > > > >> > > > which you
> > > > >> > > >     >     > deleted ? Is
> > > > >> > > >     >     >     > it
> > > > >> > > >     >     >     > > > > last
> > > > >> > > >     >     >     > > > > > > one
> > > > >> > > >     >     >     > > > > > > >>> ever created - if so we can
> > find
> > > > it
> > > > >> > > > easily...
> > > > >> > > >     >     >     > > > > > > I dont know the name
> > > > >> > > >     >     >     > > > > > > Is there a way to fetch it from
> > DB?
> > > > >> > > >     >     >     > > > > > >
> > > > >> > > >     >     >     > > > > > > On Tue, Mar 19, 2019 at 7:58 PM
> > > > Andrija
> > > > >> > > Panic <
> > > > >> > > >     >     >     > > > andrija.pa...@gmail.com
> > > > >> > > >     >     >     > > > > >
> > > > >> > > >     >     >     > > > > > > wrote:
> > > > >> > > >     >     >     > > > > > >
> > > > >> > > >     >     >     > > > > > > > Right...still complaining on
> > > missing
> > > > >> > > running
> > > > >> > > >     > routers.
> > > > >> > > >     >     >     > > > > > > >
> > > > >> > > >     >     >     > > > > > > > 1. Confirm please rolling
> > restart
> > > is
> > > > >> set
> > > > >> > to
> > > > >> > > > false
> > > > >> > > >     > please
> > > > >> > > >     >     > -
> > > > >> > > >     >     >     > double
> > > > >> > > >     >     >     > > > > check
> > > > >> > > >     >     >     > > > > > > > 2. If so - do you know the
> name
> > of
> > > > VR
> > > > >> > which
> > > > >> > > > you
> > > > >> > > >     > deleted
> > > > >> > > >     >     > ? Is it
> > > > >> > > >     >     >     > > > last
> > > > >> > > >     >     >     > > > > > one
> > > > >> > > >     >     >     > > > > > > > ever created - if so we can
> find
> > > it
> > > > >> > > easily...
> > > > >> > > >     >     >     > > > > > > >
> > > > >> > > >     >     >     > > > > > > >
> > > > >> > > >     >     >     > > > > > > >
> > > > >> > > >     >     >     > > > > > > > On Tue, Mar 19, 2019, 18:40
> > > Jevgeni
> > > > >> > > > Zolotarjov <
> > > > >> > > >     >     >     > > > > j.zolotar...@gmail.com
> > > > >> > > >     >     >     > > > > > >
> > > > >> > > >     >     >     > > > > > > > wrote:
> > > > >> > > >     >     >     > > > > > > >
> > > > >> > > >     >     >     > > > > > > > > Here is management server
> log
> > > > >> > > >     >     >     > > > > > > > >
> > > > >> > > >     >     >     > > >
> > > > >> > > >     >     >
> > > > >> > > >
> > > > https://drive.google.com/open?id=1H2jI0roeiWxtzReB8qV6QxDkNpaki99A
> > > > >> > > >     >     >     > > > > > > > >
> > > > >> > > >     >     >     > > > > > > > >
> > > > >> > > >     >     >     > > > > > > > > On Tue, Mar 19, 2019 at 7:29
> > PM
> > > > >> Andrija
> > > > >> > > > Panic <
> > > > >> > > >     >     >     > > > > > andrija.pa...@gmail.com
> > > > >> > > >     >     >     > > > > > > >
> > > > >> > > >     >     >     > > > > > > > > wrote:
> > > > >> > > >     >     >     > > > > > > > >
> > > > >> > > >     >     >     > > > > > > > > > I would also try to
> > "undelete"
> > > > VR
> > > > >> in
> > > > >> > > DB,
> > > > >> > > > but
> > > > >> > > >     > let's
> > > > >> > > >     >     > keep
> > > > >> > > >     >     >     > this
> > > > >> > > >     >     >     > > as
> > > > >> > > >     >     >     > > > > > last
> > > > >> > > >     >     >     > > > > > > > > step.
> > > > >> > > >     >     >     > > > > > > > > >
> > > > >> > > >     >     >     > > > > > > > > > On Tue, Mar 19, 2019,
> 18:24
> > > > >> Andrija
> > > > >> > > > Panic <
> > > > >> > > >     >     >     > > > > andrija.pa...@gmail.com
> > > > >> > > >     >     >     > > > > > >
> > > > >> > > >     >     >     > > > > > > > > wrote:
> > > > >> > > >     >     >     > > > > > > > > >
> > > > >> > > >     >     >     > > > > > > > > > > Disclaimer: from mobile
> > > > device.
> > > > >> > > >     >     >     > > > > > > > > > >
> > > > >> > > >     >     >     > > > > > > > > > > I don't see any special
> > > > failure
> > > > >> in
> > > > >> > > > agent log,
> > > > >> > > >     >     > except some
> > > > >> > > >     >     >     > > > long
> > > > >> > > >     >     >     > > > > > > > running
> > > > >> > > >     >     >     > > > > > > > > > > migration, timeout for
> > > > graceful
> > > > >> VM
> > > > >> > > > shutdown
> > > > >> > > >     > etc
> > > > >> > > >     >     > (and
> > > > >> > > >     >     >     > agent
> > > > >> > > >     >     >     > > > > > restart)
> > > > >> > > >     >     >     > > > > > > > > > >
> > > > >> > > >     >     >     > > > > > > > > > > You did not send mgmt
> log
> > > > after
> > > > >> > last
> > > > >> > > > network
> > > > >> > > >     >     > restart
> > > > >> > > >     >     >     > > failure
> > > > >> > > >     >     >     > > > ?
> > > > >> > > >     >     >     > > > > > > > > > >
> > > > >> > > >     >     >     > > > > > > > > > > On Tue, Mar 19, 2019,
> > 17:29
> > > > >> Jevgeni
> > > > >> > > >     > Zolotarjov <
> > > > >> > > >     >     >     > > > > > > > j.zolotar...@gmail.com
> > > > >> > > >     >     >     > > > > > > > > >
> > > > >> > > >     >     >     > > > > > > > > > > wrote:
> > > > >> > > >     >     >     > > > > > > > > > >
> > > > >> > > >     >     >     > > > > > > > > > >> Under
> > > > >> > > >     >     >     > > > > > > > > > >> Infrastructure / Hosts:
> > > > >> > > >     >     >     > > > > > > > > > >> both hosts are enabled
> > and
> > > > >> green
> > > > >> > > > (Unsecure
> > > > >> > > >     > though)
> > > > >> > > >     >     >     > > > > > > > > > >>
> > > > >> > > >     >     >     > > > > > > > > > >> agent.log -
> > > > >> > > >     >     >     > > > > > > > > > >>
> > > > >> > > >     >     >     > > > > > >
> > > > >> > > >     >     >     >
> > > > >> > > >     >
> > > > >> >
> > https://drive.google.com/open?id=1rATxHKqgNKo2kD23BtlrZy_9gFXC-Bq-
> > > > >> > > >     >     >     > > > > > > > > > >>
> > > > >> > > >     >     >     > > > > > > > > > >> I set
> > > network.rolling.restart
> > > > >> to
> > > > >> > > > false now.
> > > > >> > > >     >     > Restarted
> > > > >> > > >     >     >     > > > > management
> > > > >> > > >     >     >     > > > > > > > > server
> > > > >> > > >     >     >     > > > > > > > > > -
> > > > >> > > >     >     >     > > > > > > > > > >> same problem - cannot
> > > restart
> > > > >> not
> > > > >> > > > delete
> > > > >> > > >     > network
> > > > >> > > >     >     >     > > > > > > > > > >>
> > > > >> > > >     >     >     > > > > > > > > > >>
> > > > >> > > >     >     >     > > > > > > > > > >>
> > > > >> > > >     >     >     > > > > > > > > > >> On Tue, Mar 19, 2019 at
> > > 5:48
> > > > PM
> > > > >> > > Boris
> > > > >> > > >     > Stoyanov <
> > > > >> > > >     >     >     > > > > > > > > > >>
> > > boris.stoya...@shapeblue.com
> > > > >
> > > > >> > > >     >     >     > > > > > > > > > >> wrote:
> > > > >> > > >     >     >     > > > > > > > > > >>
> > > > >> > > >     >     >     > > > > > > > > > >> > Hi, you shouldn’t be
> > > > worried
> > > > >> > about
> > > > >> > > > data
> > > > >> > > >     > as long
> > > > >> > > >     >     > as you
> > > > >> > > >     >     >     > > > use a
> > > > >> > > >     >     >     > > > > > > > > separate
> > > > >> > > >     >     >     > > > > > > > > > >> > shared storage.
> > > > >> > > >     >     >     > > > > > > > > > >> >
> > > > >> > > >     >     >     > > > > > > > > > >> > What does the
> > cloudstack
> > > > >> console
> > > > >> > > say
> > > > >> > > >     > about your
> > > > >> > > >     >     > host?
> > > > >> > > >     >     >     > Is
> > > > >> > > >     >     >     > > > it
> > > > >> > > >     >     >     > > > > > up?
> > > > >> > > >     >     >     > > > > > > If
> > > > >> > > >     >     >     > > > > > > > > > it’s
> > > > >> > > >     >     >     > > > > > > > > > >> up
> > > > >> > > >     >     >     > > > > > > > > > >> > then you should be
> able
> > > to
> > > > >> > deploy
> > > > >> > > a
> > > > >> > > > VM on
> > > > >> > > >     > it.
> > > > >> > > >     >     >     > > > > > > > > > >> >
> > > > >> > > >     >     >     > > > > > > > > > >> > If not, you’ll need
> to
> > > > >> > investigate
> > > > >> > > > the
> > > > >> > > >     > reason
> > > > >> > > >     >     >     > > > > cloudstack-agent
> > > > >> > > >     >     >     > > > > > > is
> > > > >> > > >     >     >     > > > > > > > > not
> > > > >> > > >     >     >     > > > > > > > > > >> > connected, is the
> agent
> > > > >> running?
> > > > >> > > If
> > > > >> > > > not
> > > > >> > > >     > just
> > > > >> > > >     >     > start it
> > > > >> > > >     >     >     > > and
> > > > >> > > >     >     >     > > > > see
> > > > >> > > >     >     >     > > > > > if
> > > > >> > > >     >     >     > > > > > > > it
> > > > >> > > >     >     >     > > > > > > > > > >> > connects.
> > > > >> > > >     >     >     > > > > > > > > > >> >
> > > > >> > > >     >     >     > > > > > > > > > >> > If it’s running but
> > still
> > > > not
> > > > >> > > > connected
> > > > >> > > >     > please
> > > > >> > > >     >     > share
> > > > >> > > >     >     >     > the
> > > > >> > > >     >     >     > > > > > > > agent.log.
> > > > >> > > >     >     >     > > > > > > > > > >> >
> > > > >> > > >     >     >     > > > > > > > > > >> > If nothing helps at
> all
> > > you
> > > > >> can
> > > > >> > > > remove
> > > > >> > > >     > the host
> > > > >> > > >     >     > and
> > > > >> > > >     >     >     > add
> > > > >> > > >     >     >     > > it
> > > > >> > > >     >     >     > > > > > > again,
> > > > >> > > >     >     >     > > > > > > > > but
> > > > >> > > >     >     >     > > > > > > > > > >> > let’s try to
> > troubleshoot
> > > > >> first.
> > > > >> > > >     >     >     > > > > > > > > > >> >
> > > > >> > > >     >     >     > > > > > > > > > >> > Bobby.
> > > > >> > > >     >     >     > > > > > > > > > >> >
> > > > >> > > >     >     >     > > > > > > > > > >> >
> > > > >> > > >     >     >     > > > > > > > > > >> >
> > > > boris.stoya...@shapeblue.com
> > > > >> > > >     >     >     > > > > > > > > > >> > www.shapeblue.com
> > > > >> > > >     >     >     > > > > > > > > > >> > Amadeus House, Floral
> > > > Street,
> > > > >> > > London
> > > > >> > > >     > WC2E 9DPUK
> > > > >> > > >     >     >     > > > > > > > > > >> > @shapeblue
> > > > >> > > >     >     >     > > > > > > > > > >> >
> > > > >> > > >     >     >     > > > > > > > > > >> >
> > > > >> > > >     >     >     > > > > > > > > > >> >
> > > > >> > > >     >     >     > > > > > > > > > >> > > On 19 Mar 2019, at
> > > 17:29,
> > > > >> > > Jevgeni
> > > > >> > > >     > Zolotarjov <
> > > > >> > > >     >     >     > > > > > > > > > j.zolotar...@gmail.com>
> > > > >> > > >     >     >     > > > > > > > > > >> > wrote:
> > > > >> > > >     >     >     > > > > > > > > > >> > >
> > > > >> > > >     >     >     > > > > > > > > > >> > > Guys, please help
> > with
> > > > it.
> > > > >> > > >     >     >     > > > > > > > > > >> > > What can be done
> > here?
> > > > >> > > >     >     >     > > > > > > > > > >> > > There is too much
> > > > valuable
> > > > >> > data.
> > > > >> > > >     >     >     > > > > > > > > > >> > >
> > > > >> > > >     >     >     > > > > > > > > > >> > > On Tue, Mar 19,
> 2019
> > at
> > > > >> 4:21
> > > > >> > PM
> > > > >> > > > Jevgeni
> > > > >> > > >     >     > Zolotarjov <
> > > > >> > > >     >     >     > > > > > > > > > >> >
> j.zolotar...@gmail.com
> > >
> > > > >> > > >     >     >     > > > > > > > > > >> > > wrote:
> > > > >> > > >     >     >     > > > > > > > > > >> > >
> > > > >> > > >     >     >     > > > > > > > > > >> > >> Tried that just
> now
> > > and
> > > > >> got
> > > > >> > > > error:
> > > > >> > > >     >     >     > > > > > > > > > >> > >> Resource
> > > [DataCenter:1]
> > > > is
> > > > >> > > > unreachable:
> > > > >> > > >     >     > Can't find
> > > > >> > > >     >     >     > > all
> > > > >> > > >     >     >     > > > > > > > necessary
> > > > >> > > >     >     >     > > > > > > > > > >> running
> > > > >> > > >     >     >     > > > > > > > > > >> > >> routers!
> > > > >> > > >     >     >     > > > > > > > > > >> > >>
> > > > >> > > >     >     >     > > > > > > > > > >> > >> In the log I see:
> > > > >> > > >     >     >     > > > > > > > > > >> > >>
> > > > >> > > >     > =============================================
> > > > >> > > >     >     >     > > > > > > > > > >> > >>
> > > > >> > > >     >     >     > > > > > > > > > >> > >> 2019-03-19
> > > 14:20:39,644
> > > > >> DEBUG
> > > > >> > > >     >     >     > > > > > [o.a.c.e.o.NetworkOrchestrator]
> > > > >> > > >     >     >     > > > > > > > > > >> > >>
> > > > >> > > (API-Job-Executor-4:ctx-7b6b69eb
> > > > >> > > >     > job-5093
> > > > >> > > >     >     >     > > ctx-9be30648)
> > > > >> > > >     >     >     > > > > > > > > > >> (logid:265a6099)
> > > > >> > > >     >     >     > > > > > > > > > >> > >> Restarting network
> > > > 204...
> > > > >> > > >     >     >     > > > > > > > > > >> > >> 2019-03-19
> > > 14:20:39,645
> > > > >> DEBUG
> > > > >> > > >     >     >     > > > > > [o.a.c.e.o.NetworkOrchestrator]
> > > > >> > > >     >     >     > > > > > > > > > >> > >>
> > > > >> > > (API-Job-Executor-4:ctx-7b6b69eb
> > > > >> > > >     > job-5093
> > > > >> > > >     >     >     > > ctx-9be30648)
> > > > >> > > >     >     >     > > > > > > > > > >> (logid:265a6099)
> > > > >> > > >     >     >     > > > > > > > > > >> > >> Performing rolling
> > > > >> restart of
> > > > >> > > > routers
> > > > >> > > >     > of
> > > > >> > > >     >     > network
> > > > >> > > >     >     >     > > > > > > > > Ntwk[204|Guest|6]
> > > > >> > > >     >     >     > > > > > > > > > >> > >> 2019-03-19
> > > 14:20:39,649
> > > > >> DEBUG
> > > > >> > > >     >     >     > > > > > [o.a.c.e.o.NetworkOrchestrator]
> > > > >> > > >     >     >     > > > > > > > > > >> > >>
> > > > >> > > (API-Job-Executor-4:ctx-7b6b69eb
> > > > >> > > >     > job-5093
> > > > >> > > >     >     >     > > ctx-9be30648)
> > > > >> > > >     >     >     > > > > > > > > > >> (logid:265a6099)
> > > > >> > > >     >     >     > > > > > > > > > >> > >> Asking
> VirtualRouter
> > > to
> > > > >> > > > implemenet
> > > > >> > > >     >     >     > Ntwk[204|Guest|6]
> > > > >> > > >     >     >     > > > > > > > > > >> > >> 2019-03-19
> > > 14:20:39,658
> > > > >> ERROR
> > > > >> > > >     >     >     > > > > [c.c.a.ApiAsyncJobDispatcher]
> > > > >> > > >     >     >     > > > > > > > > > >> > >>
> > > > >> > > (API-Job-Executor-4:ctx-7b6b69eb
> > > > >> > > >     > job-5093)
> > > > >> > > >     >     >     > > > > (logid:265a6099)
> > > > >> > > >     >     >     > > > > > > > > > >> Unexpected
> > > > >> > > >     >     >     > > > > > > > > > >> > >> exception while
> > > > executing
> > > > >> > > >     >     >     > > > > > > > > > >> > >>
> > > > >> > > >     >     >     > > > > > > >
> > > > >> > > >     >     >     >
> > > > >> > > >     >
> > > > >> org.apache.cloudstack.api.command.user.network.RestartNetworkCmd
> > > > >> > > >     >     >     > > > > > > > > > >> > >>
> > > > >> > > >     >     >
> com.cloud.exception.ResourceUnavailableException:
> > > > >> > > >     >     >     > > > > Resource
> > > > >> > > >     >     >     > > > > > > > > > >> > [DataCenter:1]
> > > > >> > > >     >     >     > > > > > > > > > >> > >> is unreachable:
> > Can't
> > > > find
> > > > >> > all
> > > > >> > > >     > necessary
> > > > >> > > >     >     > running
> > > > >> > > >     >     >     > > > routers!
> > > > >> > > >     >     >     > > > > > > > > > >> > >>        at
> > > > >> > > >     >     >     > > > > > > > > > >> > >>
> > > > >> > > >     >     >     > > > > > > > > > >> >
> > > > >> > > >     >     >     > > > > > > > > > >>
> > > > >> > > >     >     >     > > > > > > > > >
> > > > >> > > >     >     >     > > > > > > > >
> > > > >> > > >     >     >     > > > > > > >
> > > > >> > > >     >     >     > > > > > >
> > > > >> > > >     >     >     > > > > >
> > > > >> > > >     >     >     > > > >
> > > > >> > > >     >     >     > > >
> > > > >> > > >     >     >     > >
> > > > >> > > >     >     >     >
> > > > >> > > >     >     >
> > > > >> > > >     >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:243)
> > > > >> > > >     >     >     > > > > > > > > > >> > >>        at
> > > > >> > > >     >     >     > > > > > > > > > >> > >>
> > > > >> > > >     >     >     > > > > > > > > > >> >
> > > > >> > > >     >     >     > > > > > > > > > >>
> > > > >> > > >     >     >     > > > > > > > > >
> > > > >> > > >     >     >     > > > > > > > >
> > > > >> > > >     >     >     > > > > > > >
> > > > >> > > >     >     >     > > > > > >
> > > > >> > > >     >     >     > > > > >
> > > > >> > > >     >     >     > > > >
> > > > >> > > >     >     >     > > >
> > > > >> > > >     >     >     > >
> > > > >> > > >     >     >     >
> > > > >> > > >     >     >
> > > > >> > > >     >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElements(NetworkOrchestrator.java:1203)
> > > > >> > > >     >     >     > > > > > > > > > >> > >>        at
> > > > >> > > >     >     >     > > > > > > > > > >> > >>
> > > > >> > > >     >     >     > > > > > > > > > >> >
> > > > >> > > >     >     >     > > > > > > > > > >>
> > > > >> > > >     >     >     > > > > > > > > >
> > > > >> > > >     >     >     > > > > > > > >
> > > > >> > > >     >     >     > > > > > > >
> > > > >> > > >     >     >     > > > > > >
> > > > >> > > >     >     >     > > > > >
> > > > >> > > >     >     >     > > > >
> > > > >> > > >     >     >     > > >
> > > > >> > > >     >     >     > >
> > > > >> > > >     >     >     >
> > > > >> > > >     >     >
> > > > >> > > >     >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.rollingRestartRouters(NetworkOrchestrator.java:2948)
> > > > >> > > >     >     >     > > > > > > > > > >> > >>        at
> > > > >> > > >     >     >     > > > > > > > > > >> > >>
> > > > >> > > >     >     >     > > > > > > > > > >> >
> > > > >> > > >     >     >     > > > > > > > > > >>
> > > > >> > > >     >     >     > > > > > > > > >
> > > > >> > > >     >     >     > > > > > > > >
> > > > >> > > >     >     >     > > > > > > >
> > > > >> > > >     >     >     > > > > > >
> > > > >> > > >     >     >     > > > > >
> > > > >> > > >     >     >     > > > >
> > > > >> > > >     >     >     > > >
> > > > >> > > >     >     >     > >
> > > > >> > > >     >     >     >
> > > > >> > > >     >     >
> > > > >> > > >     >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:2853)
> > > > >> > > >     >     >     > > > > > > > > > >> > >>        at
> > > > >> > > >     >     >     > > > > > > > > > >> > >>
> > > > >> > > >     >     >     > > > > > > > > > >> >
> > > > >> > > >     >     >     > > > > > > > > > >>
> > > > >> > > >     >     >     > > > > > > > > >
> > > > >> > > >     >     >     > > > > > > > >
> > > > >> > > >     >     >     > > > > > > >
> > > > >> > > >     >     >     > > > > > >
> > > > >> > > >     >     >     > > > > >
> > > > >> > > >     >     >     > > > >
> > > > >> > > >     >     >     > > >
> > > > >> > > >     >     >     > >
> > > > >> > > >     >     >     >
> > > > >> > > >     >     >
> > > > >> > > >     >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1883)
> > > > >> > > >     >     >     > > > > > > > > > >> > >>        at
> > > > >> > > >     >     >     > > > > >
> > > > >> > > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> > > > >> > > >     >     >     > > > > > > > > > Method)
> > > > >> > > >     >     >     > > > > > > > > > >> > >>        at
> > > > >> > > >     >     >     > > > > >
> > > > >> > > > sun.reflect.NativeMethodAccessorImpl.invoke(Unknown
> > > > >> > > >     >     >     > > > > > > > > > Source)
> > > > >> > > >     >     >     > > > > > > > > > >> > >>        at
> > > > >> > > >     >     >     > > > > > > >
> > > > >> > > >     >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
> > > > >> > > >     >     >     > > > > > > > > > >> > Source)
> > > > >> > > >     >     >     > > > > > > > > > >> > >>        at
> > > > >> > > >     >     > java.lang.reflect.Method.invoke(Unknown
> > > > >> > > >     >     >     > > > Source)
> > > > >> > > >     >     >     > > > > > > > > > >> > >>        at
> > > > >> > > >     >     >     > > > > > > > > > >> > >>
> > > > >> > > >     >     >     > > > > > > > > > >> >
> > > > >> > > >     >     >     > > > > > > > > > >>
> > > > >> > > >     >     >     > > > > > > > > >
> > > > >> > > >     >     >     > > > > > > > >
> > > > >> > > >     >     >     > > > > > > >
> > > > >> > > >     >     >     > > > > > >
> > > > >> > > >     >     >     > > > > >
> > > > >> > > >     >     >     > > > >
> > > > >> > > >     >     >     > > >
> > > > >> > > >     >     >     > >
> > > > >> > > >     >     >     >
> > > > >> > > >     >     >
> > > > >> > > >     >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
> > > > >> > > >     >     >     > > > > > > > > > >> > >>        at
> > > > >> > > >     >     >     > > > > > > > > > >> > >>
> > > > >> > > >     >     >     > > > > > > > > > >> >
> > > > >> > > >     >     >     > > > > > > > > > >>
> > > > >> > > >     >     >     > > > > > > > > >
> > > > >> > > >     >     >     > > > > > > > >
> > > > >> > > >     >     >     > > > > > > >
> > > > >> > > >     >     >     > > > > > >
> > > > >> > > >     >     >     > > > > >
> > > > >> > > >     >     >     > > > >
> > > > >> > > >     >     >     > > >
> > > > >> > > >     >     >     > >
> > > > >> > > >     >     >     >
> > > > >> > > >     >     >
> > > > >> > > >     >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
> > > > >> > > >     >     >     > > > > > > > > > >> > >>        at
> > > > >> > > >     >     >     > > > > > > > > > >> > >>
> > > > >> > > >     >     >     > > > > > > > > > >> >
> > > > >> > > >     >     >     > > > > > > > > > >>
> > > > >> > > >     >     >     > > > > > > > > >
> > > > >> > > >     >     >     > > > > > > > >
> > > > >> > > >     >     >     > > > > > > >
> > > > >> > > >     >     >     > > > > > >
> > > > >> > > >     >     >     > > > > >
> > > > >> > > >     >     >     > > > >
> > > > >> > > >     >     >     > > >
> > > > >> > > >     >     >     > >
> > > > >> > > >     >     >     >
> > > > >> > > >     >     >
> > > > >> > > >     >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> > > > >> > > >     >     >     > > > > > > > > > >> > >>        at
> > > > >> > > >     >     >     > > > > > > > > > >> > >>
> > > > >> > > >     >     >     > > > > > > > > > >> >
> > > > >> > > >     >     >     > > > > > > > > > >>
> > > > >> > > >     >     >     > > > > > > > > >
> > > > >> > > >     >     >     > > > > > > > >
> > > > >> > > >     >     >     > > > > > > >
> > > > >> > > >     >     >     > > > > > >
> > > > >> > > >     >     >     > > > > >
> > > > >> > > >     >     >     > > > >
> > > > >> > > >     >     >     > > >
> > > > >> > > >     >     >     > >
> > > > >> > > >     >     >     >
> > > > >> > > >     >     >
> > > > >> > > >     >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
> > > > >> > > >     >     >     > > > > > > > > > >> > >>        at
> > > > >> > > >     >     >     > > > > > > > > > >> > >>
> > > > >> > > >     >     >     > > > > > > > > > >> >
> > > > >> > > >     >     >     > > > > > > > > > >>
> > > > >> > > >     >     >     > > > > > > > > >
> > > > >> > > >     >     >     > > > > > > > >
> > > > >> > > >     >     >     > > > > > > >
> > > > >> > > >     >     >     > > > > > >
> > > > >> > > >     >     >     > > > > >
> > > > >> > > >     >     >     > > > >
> > > > >> > > >     >     >     > > >
> > > > >> > > >     >     >     > >
> > > > >> > > >     >     >     >
> > > > >> > > >     >     >
> > > > >> > > >     >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> > > > >> > > >     >     >     > > > > > > > > > >> > >>        at
> > > > >> > > >     >     >     > > > > > > > > > >> > >>
> > > > >> > > >     >     >     > > > > > > > > > >> >
> > > > >> > > >     >     >     > > > > > > > > > >>
> > > > >> > > >     >     >     > > > > > > > > >
> > > > >> > > >     >     >     > > > > > > > >
> > > > >> > > >     >     >     > > > > > > >
> > > > >> > > >     >     >     > > > > > >
> > > > >> > > >     >     >     > > > > >
> > > > >> > > >     >     >     > > > >
> > > > >> > > >     >     >     > > >
> > > > >> > > >     >     >     > >
> > > > >> > > >     >     >     >
> > > > >> > > >     >     >
> > > > >> > > >     >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
> > > > >> > > >     >     >     > > > > > > > > > >> > >>        at
> > > > >> > > >     >     >     > > > > > > > > > >> > >>
> > > > >> > > >     >     >     > > > > > > > > > >> >
> > > > >> > > >     >     >     > > > > > > > > > >>
> > > > >> > > >     >     >     > > > > > > > > >
> > > > >> > > >     >     >     > > > > > > > >
> > > > >> > > >     >     >     > > > > > > >
> > > > >> > > >     >     >     > > > > > >
> > > > >> > > >     >     >     > > > > >
> > > > >> > > >     >     >     > > > >
> > > > >> > > >     >     >     > > >
> > > > >> > > >     >     >     > >
> > > > >> > > >     >     >     >
> > > > >> > > >     >     >
> > > > >> > > >     >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> > > > >> > > >     >     >     > > > > > > > > > >> > >>        at
> > > > >> > > >     >     >     > > > > > > > > > >> > >>
> > > > >> > > >     >     >     > > > > > > > > > >> >
> > > > >> > > >     >     >     > > > > > > > > > >>
> > > > >> > > >     >     >     > > > > > > > > >
> > > > >> > > >     >     >     > > > > > > > >
> > > > >> > > >     >     >     > > > > > > >
> > > > >> > > >     >     >     > > > > > >
> > > > >> > > >     >     >     > > > > >
> > > > >> > > >     >     >     > > > >
> > > > >> > > >     >     >     > > >
> > > > >> > > >     >     >     > >
> > > > >> > > >     >     >     >
> > > > >> > > >     >     >
> > > > >> > > >     >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
> > > > >> > > >     >     >     > > > > > > > > > >> > >>        at
> > > > >> > > >     >     >     > > > > > > > > > >> > >>
> > > > >> > > >     >     >     > > > > > > > > > >> >
> > > > >> > > >     >     >     > > > > > > > > > >>
> > > > >> > > >     >     >     > > > > > > > > >
> > > > >> > > >     >     >     > > > > > > > >
> > > > >> > > >     >     >     > > > > > > >
> > > > >> > > >     >     >     > > > > > >
> > > > >> > > >     >     >     > > > > >
> > > > >> > > >     >     >     > > > >
> > > > >> > > >     >     >     > > >
> > > > >> > > >     >     >     > >
> > > > >> > > >     >     >     >
> > > > >> > > >     >     >
> > > > >> > > >     >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
> > > > >> > > >     >     >     > > > > > > > > > >> > >>        at
> > > > >> > > >     >     >     > > > > > > > > > >> > >>
> > > > >> > > >     >     >     > > > > > > > > > >> >
> > > > >> > > >     >     >     > > > > > > > > > >>
> > > > >> > > >     >     >     > > > > > > > > >
> > > > >> > > >     >     >     > > > > > > > >
> > > > >> > > >     >     >     > > > > > > >
> > > > >> > > >     >     >     > > > > > >
> > > > >> > > >     >     >     > > > > >
> > > > >> > > >     >     >     > > > >
> > > > >> > > >     >     >     > > >
> > > > >> > > >     >     >     > >
> > > > >> > > >     >     >     >
> > > > >> > > >     >     >
> > > > >> > > >     >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
> > > > >> > > >     >     >     > > > > > > > > > >> > >>        at
> > > > >> > > >     >     >     > > >
> > > > com.sun.proxy.$Proxy229.restartNetwork(Unknown
> > > > >> > > >     >     >     > > > > > > > Source)
> > > > >> > > >     >     >     > > > > > > > > > >> > >>        at
> > > > >> > > >     >     >     > > > > > > > > > >> > >>
> > > > >> > > >     >     >     > > > > > > > > > >> >
> > > > >> > > >     >     >     > > > > > > > > > >>
> > > > >> > > >     >     >     > > > > > > > > >
> > > > >> > > >     >     >     > > > > > > > >
> > > > >> > > >     >     >     > > > > > > >
> > > > >> > > >     >     >     > > > > > >
> > > > >> > > >     >     >     > > > > >
> > > > >> > > >     >     >     > > > >
> > > > >> > > >     >     >     > > >
> > > > >> > > >     >     >     > >
> > > > >> > > >     >     >     >
> > > > >> > > >     >     >
> > > > >> > > >     >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:99)
> > > > >> > > >     >     >     > > > > > > > > > >> > >>        at
> > > > >> > > >     >     >     > > > > > > > > > >>
> > > > >> > > >     >     >     > >
> > > > >> > > > com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
> > > > >> > > >     >     >     > > > > > > > > > >> > >>        at
> > > > >> > > >     >     >     > > > > > > > > > >> > >>
> > > > >> > > >     >     >     > > > > > > > > > >> >
> > > > >> > > >     >     >     > > > > > > > > > >>
> > > > >> > > >     >     >     > > > > > > > > >
> > > > >> > > >     >     >     > > > > > > > >
> > > > >> > > >     >     >     > > > > > > >
> > > > >> > > >     >     >     > > > > > >
> > > > >> > > >     >     >     > > > > >
> > > > >> > > >     >     >     > > > >
> > > > >> > > >     >     >     > > >
> > > > >> > > >     >     >     > >
> > > > >> > > >     >     >     >
> > > > >> > > >     >     >
> > > > >> > > >     >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
> > > > >> > > >     >     >     > > > > > > > > > >> > >>        at
> > > > >> > > >     >     >     > > > > > > > > > >> > >>
> > > > >> > > >     >     >     > > > > > > > > > >> >
> > > > >> > > >     >     >     > > > > > > > > > >>
> > > > >> > > >     >     >     > > > > > > > > >
> > > > >> > > >     >     >     > > > > > > > >
> > > > >> > > >     >     >     > > > > > > >
> > > > >> > > >     >     >     > > > > > >
> > > > >> > > >     >     >     > > > > >
> > > > >> > > >     >     >     > > > >
> > > > >> > > >     >     >     > > >
> > > > >> > > >     >     >     > >
> > > > >> > > >     >     >     >
> > > > >> > > >     >     >
> > > > >> > > >     >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
> > > > >> > > >     >     >     > > > > > > > > > >> > >>        at
> > > > >> > > >     >     >     > > > > > > > > > >> > >>
> > > > >> > > >     >     >     > > > > > > > > > >> >
> > > > >> > > >     >     >     > > > > > > > > > >>
> > > > >> > > >     >     >     > > > > > > > > >
> > > > >> > > >     >     >     > > > > > > > >
> > > > >> > > >     >     >     > > > > > > >
> > > > >> > > >     >     >     > > > > > >
> > > > >> > > >     >     >     > > > > >
> > > > >> > > >     >     >     > > > >
> > > > >> > > >     >     >     > > >
> > > > >> > > >     >     >     > >
> > > > >> > > >     >     >     >
> > > > >> > > >     >     >
> > > > >> > > >     >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> > > > >> > > >     >     >     > > > > > > > > > >> > >>        at
> > > > >> > > >     >     >     > > > > > > > > > >> > >>
> > > > >> > > >     >     >     > > > > > > > > > >> >
> > > > >> > > >     >     >     > > > > > > > > > >>
> > > > >> > > >     >     >     > > > > > > > > >
> > > > >> > > >     >     >     > > > > > > > >
> > > > >> > > >     >     >     > > > > > > >
> > > > >> > > >     >     >     > > > > > >
> > > > >> > > >     >     >     > > > > >
> > > > >> > > >     >     >     > > > >
> > > > >> > > >     >     >     > > >
> > > > >> > > >     >     >     > >
> > > > >> > > >     >     >     >
> > > > >> > > >     >     >
> > > > >> > > >     >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> > > > >> > > >     >     >     > > > > > > > > > >> > >>        at
> > > > >> > > >     >     >     > > > > > > > > > >> > >>
> > > > >> > > >     >     >     > > > > > > > > > >> >
> > > > >> > > >     >     >     > > > > > > > > > >>
> > > > >> > > >     >     >     > > > > > > > > >
> > > > >> > > >     >     >     > > > > > > > >
> > > > >> > > >     >     >     > > > > > > >
> > > > >> > > >     >     >     > > > > > >
> > > > >> > > >     >     >     > > > > >
> > > > >> > > >     >     >     > > > >
> > > > >> > > >     >     >     > > >
> > > > >> > > >     >     >     > >
> > > > >> > > >     >     >     >
> > > > >> > > >     >     >
> > > > >> > > >     >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> > > > >> > > >     >     >     > > > > > > > > > >> > >>        at
> > > > >> > > >     >     >     > > > > > > > > > >> > >>
> > > > >> > > >     >     >     > > > > > > > > > >> >
> > > > >> > > >     >     >     > > > > > > > > > >>
> > > > >> > > >     >     >     > > > > > > > > >
> > > > >> > > >     >     >     > > > > > > > >
> > > > >> > > >     >     >     > > > > > > >
> > > > >> > > >     >     >     > > > > > >
> > > > >> > > >     >     >     > > > > >
> > > > >> > > >     >     >     > > > >
> > > > >> > > >     >     >     > > >
> > > > >> > > >     >     >     > >
> > > > >> > > >     >     >     >
> > > > >> > > >     >     >
> > > > >> > > >     >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> > > > >> > > >     >     >     > > > > > > > > > >> > >>        at
> > > > >> > > >     >     >     > > > > > > > > > >> > >>
> > > > >> > > >     >     >     > > > > > > > > > >> >
> > > > >> > > >     >     >     > > > > > > > > > >>
> > > > >> > > >     >     >     > > > > > > > > >
> > > > >> > > >     >     >     > > > > > > > >
> > > > >> > > >     >     >     > > > > > > >
> > > > >> > > >     >     >     > > > > > >
> > > > >> > > >     >     >     > > > > >
> > > > >> > > >     >     >     > > > >
> > > > >> > > >     >     >     > > >
> > > > >> > > >     >     >     > >
> > > > >> > > >     >     >     >
> > > > >> > > >     >     >
> > > > >> > > >     >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> > > > >> > > >     >     >     > > > > > > > > > >> > >>        at
> > > > >> > > >     >     >     > > > > > > > > > >> > >>
> > > > >> > > >     >     >     > > > > > > > > > >> >
> > > > >> > > >     >     >     > > > > > > > > > >>
> > > > >> > > >     >     >     > > > > > > > > >
> > > > >> > > >     >     >     > > > > > > > >
> > > > >> > > >     >     >     > > > > > > >
> > > > >> > > >     >     >     > > > > > >
> > > > >> > > >     >     >     > > > > >
> > > > >> > > >     >     >     > > > >
> > > > >> > > >     >     >     > > >
> > > > >> > > >     >     >     > >
> > > > >> > > >     >     >     >
> > > > >> > > >     >     >
> > > > >> > > >     >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
> > > > >> > > >     >     >     > > > > > > > > > >> > >>        at
> > > > >> > > >     >     >     > > > > > > > > >
> > > > >> > > >     >     >
> > > > >> java.util.concurrent.Executors$RunnableAdapter.call(Unknown
> > > > >> > > >     >     >     > > > > > > > > > >> > >> Source)
> > > > >> > > >     >     >     > > > > > > > > > >> > >>        at
> > > > >> > > >     >     >     >
> java.util.concurrent.FutureTask.run(Unknown
> > > > >> > > >     >     >     > > > > > Source)
> > > > >> > > >     >     >     > > > > > > > > > >> > >>        at
> > > > >> > > >     >     >     > > > > > > > >
> > > > >> > > >     >     >
> > > > >> java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> > > > >> > > >     >     >     > > > > > > > > > >> > >> Source)
> > > > >> > > >     >     >     > > > > > > > > > >> > >>        at
> > > > >> > > >     >     >     > > > > > > > > >
> > > > >> > > >     >     >
> > > > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> > > > >> > > >     >     >     > > > > > > > > > >> > >> Source)
> > > > >> > > >     >     >     > > > > > > > > > >> > >>        at
> > > > >> > > > java.lang.Thread.run(Unknown
> > > > >> > > >     >     > Source)
> > > > >> > > >     >     >     > > > > > > > > > >> > >> 2019-03-19
> > > 14:20:39,659
> > > > >> DEBUG
> > > > >> > > >     >     >     > > > > > >
> [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > > >> > > >     >     >     > > > > > > > > > >> > >>
> > > > >> > > (API-Job-Executor-4:ctx-7b6b69eb
> > > > >> > > >     > job-5093)
> > > > >> > > >     >     >     > > > > (logid:265a6099)
> > > > >> > > >     >     >     > > > > > > > > > Complete
> > > > >> > > >     >     >     > > > > > > > > > >> > async
> > > > >> > > >     >     >     > > > > > > > > > >> > >> job-5093,
> jobStatus:
> > > > >> FAILED,
> > > > >> > > >     > resultCode: 530,
> > > > >> > > >     >     >     > result:
> > > > >> > > >     >     >     > > > > > > > > > >> > >>
> > > > >> > > >     >     >     > > > > > > > > > >> >
> > > > >> > > >     >     >     > > > > > > > > > >>
> > > > >> > > >     >     >     > > > > > > > > >
> > > > >> > > >     >     >     > > > > > > > >
> > > > >> > > >     >     >     > > > > > > >
> > > > >> > > >     >     >     > > > > > >
> > > > >> > > >     >     >     > > > > >
> > > > >> > > >     >     >     > > > >
> > > > >> > > >     >     >     > > >
> > > > >> > > >     >     >     > >
> > > > >> > > >     >     >     >
> > > > >> > > >     >     >
> > > > >> > > >     >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Resource
> > > > >> > > >     >     >     > > > > > > > > > >> > >> [DataCenter:1] is
> un
> > > > >> > > >     >     >     > > > > > > > > > >> > >>
> > > > >> > > >     >     >     > > > > > > > > > >> > >> On Tue, Mar 19,
> 2019
> > > at
> > > > >> 4:19
> > > > >> > PM
> > > > >> > > > Andrija
> > > > >> > > >     >     > Panic <
> > > > >> > > >     >     >     > > > > > > > > > >> >
> > > > andrija.pa...@shapeblue.com>
> > > > >> > > >     >     >     > > > > > > > > > >> > >> wrote:
> > > > >> > > >     >     >     > > > > > > > > > >> > >>
> > > > >> > > >     >     >     > > > > > > > > > >> > >>> ​​
> > > > >> > > >     >     >     > > > > > > > > > >> > >>> Your network
> can't
> > be
> > > > >> > deleted
> > > > >> > > > due to
> > > > >> > > >     > "Can't
> > > > >> > > >     >     > delete
> > > > >> > > >     >     >     > > the
> > > > >> > > >     >     >     > > > > > > > network,
> > > > >> > > >     >     >     > > > > > > > > > not
> > > > >> > > >     >     >     > > > > > > > > > >> all
> > > > >> > > >     >     >     > > > > > > > > > >> > >>> user vms are
> > > expunged.
> > > > Vm
> > > > >> > > >     >     >     > > > > > > > > > >> > >>>
> VM[User|i-2-11-VM]
> > is
> > > > in
> > > > >> > > Stopped
> > > > >> > > >     > state" -
> > > > >> > > >     >     > which is
> > > > >> > > >     >     >     > > > fine.
> > > > >> > > >     >     >     > > > > > > > > > >> > >>>
> > > > >> > > >     >     >     > > > > > > > > > >> > >>> You should be
> able
> > to
> > > > >> just
> > > > >> > > > start the
> > > > >> > > >     > user
> > > > >> > > >     >     > VM - but
> > > > >> > > >     >     >     > > if
> > > > >> > > >     >     >     > > > > you
> > > > >> > > >     >     >     > > > > > > have
> > > > >> > > >     >     >     > > > > > > > > > >> actually
> > > > >> > > >     >     >     > > > > > > > > > >> > >>> delete the VR
> > itself,
> > > > >> then
> > > > >> > > just
> > > > >> > > > do
> > > > >> > > >     > Network
> > > > >> > > >     >     > restart
> > > > >> > > >     >     >     > > > with
> > > > >> > > >     >     >     > > > > > > > > "cleanup"
> > > > >> > > >     >     >     > > > > > > > > > >> and
> > > > >> > > >     >     >     > > > > > > > > > >> > it
> > > > >> > > >     >     >     > > > > > > > > > >> > >>> will recreate a
> new
> > > VR,
> > > > >> > after
> > > > >> > > > which
> > > > >> > > >     > you
> > > > >> > > >     >     > should be
> > > > >> > > >     >     >     > > able
> > > > >> > > >     >     >     > > > > to
> > > > >> > > >     >     >     > > > > > > > start
> > > > >> > > >     >     >     > > > > > > > > > the
> > > > >> > > >     >     >     > > > > > > > > > >> VM.
> > > > >> > > >     >     >     > > > > > > > > > >> > >>>
> > > > >> > > >     >     >     > > > > > > > > > >> > >>> Andrija
> > > > >> > > >     >     >     > > > > > > > > > >> > >>>
> > > > >> > > >     >     >     > > > > > > > > > >> > >>>
> > > > >> andrija.pa...@shapeblue.com
> > > > >> > > >     >     >     > > > > > > > > > >> > >>>
> www.shapeblue.com
> > > > >> > > >     >     >     > > > > > > > > > >> > >>> Amadeus House,
> > Floral
> > > > >> > Street,
> > > > >> > > > London
> > > > >> > > >     > WC2E
> > > > >> > > >     >     > 9DPUK
> > > > >> > > >     >     >     > > > > > > > > > >> > >>> @shapeblue
> > > > >> > > >     >     >     > > > > > > > > > >> > >>>
> > > > >> > > >     >     >     > > > > > > > > > >> > >>>
> > > > >> > > >     >     >     > > > > > > > > > >> > >>>
> > > > >> > > >     >     >     > > > > > > > > > >> > >>>
> > > > >> > > >     >     >     > > > > > > > > > >> > >>> -----Original
> > > > >> Message-----
> > > > >> > > >     >     >     > > > > > > > > > >> > >>> From: Jevgeni
> > > > Zolotarjov
> > > > >> <
> > > > >> > > >     >     > j.zolotar...@gmail.com>
> > > > >> > > >     >     >     > > > > > > > > > >> > >>> Sent: 19 March
> 2019
> > > > 15:10
> > > > >> > > >     >     >     > > > > > > > > > >> > >>> To:
> > > > >> > > users@cloudstack.apache.org
> > > > >> > > >     >     >     > > > > > > > > > >> > >>> Subject: Re:
> > Disaster
> > > > >> after
> > > > >> > > >     > maintenance
> > > > >> > > >     >     >     > > > > > > > > > >> > >>>
> > > > >> > > >     >     >     > > > > > > > > > >> > >>> I mean I cannot
> > > delete
> > > > >> > > network:
> > > > >> > > > In the
> > > > >> > > >     >     > management
> > > > >> > > >     >     >     > > > server
> > > > >> > > >     >     >     > > > > > > log I
> > > > >> > > >     >     >     > > > > > > > > see
> > > > >> > > >     >     >     > > > > > > > > > >> > >>>
> > > > >> > > >     >     >     > > > > > > > > > >> > >>>
> > > > >> > > >     >     >     > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >>
> > > > >> --
> > > > >>
> > > > >> Andrija Panić
> > > > >>
> > > > >
> > > >
> > >
> > >
> > > --
> > >
> > > Andrija Panić
> > >
> >
>
>
> --
>
> Andrija Panić
>

Reply via email to