Under this defaultGuestNetwork, I go to Virtual Appliances. There is no VMS - "no data to show"
I dont have any network, other than this single default one. I've tried adding new network - Add guest network. But I am not able to do so, cause in the wizard popup, it offers empty dropdown with Zones selection. And this wizard doesnt not allow to go further without selecting Zone On Wed, Mar 20, 2019 at 10:28 AM Andrija Panic <andrija.pa...@gmail.com> wrote: > you need to delete/remove all VMs inside this network (tick the "Expunge" > button during VM deletion - if you want to really delete the VMs) in order > to be able to delete the network - OR simply attach this VM to another > network, make this new network a DEFAULT one (NIC that is...), and then > detach from old network - and then effectively your VM was "removed" from > old network - after this you should be able to delete the old network. I > assume some DB incosistencies perhaps, being the reason you can not restart > the network. > > Did you try restarting some other Network - or deploying a new network, > spin a VM in it, then again try to restart this new network - does it work > ? > > Andrija > > On Wed, 20 Mar 2019 at 08:58, Jevgeni Zolotarjov <j.zolotar...@gmail.com> > wrote: > > > >>>Stop mgmt, > > >>>Stop all agents > > >>>Restart libvirtd (and check libvirt logs afterwards) > > >>>Start agents > > >>>Start mgmt. > > > > I did that numerous time. Nothing really suspicious > > I can see that systems VMs are running - both in cloudstack console and > > with virsh list -all > > > > It is apparently problem with network. > > Is there a way to force recreation of defaultGuestNetwork? or force > > recreation of Virtual Router. > > I am unable to delete network, which is supposed to rebuild network with > > its router. Thats the issue > > > > The issue with libvirtd was, that eventually at some point it was updated > > during 4 months of running, and not rebooted. It still worked. We had to > > add listen_tcp = 1 for libvirtd to start working again. > > > > On Wed, Mar 20, 2019 at 9:49 AM Andrija Panic <andrija.pa...@gmail.com> > > wrote: > > > > > As Sergey suggested... but i would also verify no libvirt issues or > > storage > > > pool issues - so perhaps: > > > > > > Stop mgmt, > > > Stop all agents > > > Restart libvirtd (and check libvirt logs afterwards) > > > Start agents > > > Start mgmt. > > > > > > What was originally issue with libvirtd ? > > > That sounds fishy to me... > > > > > > Andrija > > > > > > On Wed, Mar 20, 2019, 02:15 Sergey Levitskiy <serg...@hotmail.com> > > wrote: > > > > > > > select * from networks where removed is null; > > > > select * from vm_instance where id=87; > > > > select id,name from vm_instance where name like 'r%' and removed is > > null; > > > > > > > > Basically since the network offering is not redundant this error is > > only > > > > thrown when there is no router associated with your network. Usually > > > > management server restart tries to implement network again. Please > > > restart > > > > management server, save and share management server log. > > > > > > > > > > > > > > > > > > > > On 3/19/19, 3:31 PM, "Jevgeni Zolotarjov" <j.zolotar...@gmail.com> > > > wrote: > > > > > > > > >>>>Check network_offering table for value in column > > > > redundant_router_service for the network offering you use. > > > > in table network_offering_table all records have > > > > redundant_router_service = > > > > 0 > > > > > > > > Can you also run the following: > > > > >>>select name, state, removed from host where name like 'r%' > > > > returns zero rows - nothing > > > > > > > > >>>select * from domain_router; > > > > # id, element_id, public_mac_address, public_ip_address, > > > > public_netmask, > > > > guest_netmask, guest_ip_address, is_redundant_router, priority, > > > > redundant_state, stop_pending, role, template_version, > > > scripts_version, > > > > vpc_id, update_state > > > > '4', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', > '0', > > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 15:37:28 > > UTC > > > > 2018', > > > > '57db7bd8118977a5f2cd3ef1c7503633\n', NULL, NULL > > > > '49', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', > '0', > > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 15:37:28 > > UTC > > > > 2018', > > > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL > > > > '73', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', > '0', > > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 15:37:28 > > UTC > > > > 2018', > > > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL > > > > '74', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', > '0', > > > > 'VIRTUAL_ROUTER', NULL, NULL, NULL, NULL > > > > '75', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', > '0', > > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 15:37:28 > > UTC > > > > 2018', > > > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL > > > > '76', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', > '0', > > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 15:37:28 > > UTC > > > > 2018', > > > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL > > > > '77', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', > '0', > > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.1 Fri Jun 22 07:52:17 > > UTC > > > > 2018', > > > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, 'UPDATE_FAILED' > > > > '80', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', > '0', > > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.1 Fri Jun 22 07:52:17 > > UTC > > > > 2018', > > > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL > > > > '85', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', > '0', > > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.1 Fri Jun 22 07:52:17 > > UTC > > > > 2018', > > > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL > > > > '86', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', > '0', > > > > 'VIRTUAL_ROUTER', NULL, NULL, NULL, NULL > > > > '87', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', > '0', > > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.2.0 Mon Nov 12 > 15:06:49 > > > UTC > > > > 2018', '873057731ff2cba4a1f3b2411765407c\n', NULL, NULL > > > > > > > > > > > > >>>select * from router_network_ref; > > > > # id, router_id, network_id, guest_type > > > > '1', '4', '204', 'Shared' > > > > '2', '49', '204', 'Shared' > > > > '3', '73', '204', 'Shared' > > > > '4', '75', '204', 'Shared' > > > > '5', '76', '204', 'Shared' > > > > '6', '77', '204', 'Shared' > > > > '7', '80', '204', 'Shared' > > > > '8', '85', '204', 'Shared' > > > > '9', '86', '204', 'Shared' > > > > '10', '87', '204', 'Shared' > > > > > > > > > > > > On Wed, Mar 20, 2019 at 12:18 AM Sergey Levitskiy < > > > serg...@hotmail.com > > > > > > > > > wrote: > > > > > > > > > Check network_offering table for value in column > > > > > redundant_router_service for the network offering you use. > > > > > Can you also run the following: > > > > > select name, state, removed from host where name like 'r%' > > > > > select * from domain_router; > > > > > select * from router_network_ref; > > > > > > > > > > Cloudstack is supposed to recreate you VR. If it is not > happening > > > > there is > > > > > something fundamentally wrong. I would advise to destroy your > VR > > > > again. > > > > > Stop you management server. Rotate management server log and > > start > > > it > > > > > again. If your VR doesn't start in few min, post your complete > > > > management > > > > > server log and agent log again. > > > > > > > > > > > > > > > > > > > > > > > > > On 3/19/19, 2:56 PM, "Jevgeni Zolotarjov" < > > j.zolotar...@gmail.com> > > > > wrote: > > > > > > > > > > >>>Network offering needs to be change to non-redundant > > > > > How do I do that? > > > > > > > > > > On Tue, Mar 19, 2019 at 11:47 PM Sergey Levitskiy < > > > > serg...@hotmail.com > > > > > > > > > > > wrote: > > > > > > > > > > > Network offering needs to be change to non-redundant. > Most > > > > likely > > > > > old bug > > > > > > is resurfaced. > > > > > > https://issues.apache.org/jira/browse/CLOUDSTACK-9024 > > > > > > > > > > > > > > > > > > > > > > > > On 3/19/19, 2:19 PM, "Jevgeni Zolotarjov" < > > > > j.zolotar...@gmail.com> > > > > > wrote: > > > > > > > > > > > > Hello > > > > > > > > > > > > I did exactly like you suggested. > > > > > > > > > > > > After UPDATE on db, I can see the router in > cloudstack > > > > console. > > > > > > But attempt to restart network fails. > > > > > > I get error: > > > > > > Resource [DataCenter:1] is unreachable: Can't find > all > > > > necessary > > > > > > running > > > > > > routers! > > > > > > > > > > > > I rechecked agents on both servers. They look running > > OK. > > > > > > > > > > > > On Tue, Mar 19, 2019 at 10:00 PM Andrija Panic < > > > > > > andrija.pa...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > Ok, so: > > > > > > > > > > > > > > 1.BACKUP YOUR DB - in case there is issues, you > will > > > > want to > > > > > restore > > > > > > > 2. I tried to reproduce your case on 4.11.2 (though > > > > didn't do > > > > > any > > > > > > > maintenance etc.) - and could not - I have > > artificially > > > > marked > > > > > > existing VR > > > > > > > as removed in DB - and tried to restart network and > > it > > > > worked > > > > > just > > > > > > fine. > > > > > > > > > > > > > > Let's try to "undelete" the VR for that network - > > > though > > > > I > > > > > can't be > > > > > > sure if > > > > > > > this will work or not. > > > > > > > > > > > > > > Find VR based on the network UUID: > > > > > > > SELECT * FROM vm_instance WHERE id IN (SELECT > > > > > instance_id from > > > > > > nics > > > > > > > where network_id IN (SELECT id FROM networks WHERE > > > > > > > UUID="65ca9a05-ff96-4563-ab9c-ffb610dc8b73")); > > > > > > > Obviously, replace this UUID with your network > UUID. > > > > > > > SQL above will return all VMs (user VMs, VRs, etc, > > > > delete or > > > > > not...) > > > > > > that > > > > > > > have NIC inside the network with that UUID - look > for > > > the > > > > > r-XX-VM > > > > > > being a > > > > > > > router. > > > > > > > > > > > > > > Now that you have the name of the VR (i.e. r-4-VM) > - > > > you > > > > want > > > > > to set > > > > > > 2 > > > > > > > fields to some other value than what it might be > > > > already, as > > > > > below: > > > > > > > > > > > > > > Need to set field "state" to Stopped: > > > > > > > UPDATE cloud.vm_instance SET state='Stopped' > > > WHERE > > > > > > name="r-4-VM"; > > > > > > > (what state is set currently ???) > > > > > > > > > > > > > > Need to set "removed" to NULL value: > > > > > > > UPDATE cloud.vm_instance SET removed = NULL > > WHERE > > > > > > NAME="r-4-VM"; (is > > > > > > > there a removed date already set ???) > > > > > > > > > > > > > > Obviously make sure that this VR is not running on > > any > > > > > hypervisor > > > > > > and if > > > > > > > running, virsh destroy it... > > > > > > > > > > > > > > AFTER above has been done (VR is considered to > exist > > > but > > > > > stopped, > > > > > > from > > > > > > > CloudStack point of view) - try to RESTART the > > network > > > > (don't > > > > > bother > > > > > > > deleteing it, since there are VMs in that network) > > > > > > > > > > > > > > I'm not 100% positive this will fix your issue, but > > > > doesn't > > > > > hurt to > > > > > > try > > > > > > > > > > > > > > If above doesn't work - I would still take a look > > into > > > > agents > > > > > and if > > > > > > they > > > > > > > are still connected - optionally, restart agents on > > > both > > > > hosts > > > > > once > > > > > > more > > > > > > > and confirm they are connected and up. > > > > > > > > > > > > > > Let us know how it goes (and please backup DB once > > more > > > > before > > > > > any > > > > > > actions > > > > > > > !) > > > > > > > > > > > > > > Andrija > > > > > > > > > > > > > > > > > > > > > On Tue, 19 Mar 2019 at 20:41, Jevgeni Zolotarjov < > > > > > > j.zolotar...@gmail.com> > > > > > > > wrote: > > > > > > > > > > > > > > > Yes. Just a single network. > > > > > > > > > > > > > > > > On Tue, 19 Mar 2019, 21:39 Andrija Panic, < > > > > > andrija.pa...@gmail.com > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Just one more clarification - this is Isolate > > > single > > > > > network (not > > > > > > > Shared > > > > > > > > > Network, not VPC) ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 19, 2019, 19:36 Jevgeni Zolotarjov > < > > > > > > j.zolotar...@gmail.com > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Name defaultGuestNetwork > > > > > > > > > > ID 4ba834ed-48f3-468f-b667-9bb2d2c258f1 > > > > > > > > > > Zone PBT zone 1 > > > > > > > > > > Description defaultGuestNetwork > > > > > > > > > > Type Shared > > > > > > > > > > State Setup > > > > > > > > > > VPC ID N/A > > > > > > > > > > Persistent No > > > > > > > > > > > > > > > > > > > > On Tue, Mar 19, 2019 at 8:29 PM Andrija > Panic < > > > > > > > andrija.pa...@gmail.com > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Share the network id and name as seen from > > > GUI... > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 19, 2019, 19:27 Jevgeni > > Zolotarjov > > > < > > > > > > > > j.zolotar...@gmail.com > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > >>> 1. Confirm please rolling restart is > > set > > > > to false > > > > > > please - > > > > > > > > double > > > > > > > > > > > check > > > > > > > > > > > > Double checked - It is set to false > > > > > > > > > > > > > > > > > > > > > > > > >>>>2. If so - do you know the name of VR > > > > which you > > > > > > deleted ? Is > > > > > > > it > > > > > > > > > > last > > > > > > > > > > > > one > > > > > > > > > > > > >>> ever created - if so we can find it > > > > easily... > > > > > > > > > > > > I dont know the name > > > > > > > > > > > > Is there a way to fetch it from DB? > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 19, 2019 at 7:58 PM Andrija > > > Panic < > > > > > > > > > andrija.pa...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Right...still complaining on missing > > > running > > > > > routers. > > > > > > > > > > > > > > > > > > > > > > > > > > 1. Confirm please rolling restart is > set > > to > > > > false > > > > > please > > > > > > - > > > > > > > double > > > > > > > > > > check > > > > > > > > > > > > > 2. If so - do you know the name of VR > > which > > > > you > > > > > deleted > > > > > > ? Is it > > > > > > > > > last > > > > > > > > > > > one > > > > > > > > > > > > > ever created - if so we can find it > > > easily... > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 19, 2019, 18:40 Jevgeni > > > > Zolotarjov < > > > > > > > > > > j.zolotar...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Here is management server log > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://drive.google.com/open?id=1H2jI0roeiWxtzReB8qV6QxDkNpaki99A > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 19, 2019 at 7:29 PM > Andrija > > > > Panic < > > > > > > > > > > > andrija.pa...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I would also try to "undelete" VR > in > > > DB, > > > > but > > > > > let's > > > > > > keep > > > > > > > this > > > > > > > > as > > > > > > > > > > > last > > > > > > > > > > > > > > step. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 19, 2019, 18:24 Andrija > > > > Panic < > > > > > > > > > > andrija.pa...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Disclaimer: from mobile device. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I don't see any special failure > in > > > > agent log, > > > > > > except some > > > > > > > > > long > > > > > > > > > > > > > running > > > > > > > > > > > > > > > > migration, timeout for graceful > VM > > > > shutdown > > > > > etc > > > > > > (and > > > > > > > agent > > > > > > > > > > > restart) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > You did not send mgmt log after > > last > > > > network > > > > > > restart > > > > > > > > failure > > > > > > > > > ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 19, 2019, 17:29 > Jevgeni > > > > > Zolotarjov < > > > > > > > > > > > > > j.zolotar...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Under > > > > > > > > > > > > > > > >> Infrastructure / Hosts: > > > > > > > > > > > > > > > >> both hosts are enabled and green > > > > (Unsecure > > > > > though) > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> agent.log - > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > https://drive.google.com/open?id=1rATxHKqgNKo2kD23BtlrZy_9gFXC-Bq- > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> I set network.rolling.restart to > > > > false now. > > > > > > Restarted > > > > > > > > > > management > > > > > > > > > > > > > > server > > > > > > > > > > > > > > > - > > > > > > > > > > > > > > > >> same problem - cannot restart > not > > > > delete > > > > > network > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> On Tue, Mar 19, 2019 at 5:48 PM > > > Boris > > > > > Stoyanov < > > > > > > > > > > > > > > > >> boris.stoya...@shapeblue.com> > > > > > > > > > > > > > > > >> wrote: > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > Hi, you shouldn’t be worried > > about > > > > data > > > > > as long > > > > > > as you > > > > > > > > > use a > > > > > > > > > > > > > > separate > > > > > > > > > > > > > > > >> > shared storage. > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > What does the cloudstack > console > > > say > > > > > about your > > > > > > host? > > > > > > > Is > > > > > > > > > it > > > > > > > > > > > up? > > > > > > > > > > > > If > > > > > > > > > > > > > > > it’s > > > > > > > > > > > > > > > >> up > > > > > > > > > > > > > > > >> > then you should be able to > > deploy > > > a > > > > VM on > > > > > it. > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > If not, you’ll need to > > investigate > > > > the > > > > > reason > > > > > > > > > > cloudstack-agent > > > > > > > > > > > > is > > > > > > > > > > > > > > not > > > > > > > > > > > > > > > >> > connected, is the agent > running? > > > If > > > > not > > > > > just > > > > > > start it > > > > > > > > and > > > > > > > > > > see > > > > > > > > > > > if > > > > > > > > > > > > > it > > > > > > > > > > > > > > > >> > connects. > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > If it’s running but still not > > > > connected > > > > > please > > > > > > share > > > > > > > the > > > > > > > > > > > > > agent.log. > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > If nothing helps at all you > can > > > > remove > > > > > the host > > > > > > and > > > > > > > add > > > > > > > > it > > > > > > > > > > > > again, > > > > > > > > > > > > > > but > > > > > > > > > > > > > > > >> > let’s try to troubleshoot > first. > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > Bobby. > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > boris.stoya...@shapeblue.com > > > > > > > > > > > > > > > >> > www.shapeblue.com > > > > > > > > > > > > > > > >> > Amadeus House, Floral Street, > > > London > > > > > WC2E 9DPUK > > > > > > > > > > > > > > > >> > @shapeblue > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > On 19 Mar 2019, at 17:29, > > > Jevgeni > > > > > Zolotarjov < > > > > > > > > > > > > > > > j.zolotar...@gmail.com> > > > > > > > > > > > > > > > >> > wrote: > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > Guys, please help with it. > > > > > > > > > > > > > > > >> > > What can be done here? > > > > > > > > > > > > > > > >> > > There is too much valuable > > data. > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > On Tue, Mar 19, 2019 at 4:21 > > PM > > > > Jevgeni > > > > > > Zolotarjov < > > > > > > > > > > > > > > > >> > j.zolotar...@gmail.com> > > > > > > > > > > > > > > > >> > > wrote: > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > >> Tried that just now and got > > > > error: > > > > > > > > > > > > > > > >> > >> Resource [DataCenter:1] is > > > > unreachable: > > > > > > Can't find > > > > > > > > all > > > > > > > > > > > > > necessary > > > > > > > > > > > > > > > >> running > > > > > > > > > > > > > > > >> > >> routers! > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > >> In the log I see: > > > > > > > > > > > > > > > >> > >> > > > > > ============================================= > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > >> 2019-03-19 14:20:39,644 > DEBUG > > > > > > > > > > > [o.a.c.e.o.NetworkOrchestrator] > > > > > > > > > > > > > > > >> > >> > > > (API-Job-Executor-4:ctx-7b6b69eb > > > > > job-5093 > > > > > > > > ctx-9be30648) > > > > > > > > > > > > > > > >> (logid:265a6099) > > > > > > > > > > > > > > > >> > >> Restarting network 204... > > > > > > > > > > > > > > > >> > >> 2019-03-19 14:20:39,645 > DEBUG > > > > > > > > > > > [o.a.c.e.o.NetworkOrchestrator] > > > > > > > > > > > > > > > >> > >> > > > (API-Job-Executor-4:ctx-7b6b69eb > > > > > job-5093 > > > > > > > > ctx-9be30648) > > > > > > > > > > > > > > > >> (logid:265a6099) > > > > > > > > > > > > > > > >> > >> Performing rolling restart > of > > > > routers > > > > > of > > > > > > network > > > > > > > > > > > > > > Ntwk[204|Guest|6] > > > > > > > > > > > > > > > >> > >> 2019-03-19 14:20:39,649 > DEBUG > > > > > > > > > > > [o.a.c.e.o.NetworkOrchestrator] > > > > > > > > > > > > > > > >> > >> > > > (API-Job-Executor-4:ctx-7b6b69eb > > > > > job-5093 > > > > > > > > ctx-9be30648) > > > > > > > > > > > > > > > >> (logid:265a6099) > > > > > > > > > > > > > > > >> > >> Asking VirtualRouter to > > > > implemenet > > > > > > > Ntwk[204|Guest|6] > > > > > > > > > > > > > > > >> > >> 2019-03-19 14:20:39,658 > ERROR > > > > > > > > > > [c.c.a.ApiAsyncJobDispatcher] > > > > > > > > > > > > > > > >> > >> > > > (API-Job-Executor-4:ctx-7b6b69eb > > > > > job-5093) > > > > > > > > > > (logid:265a6099) > > > > > > > > > > > > > > > >> Unexpected > > > > > > > > > > > > > > > >> > >> exception while executing > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.api.command.user.network.RestartNetworkCmd > > > > > > > > > > > > > > > >> > >> > > > > > > com.cloud.exception.ResourceUnavailableException: > > > > > > > > > > Resource > > > > > > > > > > > > > > > >> > [DataCenter:1] > > > > > > > > > > > > > > > >> > >> is unreachable: Can't find > > all > > > > > necessary > > > > > > running > > > > > > > > > routers! > > > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:243) > > > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElements(NetworkOrchestrator.java:1203) > > > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.rollingRestartRouters(NetworkOrchestrator.java:2948) > > > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:2853) > > > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1883) > > > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native > > > > > > > > > > > > > > > Method) > > > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(Unknown > > > > > > > > > > > > > > > Source) > > > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown > > > > > > > > > > > > > > > >> > Source) > > > > > > > > > > > > > > > >> > >> at > > > > > > java.lang.reflect.Method.invoke(Unknown > > > > > > > > > Source) > > > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338) > > > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197) > > > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) > > > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107) > > > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) > > > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51) > > > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) > > > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) > > > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) > > > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) > > > > > > > > > > > > > > > >> > >> at > > > > > > > > > com.sun.proxy.$Proxy229.restartNetwork(Unknown > > > > > > > > > > > > > Source) > > > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:99) > > > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > > > >> > > > > > > > > > > > > com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150) > > > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108) > > > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581) > > > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > > > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > > > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > > > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > > > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > > > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529) > > > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > > > > > > > > > > java.util.concurrent.Executors$RunnableAdapter.call(Unknown > > > > > > > > > > > > > > > >> > >> Source) > > > > > > > > > > > > > > > >> > >> at > > > > > > > java.util.concurrent.FutureTask.run(Unknown > > > > > > > > > > > Source) > > > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown > > > > > > > > > > > > > > > >> > >> Source) > > > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown > > > > > > > > > > > > > > > >> > >> Source) > > > > > > > > > > > > > > > >> > >> at > > > > java.lang.Thread.run(Unknown > > > > > > Source) > > > > > > > > > > > > > > > >> > >> 2019-03-19 14:20:39,659 > DEBUG > > > > > > > > > > > > [o.a.c.f.j.i.AsyncJobManagerImpl] > > > > > > > > > > > > > > > >> > >> > > > (API-Job-Executor-4:ctx-7b6b69eb > > > > > job-5093) > > > > > > > > > > (logid:265a6099) > > > > > > > > > > > > > > > Complete > > > > > > > > > > > > > > > >> > async > > > > > > > > > > > > > > > >> > >> job-5093, jobStatus: > FAILED, > > > > > resultCode: 530, > > > > > > > result: > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Resource > > > > > > > > > > > > > > > >> > >> [DataCenter:1] is un > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > >> On Tue, Mar 19, 2019 at > 4:19 > > PM > > > > Andrija > > > > > > Panic < > > > > > > > > > > > > > > > >> > andrija.pa...@shapeblue.com> > > > > > > > > > > > > > > > >> > >> wrote: > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > >>> > > > > > > > > > > > > > > > >> > >>> Your network can't be > > deleted > > > > due to > > > > > "Can't > > > > > > delete > > > > > > > > the > > > > > > > > > > > > > network, > > > > > > > > > > > > > > > not > > > > > > > > > > > > > > > >> all > > > > > > > > > > > > > > > >> > >>> user vms are expunged. Vm > > > > > > > > > > > > > > > >> > >>> VM[User|i-2-11-VM] is in > > > Stopped > > > > > state" - > > > > > > which is > > > > > > > > > fine. > > > > > > > > > > > > > > > >> > >>> > > > > > > > > > > > > > > > >> > >>> You should be able to just > > > > start the > > > > > user > > > > > > VM - but > > > > > > > > if > > > > > > > > > > you > > > > > > > > > > > > have > > > > > > > > > > > > > > > >> actually > > > > > > > > > > > > > > > >> > >>> delete the VR itself, then > > > just > > > > do > > > > > Network > > > > > > restart > > > > > > > > > with > > > > > > > > > > > > > > "cleanup" > > > > > > > > > > > > > > > >> and > > > > > > > > > > > > > > > >> > it > > > > > > > > > > > > > > > >> > >>> will recreate a new VR, > > after > > > > which > > > > > you > > > > > > should be > > > > > > > > able > > > > > > > > > > to > > > > > > > > > > > > > start > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > >> VM. > > > > > > > > > > > > > > > >> > >>> > > > > > > > > > > > > > > > >> > >>> Andrija > > > > > > > > > > > > > > > >> > >>> > > > > > > > > > > > > > > > >> > >>> > andrija.pa...@shapeblue.com > > > > > > > > > > > > > > > >> > >>> www.shapeblue.com > > > > > > > > > > > > > > > >> > >>> Amadeus House, Floral > > Street, > > > > London > > > > > WC2E > > > > > > 9DPUK > > > > > > > > > > > > > > > >> > >>> @shapeblue > > > > > > > > > > > > > > > >> > >>> > > > > > > > > > > > > > > > >> > >>> > > > > > > > > > > > > > > > >> > >>> > > > > > > > > > > > > > > > >> > >>> > > > > > > > > > > > > > > > >> > >>> -----Original Message----- > > > > > > > > > > > > > > > >> > >>> From: Jevgeni Zolotarjov < > > > > > > j.zolotar...@gmail.com> > > > > > > > > > > > > > > > >> > >>> Sent: 19 March 2019 15:10 > > > > > > > > > > > > > > > >> > >>> To: > > > users@cloudstack.apache.org > > > > > > > > > > > > > > > >> > >>> Subject: Re: Disaster > after > > > > > maintenance > > > > > > > > > > > > > > > >> > >>> > > > > > > > > > > > > > > > >> > >>> I mean I cannot delete > > > network: > > > > In the > > > > > > management > > > > > > > > > server > > > > > > > > > > > > log I > > > > > > > > > > > > > > see > > > > > > > > > > > > > > > >> > >>> > > > > > > > > > > > > > > > >> > >>> > > > > > > > > > > > > > > > > > -- > > Andrija Panić >
