>>>Stop mgmt, >>>Stop all agents >>>Restart libvirtd (and check libvirt logs afterwards) >>>Start agents >>>Start mgmt.
I did that numerous time. Nothing really suspicious I can see that systems VMs are running - both in cloudstack console and with virsh list -all It is apparently problem with network. Is there a way to force recreation of defaultGuestNetwork? or force recreation of Virtual Router. I am unable to delete network, which is supposed to rebuild network with its router. Thats the issue The issue with libvirtd was, that eventually at some point it was updated during 4 months of running, and not rebooted. It still worked. We had to add listen_tcp = 1 for libvirtd to start working again. On Wed, Mar 20, 2019 at 9:49 AM Andrija Panic <andrija.pa...@gmail.com> wrote: > As Sergey suggested... but i would also verify no libvirt issues or storage > pool issues - so perhaps: > > Stop mgmt, > Stop all agents > Restart libvirtd (and check libvirt logs afterwards) > Start agents > Start mgmt. > > What was originally issue with libvirtd ? > That sounds fishy to me... > > Andrija > > On Wed, Mar 20, 2019, 02:15 Sergey Levitskiy <serg...@hotmail.com> wrote: > > > select * from networks where removed is null; > > select * from vm_instance where id=87; > > select id,name from vm_instance where name like 'r%' and removed is null; > > > > Basically since the network offering is not redundant this error is only > > thrown when there is no router associated with your network. Usually > > management server restart tries to implement network again. Please > restart > > management server, save and share management server log. > > > > > > > > > > On 3/19/19, 3:31 PM, "Jevgeni Zolotarjov" <j.zolotar...@gmail.com> > wrote: > > > > >>>>Check network_offering table for value in column > > redundant_router_service for the network offering you use. > > in table network_offering_table all records have > > redundant_router_service = > > 0 > > > > Can you also run the following: > > >>>select name, state, removed from host where name like 'r%' > > returns zero rows - nothing > > > > >>>select * from domain_router; > > # id, element_id, public_mac_address, public_ip_address, > > public_netmask, > > guest_netmask, guest_ip_address, is_redundant_router, priority, > > redundant_state, stop_pending, role, template_version, > scripts_version, > > vpc_id, update_state > > '4', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0', > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 15:37:28 UTC > > 2018', > > '57db7bd8118977a5f2cd3ef1c7503633\n', NULL, NULL > > '49', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0', > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 15:37:28 UTC > > 2018', > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL > > '73', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0', > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 15:37:28 UTC > > 2018', > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL > > '74', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0', > > 'VIRTUAL_ROUTER', NULL, NULL, NULL, NULL > > '75', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0', > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 15:37:28 UTC > > 2018', > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL > > '76', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0', > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 15:37:28 UTC > > 2018', > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL > > '77', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0', > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.1 Fri Jun 22 07:52:17 UTC > > 2018', > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, 'UPDATE_FAILED' > > '80', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0', > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.1 Fri Jun 22 07:52:17 UTC > > 2018', > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL > > '85', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0', > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.1 Fri Jun 22 07:52:17 UTC > > 2018', > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL > > '86', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0', > > 'VIRTUAL_ROUTER', NULL, NULL, NULL, NULL > > '87', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0', > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.2.0 Mon Nov 12 15:06:49 > UTC > > 2018', '873057731ff2cba4a1f3b2411765407c\n', NULL, NULL > > > > > > >>>select * from router_network_ref; > > # id, router_id, network_id, guest_type > > '1', '4', '204', 'Shared' > > '2', '49', '204', 'Shared' > > '3', '73', '204', 'Shared' > > '4', '75', '204', 'Shared' > > '5', '76', '204', 'Shared' > > '6', '77', '204', 'Shared' > > '7', '80', '204', 'Shared' > > '8', '85', '204', 'Shared' > > '9', '86', '204', 'Shared' > > '10', '87', '204', 'Shared' > > > > > > On Wed, Mar 20, 2019 at 12:18 AM Sergey Levitskiy < > serg...@hotmail.com > > > > > wrote: > > > > > Check network_offering table for value in column > > > redundant_router_service for the network offering you use. > > > Can you also run the following: > > > select name, state, removed from host where name like 'r%' > > > select * from domain_router; > > > select * from router_network_ref; > > > > > > Cloudstack is supposed to recreate you VR. If it is not happening > > there is > > > something fundamentally wrong. I would advise to destroy your VR > > again. > > > Stop you management server. Rotate management server log and start > it > > > again. If your VR doesn't start in few min, post your complete > > management > > > server log and agent log again. > > > > > > > > > > > > > > > On 3/19/19, 2:56 PM, "Jevgeni Zolotarjov" <j.zolotar...@gmail.com> > > wrote: > > > > > > >>>Network offering needs to be change to non-redundant > > > How do I do that? > > > > > > On Tue, Mar 19, 2019 at 11:47 PM Sergey Levitskiy < > > serg...@hotmail.com > > > > > > > wrote: > > > > > > > Network offering needs to be change to non-redundant. Most > > likely > > > old bug > > > > is resurfaced. > > > > https://issues.apache.org/jira/browse/CLOUDSTACK-9024 > > > > > > > > > > > > > > > > On 3/19/19, 2:19 PM, "Jevgeni Zolotarjov" < > > j.zolotar...@gmail.com> > > > wrote: > > > > > > > > Hello > > > > > > > > I did exactly like you suggested. > > > > > > > > After UPDATE on db, I can see the router in cloudstack > > console. > > > > But attempt to restart network fails. > > > > I get error: > > > > Resource [DataCenter:1] is unreachable: Can't find all > > necessary > > > > running > > > > routers! > > > > > > > > I rechecked agents on both servers. They look running OK. > > > > > > > > On Tue, Mar 19, 2019 at 10:00 PM Andrija Panic < > > > > andrija.pa...@gmail.com> > > > > wrote: > > > > > > > > > Ok, so: > > > > > > > > > > 1.BACKUP YOUR DB - in case there is issues, you will > > want to > > > restore > > > > > 2. I tried to reproduce your case on 4.11.2 (though > > didn't do > > > any > > > > > maintenance etc.) - and could not - I have artificially > > marked > > > > existing VR > > > > > as removed in DB - and tried to restart network and it > > worked > > > just > > > > fine. > > > > > > > > > > Let's try to "undelete" the VR for that network - > though > > I > > > can't be > > > > sure if > > > > > this will work or not. > > > > > > > > > > Find VR based on the network UUID: > > > > > SELECT * FROM vm_instance WHERE id IN (SELECT > > > instance_id from > > > > nics > > > > > where network_id IN (SELECT id FROM networks WHERE > > > > > UUID="65ca9a05-ff96-4563-ab9c-ffb610dc8b73")); > > > > > Obviously, replace this UUID with your network UUID. > > > > > SQL above will return all VMs (user VMs, VRs, etc, > > delete or > > > not...) > > > > that > > > > > have NIC inside the network with that UUID - look for > the > > > r-XX-VM > > > > being a > > > > > router. > > > > > > > > > > Now that you have the name of the VR (i.e. r-4-VM) - > you > > want > > > to set > > > > 2 > > > > > fields to some other value than what it might be > > already, as > > > below: > > > > > > > > > > Need to set field "state" to Stopped: > > > > > UPDATE cloud.vm_instance SET state='Stopped' > WHERE > > > > name="r-4-VM"; > > > > > (what state is set currently ???) > > > > > > > > > > Need to set "removed" to NULL value: > > > > > UPDATE cloud.vm_instance SET removed = NULL WHERE > > > > NAME="r-4-VM"; (is > > > > > there a removed date already set ???) > > > > > > > > > > Obviously make sure that this VR is not running on any > > > hypervisor > > > > and if > > > > > running, virsh destroy it... > > > > > > > > > > AFTER above has been done (VR is considered to exist > but > > > stopped, > > > > from > > > > > CloudStack point of view) - try to RESTART the network > > (don't > > > bother > > > > > deleteing it, since there are VMs in that network) > > > > > > > > > > I'm not 100% positive this will fix your issue, but > > doesn't > > > hurt to > > > > try > > > > > > > > > > If above doesn't work - I would still take a look into > > agents > > > and if > > > > they > > > > > are still connected - optionally, restart agents on > both > > hosts > > > once > > > > more > > > > > and confirm they are connected and up. > > > > > > > > > > Let us know how it goes (and please backup DB once more > > before > > > any > > > > actions > > > > > !) > > > > > > > > > > Andrija > > > > > > > > > > > > > > > On Tue, 19 Mar 2019 at 20:41, Jevgeni Zolotarjov < > > > > j.zolotar...@gmail.com> > > > > > wrote: > > > > > > > > > > > Yes. Just a single network. > > > > > > > > > > > > On Tue, 19 Mar 2019, 21:39 Andrija Panic, < > > > andrija.pa...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > > > Just one more clarification - this is Isolate > single > > > network (not > > > > > Shared > > > > > > > Network, not VPC) ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 19, 2019, 19:36 Jevgeni Zolotarjov < > > > > j.zolotar...@gmail.com > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Name defaultGuestNetwork > > > > > > > > ID 4ba834ed-48f3-468f-b667-9bb2d2c258f1 > > > > > > > > Zone PBT zone 1 > > > > > > > > Description defaultGuestNetwork > > > > > > > > Type Shared > > > > > > > > State Setup > > > > > > > > VPC ID N/A > > > > > > > > Persistent No > > > > > > > > > > > > > > > > On Tue, Mar 19, 2019 at 8:29 PM Andrija Panic < > > > > > andrija.pa...@gmail.com > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Share the network id and name as seen from > GUI... > > > > > > > > > > > > > > > > > > On Tue, Mar 19, 2019, 19:27 Jevgeni Zolotarjov > < > > > > > > j.zolotar...@gmail.com > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > >>> 1. Confirm please rolling restart is set > > to false > > > > please - > > > > > > double > > > > > > > > > check > > > > > > > > > > Double checked - It is set to false > > > > > > > > > > > > > > > > > > > > >>>>2. If so - do you know the name of VR > > which you > > > > deleted ? Is > > > > > it > > > > > > > > last > > > > > > > > > > one > > > > > > > > > > >>> ever created - if so we can find it > > easily... > > > > > > > > > > I dont know the name > > > > > > > > > > Is there a way to fetch it from DB? > > > > > > > > > > > > > > > > > > > > On Tue, Mar 19, 2019 at 7:58 PM Andrija > Panic < > > > > > > > andrija.pa...@gmail.com > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Right...still complaining on missing > running > > > routers. > > > > > > > > > > > > > > > > > > > > > > 1. Confirm please rolling restart is set to > > false > > > please > > > > - > > > > > double > > > > > > > > check > > > > > > > > > > > 2. If so - do you know the name of VR which > > you > > > deleted > > > > ? Is it > > > > > > > last > > > > > > > > > one > > > > > > > > > > > ever created - if so we can find it > easily... > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 19, 2019, 18:40 Jevgeni > > Zolotarjov < > > > > > > > > j.zolotar...@gmail.com > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Here is management server log > > > > > > > > > > > > > > > > > > > > > > > > > https://drive.google.com/open?id=1H2jI0roeiWxtzReB8qV6QxDkNpaki99A > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 19, 2019 at 7:29 PM Andrija > > Panic < > > > > > > > > > andrija.pa...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > I would also try to "undelete" VR in > DB, > > but > > > let's > > > > keep > > > > > this > > > > > > as > > > > > > > > > last > > > > > > > > > > > > step. > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 19, 2019, 18:24 Andrija > > Panic < > > > > > > > > andrija.pa...@gmail.com > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Disclaimer: from mobile device. > > > > > > > > > > > > > > > > > > > > > > > > > > > > I don't see any special failure in > > agent log, > > > > except some > > > > > > > long > > > > > > > > > > > running > > > > > > > > > > > > > > migration, timeout for graceful VM > > shutdown > > > etc > > > > (and > > > > > agent > > > > > > > > > restart) > > > > > > > > > > > > > > > > > > > > > > > > > > > > You did not send mgmt log after last > > network > > > > restart > > > > > > failure > > > > > > > ? > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 19, 2019, 17:29 Jevgeni > > > Zolotarjov < > > > > > > > > > > > j.zolotar...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > >> Under > > > > > > > > > > > > > >> Infrastructure / Hosts: > > > > > > > > > > > > > >> both hosts are enabled and green > > (Unsecure > > > though) > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> agent.log - > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > https://drive.google.com/open?id=1rATxHKqgNKo2kD23BtlrZy_9gFXC-Bq- > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> I set network.rolling.restart to > > false now. > > > > Restarted > > > > > > > > management > > > > > > > > > > > > server > > > > > > > > > > > > > - > > > > > > > > > > > > > >> same problem - cannot restart not > > delete > > > network > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> On Tue, Mar 19, 2019 at 5:48 PM > Boris > > > Stoyanov < > > > > > > > > > > > > > >> boris.stoya...@shapeblue.com> > > > > > > > > > > > > > >> wrote: > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > Hi, you shouldn’t be worried about > > data > > > as long > > > > as you > > > > > > > use a > > > > > > > > > > > > separate > > > > > > > > > > > > > >> > shared storage. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > What does the cloudstack console > say > > > about your > > > > host? > > > > > Is > > > > > > > it > > > > > > > > > up? > > > > > > > > > > If > > > > > > > > > > > > > it’s > > > > > > > > > > > > > >> up > > > > > > > > > > > > > >> > then you should be able to deploy > a > > VM on > > > it. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > If not, you’ll need to investigate > > the > > > reason > > > > > > > > cloudstack-agent > > > > > > > > > > is > > > > > > > > > > > > not > > > > > > > > > > > > > >> > connected, is the agent running? > If > > not > > > just > > > > start it > > > > > > and > > > > > > > > see > > > > > > > > > if > > > > > > > > > > > it > > > > > > > > > > > > > >> > connects. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > If it’s running but still not > > connected > > > please > > > > share > > > > > the > > > > > > > > > > > agent.log. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > If nothing helps at all you can > > remove > > > the host > > > > and > > > > > add > > > > > > it > > > > > > > > > > again, > > > > > > > > > > > > but > > > > > > > > > > > > > >> > let’s try to troubleshoot first. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > Bobby. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > boris.stoya...@shapeblue.com > > > > > > > > > > > > > >> > www.shapeblue.com > > > > > > > > > > > > > >> > Amadeus House, Floral Street, > London > > > WC2E 9DPUK > > > > > > > > > > > > > >> > @shapeblue > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > On 19 Mar 2019, at 17:29, > Jevgeni > > > Zolotarjov < > > > > > > > > > > > > > j.zolotar...@gmail.com> > > > > > > > > > > > > > >> > wrote: > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > Guys, please help with it. > > > > > > > > > > > > > >> > > What can be done here? > > > > > > > > > > > > > >> > > There is too much valuable data. > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > On Tue, Mar 19, 2019 at 4:21 PM > > Jevgeni > > > > Zolotarjov < > > > > > > > > > > > > > >> > j.zolotar...@gmail.com> > > > > > > > > > > > > > >> > > wrote: > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > >> Tried that just now and got > > error: > > > > > > > > > > > > > >> > >> Resource [DataCenter:1] is > > unreachable: > > > > Can't find > > > > > > all > > > > > > > > > > > necessary > > > > > > > > > > > > > >> running > > > > > > > > > > > > > >> > >> routers! > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > >> In the log I see: > > > > > > > > > > > > > >> > >> > > > ============================================= > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > >> 2019-03-19 14:20:39,644 DEBUG > > > > > > > > > [o.a.c.e.o.NetworkOrchestrator] > > > > > > > > > > > > > >> > >> > (API-Job-Executor-4:ctx-7b6b69eb > > > job-5093 > > > > > > ctx-9be30648) > > > > > > > > > > > > > >> (logid:265a6099) > > > > > > > > > > > > > >> > >> Restarting network 204... > > > > > > > > > > > > > >> > >> 2019-03-19 14:20:39,645 DEBUG > > > > > > > > > [o.a.c.e.o.NetworkOrchestrator] > > > > > > > > > > > > > >> > >> > (API-Job-Executor-4:ctx-7b6b69eb > > > job-5093 > > > > > > ctx-9be30648) > > > > > > > > > > > > > >> (logid:265a6099) > > > > > > > > > > > > > >> > >> Performing rolling restart of > > routers > > > of > > > > network > > > > > > > > > > > > Ntwk[204|Guest|6] > > > > > > > > > > > > > >> > >> 2019-03-19 14:20:39,649 DEBUG > > > > > > > > > [o.a.c.e.o.NetworkOrchestrator] > > > > > > > > > > > > > >> > >> > (API-Job-Executor-4:ctx-7b6b69eb > > > job-5093 > > > > > > ctx-9be30648) > > > > > > > > > > > > > >> (logid:265a6099) > > > > > > > > > > > > > >> > >> Asking VirtualRouter to > > implemenet > > > > > Ntwk[204|Guest|6] > > > > > > > > > > > > > >> > >> 2019-03-19 14:20:39,658 ERROR > > > > > > > > [c.c.a.ApiAsyncJobDispatcher] > > > > > > > > > > > > > >> > >> > (API-Job-Executor-4:ctx-7b6b69eb > > > job-5093) > > > > > > > > (logid:265a6099) > > > > > > > > > > > > > >> Unexpected > > > > > > > > > > > > > >> > >> exception while executing > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.api.command.user.network.RestartNetworkCmd > > > > > > > > > > > > > >> > >> > > > > com.cloud.exception.ResourceUnavailableException: > > > > > > > > Resource > > > > > > > > > > > > > >> > [DataCenter:1] > > > > > > > > > > > > > >> > >> is unreachable: Can't find all > > > necessary > > > > running > > > > > > > routers! > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:243) > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElements(NetworkOrchestrator.java:1203) > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.rollingRestartRouters(NetworkOrchestrator.java:2948) > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:2853) > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1883) > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native > > > > > > > > > > > > > Method) > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(Unknown > > > > > > > > > > > > > Source) > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown > > > > > > > > > > > > > >> > Source) > > > > > > > > > > > > > >> > >> at > > > > java.lang.reflect.Method.invoke(Unknown > > > > > > > Source) > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338) > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197) > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107) > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51) > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) > > > > > > > > > > > > > >> > >> at > > > > > > > com.sun.proxy.$Proxy229.restartNetwork(Unknown > > > > > > > > > > > Source) > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:99) > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > >> > > > > > > > > com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150) > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108) > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581) > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529) > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > > > > > java.util.concurrent.Executors$RunnableAdapter.call(Unknown > > > > > > > > > > > > > >> > >> Source) > > > > > > > > > > > > > >> > >> at > > > > > java.util.concurrent.FutureTask.run(Unknown > > > > > > > > > Source) > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown > > > > > > > > > > > > > >> > >> Source) > > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown > > > > > > > > > > > > > >> > >> Source) > > > > > > > > > > > > > >> > >> at > > java.lang.Thread.run(Unknown > > > > Source) > > > > > > > > > > > > > >> > >> 2019-03-19 14:20:39,659 DEBUG > > > > > > > > > > [o.a.c.f.j.i.AsyncJobManagerImpl] > > > > > > > > > > > > > >> > >> > (API-Job-Executor-4:ctx-7b6b69eb > > > job-5093) > > > > > > > > (logid:265a6099) > > > > > > > > > > > > > Complete > > > > > > > > > > > > > >> > async > > > > > > > > > > > > > >> > >> job-5093, jobStatus: FAILED, > > > resultCode: 530, > > > > > result: > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Resource > > > > > > > > > > > > > >> > >> [DataCenter:1] is un > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > >> On Tue, Mar 19, 2019 at 4:19 PM > > Andrija > > > > Panic < > > > > > > > > > > > > > >> > andrija.pa...@shapeblue.com> > > > > > > > > > > > > > >> > >> wrote: > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > >>> > > > > > > > > > > > > > >> > >>> Your network can't be deleted > > due to > > > "Can't > > > > delete > > > > > > the > > > > > > > > > > > network, > > > > > > > > > > > > > not > > > > > > > > > > > > > >> all > > > > > > > > > > > > > >> > >>> user vms are expunged. Vm > > > > > > > > > > > > > >> > >>> VM[User|i-2-11-VM] is in > Stopped > > > state" - > > > > which is > > > > > > > fine. > > > > > > > > > > > > > >> > >>> > > > > > > > > > > > > > >> > >>> You should be able to just > > start the > > > user > > > > VM - but > > > > > > if > > > > > > > > you > > > > > > > > > > have > > > > > > > > > > > > > >> actually > > > > > > > > > > > > > >> > >>> delete the VR itself, then > just > > do > > > Network > > > > restart > > > > > > > with > > > > > > > > > > > > "cleanup" > > > > > > > > > > > > > >> and > > > > > > > > > > > > > >> > it > > > > > > > > > > > > > >> > >>> will recreate a new VR, after > > which > > > you > > > > should be > > > > > > able > > > > > > > > to > > > > > > > > > > > start > > > > > > > > > > > > > the > > > > > > > > > > > > > >> VM. > > > > > > > > > > > > > >> > >>> > > > > > > > > > > > > > >> > >>> Andrija > > > > > > > > > > > > > >> > >>> > > > > > > > > > > > > > >> > >>> andrija.pa...@shapeblue.com > > > > > > > > > > > > > >> > >>> www.shapeblue.com > > > > > > > > > > > > > >> > >>> Amadeus House, Floral Street, > > London > > > WC2E > > > > 9DPUK > > > > > > > > > > > > > >> > >>> @shapeblue > > > > > > > > > > > > > >> > >>> > > > > > > > > > > > > > >> > >>> > > > > > > > > > > > > > >> > >>> > > > > > > > > > > > > > >> > >>> > > > > > > > > > > > > > >> > >>> -----Original Message----- > > > > > > > > > > > > > >> > >>> From: Jevgeni Zolotarjov < > > > > j.zolotar...@gmail.com> > > > > > > > > > > > > > >> > >>> Sent: 19 March 2019 15:10 > > > > > > > > > > > > > >> > >>> To: > users@cloudstack.apache.org > > > > > > > > > > > > > >> > >>> Subject: Re: Disaster after > > > maintenance > > > > > > > > > > > > > >> > >>> > > > > > > > > > > > > > >> > >>> I mean I cannot delete > network: > > In the > > > > management > > > > > > > server > > > > > > > > > > log I > > > > > > > > > > > > see > > > > > > > > > > > > > >> > >>> > > > > > > > > > > > > > >> > >>> > > > > > > > >
