As Sergey suggested... but i would also verify no libvirt issues or storage pool issues - so perhaps:
Stop mgmt, Stop all agents Restart libvirtd (and check libvirt logs afterwards) Start agents Start mgmt. What was originally issue with libvirtd ? That sounds fishy to me... Andrija On Wed, Mar 20, 2019, 02:15 Sergey Levitskiy <serg...@hotmail.com> wrote: > select * from networks where removed is null; > select * from vm_instance where id=87; > select id,name from vm_instance where name like 'r%' and removed is null; > > Basically since the network offering is not redundant this error is only > thrown when there is no router associated with your network. Usually > management server restart tries to implement network again. Please restart > management server, save and share management server log. > > > > > On 3/19/19, 3:31 PM, "Jevgeni Zolotarjov" <j.zolotar...@gmail.com> wrote: > > >>>>Check network_offering table for value in column > redundant_router_service for the network offering you use. > in table network_offering_table all records have > redundant_router_service = > 0 > > Can you also run the following: > >>>select name, state, removed from host where name like 'r%' > returns zero rows - nothing > > >>>select * from domain_router; > # id, element_id, public_mac_address, public_ip_address, > public_netmask, > guest_netmask, guest_ip_address, is_redundant_router, priority, > redundant_state, stop_pending, role, template_version, scripts_version, > vpc_id, update_state > '4', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0', > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 15:37:28 UTC > 2018', > '57db7bd8118977a5f2cd3ef1c7503633\n', NULL, NULL > '49', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0', > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 15:37:28 UTC > 2018', > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL > '73', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0', > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 15:37:28 UTC > 2018', > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL > '74', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0', > 'VIRTUAL_ROUTER', NULL, NULL, NULL, NULL > '75', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0', > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 15:37:28 UTC > 2018', > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL > '76', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0', > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 15:37:28 UTC > 2018', > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL > '77', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0', > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.1 Fri Jun 22 07:52:17 UTC > 2018', > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, 'UPDATE_FAILED' > '80', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0', > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.1 Fri Jun 22 07:52:17 UTC > 2018', > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL > '85', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0', > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.1 Fri Jun 22 07:52:17 UTC > 2018', > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL > '86', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0', > 'VIRTUAL_ROUTER', NULL, NULL, NULL, NULL > '87', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0', > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.2.0 Mon Nov 12 15:06:49 UTC > 2018', '873057731ff2cba4a1f3b2411765407c\n', NULL, NULL > > > >>>select * from router_network_ref; > # id, router_id, network_id, guest_type > '1', '4', '204', 'Shared' > '2', '49', '204', 'Shared' > '3', '73', '204', 'Shared' > '4', '75', '204', 'Shared' > '5', '76', '204', 'Shared' > '6', '77', '204', 'Shared' > '7', '80', '204', 'Shared' > '8', '85', '204', 'Shared' > '9', '86', '204', 'Shared' > '10', '87', '204', 'Shared' > > > On Wed, Mar 20, 2019 at 12:18 AM Sergey Levitskiy <serg...@hotmail.com > > > wrote: > > > Check network_offering table for value in column > > redundant_router_service for the network offering you use. > > Can you also run the following: > > select name, state, removed from host where name like 'r%' > > select * from domain_router; > > select * from router_network_ref; > > > > Cloudstack is supposed to recreate you VR. If it is not happening > there is > > something fundamentally wrong. I would advise to destroy your VR > again. > > Stop you management server. Rotate management server log and start it > > again. If your VR doesn't start in few min, post your complete > management > > server log and agent log again. > > > > > > > > > > On 3/19/19, 2:56 PM, "Jevgeni Zolotarjov" <j.zolotar...@gmail.com> > wrote: > > > > >>>Network offering needs to be change to non-redundant > > How do I do that? > > > > On Tue, Mar 19, 2019 at 11:47 PM Sergey Levitskiy < > serg...@hotmail.com > > > > > wrote: > > > > > Network offering needs to be change to non-redundant. Most > likely > > old bug > > > is resurfaced. > > > https://issues.apache.org/jira/browse/CLOUDSTACK-9024 > > > > > > > > > > > > On 3/19/19, 2:19 PM, "Jevgeni Zolotarjov" < > j.zolotar...@gmail.com> > > wrote: > > > > > > Hello > > > > > > I did exactly like you suggested. > > > > > > After UPDATE on db, I can see the router in cloudstack > console. > > > But attempt to restart network fails. > > > I get error: > > > Resource [DataCenter:1] is unreachable: Can't find all > necessary > > > running > > > routers! > > > > > > I rechecked agents on both servers. They look running OK. > > > > > > On Tue, Mar 19, 2019 at 10:00 PM Andrija Panic < > > > andrija.pa...@gmail.com> > > > wrote: > > > > > > > Ok, so: > > > > > > > > 1.BACKUP YOUR DB - in case there is issues, you will > want to > > restore > > > > 2. I tried to reproduce your case on 4.11.2 (though > didn't do > > any > > > > maintenance etc.) - and could not - I have artificially > marked > > > existing VR > > > > as removed in DB - and tried to restart network and it > worked > > just > > > fine. > > > > > > > > Let's try to "undelete" the VR for that network - though > I > > can't be > > > sure if > > > > this will work or not. > > > > > > > > Find VR based on the network UUID: > > > > SELECT * FROM vm_instance WHERE id IN (SELECT > > instance_id from > > > nics > > > > where network_id IN (SELECT id FROM networks WHERE > > > > UUID="65ca9a05-ff96-4563-ab9c-ffb610dc8b73")); > > > > Obviously, replace this UUID with your network UUID. > > > > SQL above will return all VMs (user VMs, VRs, etc, > delete or > > not...) > > > that > > > > have NIC inside the network with that UUID - look for the > > r-XX-VM > > > being a > > > > router. > > > > > > > > Now that you have the name of the VR (i.e. r-4-VM) - you > want > > to set > > > 2 > > > > fields to some other value than what it might be > already, as > > below: > > > > > > > > Need to set field "state" to Stopped: > > > > UPDATE cloud.vm_instance SET state='Stopped' WHERE > > > name="r-4-VM"; > > > > (what state is set currently ???) > > > > > > > > Need to set "removed" to NULL value: > > > > UPDATE cloud.vm_instance SET removed = NULL WHERE > > > NAME="r-4-VM"; (is > > > > there a removed date already set ???) > > > > > > > > Obviously make sure that this VR is not running on any > > hypervisor > > > and if > > > > running, virsh destroy it... > > > > > > > > AFTER above has been done (VR is considered to exist but > > stopped, > > > from > > > > CloudStack point of view) - try to RESTART the network > (don't > > bother > > > > deleteing it, since there are VMs in that network) > > > > > > > > I'm not 100% positive this will fix your issue, but > doesn't > > hurt to > > > try > > > > > > > > If above doesn't work - I would still take a look into > agents > > and if > > > they > > > > are still connected - optionally, restart agents on both > hosts > > once > > > more > > > > and confirm they are connected and up. > > > > > > > > Let us know how it goes (and please backup DB once more > before > > any > > > actions > > > > !) > > > > > > > > Andrija > > > > > > > > > > > > On Tue, 19 Mar 2019 at 20:41, Jevgeni Zolotarjov < > > > j.zolotar...@gmail.com> > > > > wrote: > > > > > > > > > Yes. Just a single network. > > > > > > > > > > On Tue, 19 Mar 2019, 21:39 Andrija Panic, < > > andrija.pa...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > Just one more clarification - this is Isolate single > > network (not > > > > Shared > > > > > > Network, not VPC) ? > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 19, 2019, 19:36 Jevgeni Zolotarjov < > > > j.zolotar...@gmail.com > > > > > > > > > > > wrote: > > > > > > > > > > > > > Name defaultGuestNetwork > > > > > > > ID 4ba834ed-48f3-468f-b667-9bb2d2c258f1 > > > > > > > Zone PBT zone 1 > > > > > > > Description defaultGuestNetwork > > > > > > > Type Shared > > > > > > > State Setup > > > > > > > VPC ID N/A > > > > > > > Persistent No > > > > > > > > > > > > > > On Tue, Mar 19, 2019 at 8:29 PM Andrija Panic < > > > > andrija.pa...@gmail.com > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Share the network id and name as seen from GUI... > > > > > > > > > > > > > > > > On Tue, Mar 19, 2019, 19:27 Jevgeni Zolotarjov < > > > > > j.zolotar...@gmail.com > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > >>> 1. Confirm please rolling restart is set > to false > > > please - > > > > > double > > > > > > > > check > > > > > > > > > Double checked - It is set to false > > > > > > > > > > > > > > > > > > >>>>2. If so - do you know the name of VR > which you > > > deleted ? Is > > > > it > > > > > > > last > > > > > > > > > one > > > > > > > > > >>> ever created - if so we can find it > easily... > > > > > > > > > I dont know the name > > > > > > > > > Is there a way to fetch it from DB? > > > > > > > > > > > > > > > > > > On Tue, Mar 19, 2019 at 7:58 PM Andrija Panic < > > > > > > andrija.pa...@gmail.com > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Right...still complaining on missing running > > routers. > > > > > > > > > > > > > > > > > > > > 1. Confirm please rolling restart is set to > false > > please > > > - > > > > double > > > > > > > check > > > > > > > > > > 2. If so - do you know the name of VR which > you > > deleted > > > ? Is it > > > > > > last > > > > > > > > one > > > > > > > > > > ever created - if so we can find it easily... > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 19, 2019, 18:40 Jevgeni > Zolotarjov < > > > > > > > j.zolotar...@gmail.com > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Here is management server log > > > > > > > > > > > > > > > > > > > > > https://drive.google.com/open?id=1H2jI0roeiWxtzReB8qV6QxDkNpaki99A > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 19, 2019 at 7:29 PM Andrija > Panic < > > > > > > > > andrija.pa...@gmail.com > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > I would also try to "undelete" VR in DB, > but > > let's > > > keep > > > > this > > > > > as > > > > > > > > last > > > > > > > > > > > step. > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 19, 2019, 18:24 Andrija > Panic < > > > > > > > andrija.pa...@gmail.com > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Disclaimer: from mobile device. > > > > > > > > > > > > > > > > > > > > > > > > > > I don't see any special failure in > agent log, > > > except some > > > > > > long > > > > > > > > > > running > > > > > > > > > > > > > migration, timeout for graceful VM > shutdown > > etc > > > (and > > > > agent > > > > > > > > restart) > > > > > > > > > > > > > > > > > > > > > > > > > > You did not send mgmt log after last > network > > > restart > > > > > failure > > > > > > ? > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 19, 2019, 17:29 Jevgeni > > Zolotarjov < > > > > > > > > > > j.zolotar...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > >> Under > > > > > > > > > > > > >> Infrastructure / Hosts: > > > > > > > > > > > > >> both hosts are enabled and green > (Unsecure > > though) > > > > > > > > > > > > >> > > > > > > > > > > > > >> agent.log - > > > > > > > > > > > > >> > > > > > > > > > > > > > > > https://drive.google.com/open?id=1rATxHKqgNKo2kD23BtlrZy_9gFXC-Bq- > > > > > > > > > > > > >> > > > > > > > > > > > > >> I set network.rolling.restart to > false now. > > > Restarted > > > > > > > management > > > > > > > > > > > server > > > > > > > > > > > > - > > > > > > > > > > > > >> same problem - cannot restart not > delete > > network > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> On Tue, Mar 19, 2019 at 5:48 PM Boris > > Stoyanov < > > > > > > > > > > > > >> boris.stoya...@shapeblue.com> > > > > > > > > > > > > >> wrote: > > > > > > > > > > > > >> > > > > > > > > > > > > >> > Hi, you shouldn’t be worried about > data > > as long > > > as you > > > > > > use a > > > > > > > > > > > separate > > > > > > > > > > > > >> > shared storage. > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > What does the cloudstack console say > > about your > > > host? > > > > Is > > > > > > it > > > > > > > > up? > > > > > > > > > If > > > > > > > > > > > > it’s > > > > > > > > > > > > >> up > > > > > > > > > > > > >> > then you should be able to deploy a > VM on > > it. > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > If not, you’ll need to investigate > the > > reason > > > > > > > cloudstack-agent > > > > > > > > > is > > > > > > > > > > > not > > > > > > > > > > > > >> > connected, is the agent running? If > not > > just > > > start it > > > > > and > > > > > > > see > > > > > > > > if > > > > > > > > > > it > > > > > > > > > > > > >> > connects. > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > If it’s running but still not > connected > > please > > > share > > > > the > > > > > > > > > > agent.log. > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > If nothing helps at all you can > remove > > the host > > > and > > > > add > > > > > it > > > > > > > > > again, > > > > > > > > > > > but > > > > > > > > > > > > >> > let’s try to troubleshoot first. > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > Bobby. > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > boris.stoya...@shapeblue.com > > > > > > > > > > > > >> > www.shapeblue.com > > > > > > > > > > > > >> > Amadeus House, Floral Street, London > > WC2E 9DPUK > > > > > > > > > > > > >> > @shapeblue > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > On 19 Mar 2019, at 17:29, Jevgeni > > Zolotarjov < > > > > > > > > > > > > j.zolotar...@gmail.com> > > > > > > > > > > > > >> > wrote: > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > Guys, please help with it. > > > > > > > > > > > > >> > > What can be done here? > > > > > > > > > > > > >> > > There is too much valuable data. > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > On Tue, Mar 19, 2019 at 4:21 PM > Jevgeni > > > Zolotarjov < > > > > > > > > > > > > >> > j.zolotar...@gmail.com> > > > > > > > > > > > > >> > > wrote: > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > >> Tried that just now and got > error: > > > > > > > > > > > > >> > >> Resource [DataCenter:1] is > unreachable: > > > Can't find > > > > > all > > > > > > > > > > necessary > > > > > > > > > > > > >> running > > > > > > > > > > > > >> > >> routers! > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > >> In the log I see: > > > > > > > > > > > > >> > >> > > ============================================= > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > >> 2019-03-19 14:20:39,644 DEBUG > > > > > > > > [o.a.c.e.o.NetworkOrchestrator] > > > > > > > > > > > > >> > >> (API-Job-Executor-4:ctx-7b6b69eb > > job-5093 > > > > > ctx-9be30648) > > > > > > > > > > > > >> (logid:265a6099) > > > > > > > > > > > > >> > >> Restarting network 204... > > > > > > > > > > > > >> > >> 2019-03-19 14:20:39,645 DEBUG > > > > > > > > [o.a.c.e.o.NetworkOrchestrator] > > > > > > > > > > > > >> > >> (API-Job-Executor-4:ctx-7b6b69eb > > job-5093 > > > > > ctx-9be30648) > > > > > > > > > > > > >> (logid:265a6099) > > > > > > > > > > > > >> > >> Performing rolling restart of > routers > > of > > > network > > > > > > > > > > > Ntwk[204|Guest|6] > > > > > > > > > > > > >> > >> 2019-03-19 14:20:39,649 DEBUG > > > > > > > > [o.a.c.e.o.NetworkOrchestrator] > > > > > > > > > > > > >> > >> (API-Job-Executor-4:ctx-7b6b69eb > > job-5093 > > > > > ctx-9be30648) > > > > > > > > > > > > >> (logid:265a6099) > > > > > > > > > > > > >> > >> Asking VirtualRouter to > implemenet > > > > Ntwk[204|Guest|6] > > > > > > > > > > > > >> > >> 2019-03-19 14:20:39,658 ERROR > > > > > > > [c.c.a.ApiAsyncJobDispatcher] > > > > > > > > > > > > >> > >> (API-Job-Executor-4:ctx-7b6b69eb > > job-5093) > > > > > > > (logid:265a6099) > > > > > > > > > > > > >> Unexpected > > > > > > > > > > > > >> > >> exception while executing > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > org.apache.cloudstack.api.command.user.network.RestartNetworkCmd > > > > > > > > > > > > >> > >> > > > com.cloud.exception.ResourceUnavailableException: > > > > > > > Resource > > > > > > > > > > > > >> > [DataCenter:1] > > > > > > > > > > > > >> > >> is unreachable: Can't find all > > necessary > > > running > > > > > > routers! > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:243) > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElements(NetworkOrchestrator.java:1203) > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.rollingRestartRouters(NetworkOrchestrator.java:2948) > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:2853) > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1883) > > > > > > > > > > > > >> > >> at > > > > > > > > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native > > > > > > > > > > > > Method) > > > > > > > > > > > > >> > >> at > > > > > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(Unknown > > > > > > > > > > > > Source) > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown > > > > > > > > > > > > >> > Source) > > > > > > > > > > > > >> > >> at > > > java.lang.reflect.Method.invoke(Unknown > > > > > > Source) > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338) > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197) > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107) > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51) > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) > > > > > > > > > > > > >> > >> at > > > > > > com.sun.proxy.$Proxy229.restartNetwork(Unknown > > > > > > > > > > Source) > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:99) > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > >> > > > > > > com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150) > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108) > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581) > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529) > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > > > java.util.concurrent.Executors$RunnableAdapter.call(Unknown > > > > > > > > > > > > >> > >> Source) > > > > > > > > > > > > >> > >> at > > > > java.util.concurrent.FutureTask.run(Unknown > > > > > > > > Source) > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown > > > > > > > > > > > > >> > >> Source) > > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown > > > > > > > > > > > > >> > >> Source) > > > > > > > > > > > > >> > >> at > java.lang.Thread.run(Unknown > > > Source) > > > > > > > > > > > > >> > >> 2019-03-19 14:20:39,659 DEBUG > > > > > > > > > [o.a.c.f.j.i.AsyncJobManagerImpl] > > > > > > > > > > > > >> > >> (API-Job-Executor-4:ctx-7b6b69eb > > job-5093) > > > > > > > (logid:265a6099) > > > > > > > > > > > > Complete > > > > > > > > > > > > >> > async > > > > > > > > > > > > >> > >> job-5093, jobStatus: FAILED, > > resultCode: 530, > > > > result: > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Resource > > > > > > > > > > > > >> > >> [DataCenter:1] is un > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > >> On Tue, Mar 19, 2019 at 4:19 PM > Andrija > > > Panic < > > > > > > > > > > > > >> > andrija.pa...@shapeblue.com> > > > > > > > > > > > > >> > >> wrote: > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > >>> > > > > > > > > > > > > >> > >>> Your network can't be deleted > due to > > "Can't > > > delete > > > > > the > > > > > > > > > > network, > > > > > > > > > > > > not > > > > > > > > > > > > >> all > > > > > > > > > > > > >> > >>> user vms are expunged. Vm > > > > > > > > > > > > >> > >>> VM[User|i-2-11-VM] is in Stopped > > state" - > > > which is > > > > > > fine. > > > > > > > > > > > > >> > >>> > > > > > > > > > > > > >> > >>> You should be able to just > start the > > user > > > VM - but > > > > > if > > > > > > > you > > > > > > > > > have > > > > > > > > > > > > >> actually > > > > > > > > > > > > >> > >>> delete the VR itself, then just > do > > Network > > > restart > > > > > > with > > > > > > > > > > > "cleanup" > > > > > > > > > > > > >> and > > > > > > > > > > > > >> > it > > > > > > > > > > > > >> > >>> will recreate a new VR, after > which > > you > > > should be > > > > > able > > > > > > > to > > > > > > > > > > start > > > > > > > > > > > > the > > > > > > > > > > > > >> VM. > > > > > > > > > > > > >> > >>> > > > > > > > > > > > > >> > >>> Andrija > > > > > > > > > > > > >> > >>> > > > > > > > > > > > > >> > >>> andrija.pa...@shapeblue.com > > > > > > > > > > > > >> > >>> www.shapeblue.com > > > > > > > > > > > > >> > >>> Amadeus House, Floral Street, > London > > WC2E > > > 9DPUK > > > > > > > > > > > > >> > >>> @shapeblue > > > > > > > > > > > > >> > >>> > > > > > > > > > > > > >> > >>> > > > > > > > > > > > > >> > >>> > > > > > > > > > > > > >> > >>> > > > > > > > > > > > > >> > >>> -----Original Message----- > > > > > > > > > > > > >> > >>> From: Jevgeni Zolotarjov < > > > j.zolotar...@gmail.com> > > > > > > > > > > > > >> > >>> Sent: 19 March 2019 15:10 > > > > > > > > > > > > >> > >>> To: users@cloudstack.apache.org > > > > > > > > > > > > >> > >>> Subject: Re: Disaster after > > maintenance > > > > > > > > > > > > >> > >>> > > > > > > > > > > > > >> > >>> I mean I cannot delete network: > In the > > > management > > > > > > server > > > > > > > > > log I > > > > > > > > > > > see > > > > > > > > > > > > >> > >>> > > > > > > > > > > > > >> > >>> > > > > > >
