Just to confirm, you are using Basic Zone in CloudStack, right ? Can you confirm that router has been actually started/created on KVM side, again, as requested please post logs (mgmt and agent - and note the time around which you tried to start VR last time it partially succeeded) - we can't guess what went wrong without logs.
I would push more effort solving this one, instead of reinstalling - you might hit the issue again and then it's no good. Can you confirm your zone/pod/cluster/hosts are all in Enabled state, i.e. not disabled and hosts connected AND both SSVM and CPVM are connectedUP/green Is your dashboard in GUI all green - no issues there ? Can you connect your both KVM hosts can access/mount both Primary and Secondary Storage On Wed, 20 Mar 2019 at 13:15, Jevgeni Zolotarjov <j.zolotar...@gmail.com> wrote: > After dozen of attempts, the Virtual Router could finally be recreated. But > its in eternal Starting status, and console prompts it required upgrade and > Version is UNKNOWN > > It does not resolve the problem, I cannot move further form this point. > Any hints? > > Or I am condemned to do reinstall cloudstack from scratch? > > On Wed, Mar 20, 2019 at 11:08 AM Jevgeni Zolotarjov < > j.zolotar...@gmail.com> > wrote: > > > Under this defaultGuestNetwork, I go to Virtual Appliances. There is no > > VMS - "no data to show" > > > > I dont have any network, other than this single default one. > > > > I've tried adding new network - Add guest network. But I am not able to > do > > so, cause in the wizard popup, it offers empty dropdown with Zones > > selection. And this wizard doesnt not allow to go further without > selecting > > Zone > > > > On Wed, Mar 20, 2019 at 10:28 AM Andrija Panic <andrija.pa...@gmail.com> > > wrote: > > > >> you need to delete/remove all VMs inside this network (tick the > "Expunge" > >> button during VM deletion - if you want to really delete the VMs) in > order > >> to be able to delete the network - OR simply attach this VM to another > >> network, make this new network a DEFAULT one (NIC that is...), and then > >> detach from old network - and then effectively your VM was "removed" > from > >> old network - after this you should be able to delete the old network. I > >> assume some DB incosistencies perhaps, being the reason you can not > >> restart > >> the network. > >> > >> Did you try restarting some other Network - or deploying a new network, > >> spin a VM in it, then again try to restart this new network - does it > >> work ? > >> > >> Andrija > >> > >> On Wed, 20 Mar 2019 at 08:58, Jevgeni Zolotarjov < > j.zolotar...@gmail.com> > >> wrote: > >> > >> > >>>Stop mgmt, > >> > >>>Stop all agents > >> > >>>Restart libvirtd (and check libvirt logs afterwards) > >> > >>>Start agents > >> > >>>Start mgmt. > >> > > >> > I did that numerous time. Nothing really suspicious > >> > I can see that systems VMs are running - both in cloudstack console > and > >> > with virsh list -all > >> > > >> > It is apparently problem with network. > >> > Is there a way to force recreation of defaultGuestNetwork? or force > >> > recreation of Virtual Router. > >> > I am unable to delete network, which is supposed to rebuild network > with > >> > its router. Thats the issue > >> > > >> > The issue with libvirtd was, that eventually at some point it was > >> updated > >> > during 4 months of running, and not rebooted. It still worked. We had > to > >> > add listen_tcp = 1 for libvirtd to start working again. > >> > > >> > On Wed, Mar 20, 2019 at 9:49 AM Andrija Panic < > andrija.pa...@gmail.com> > >> > wrote: > >> > > >> > > As Sergey suggested... but i would also verify no libvirt issues or > >> > storage > >> > > pool issues - so perhaps: > >> > > > >> > > Stop mgmt, > >> > > Stop all agents > >> > > Restart libvirtd (and check libvirt logs afterwards) > >> > > Start agents > >> > > Start mgmt. > >> > > > >> > > What was originally issue with libvirtd ? > >> > > That sounds fishy to me... > >> > > > >> > > Andrija > >> > > > >> > > On Wed, Mar 20, 2019, 02:15 Sergey Levitskiy <serg...@hotmail.com> > >> > wrote: > >> > > > >> > > > select * from networks where removed is null; > >> > > > select * from vm_instance where id=87; > >> > > > select id,name from vm_instance where name like 'r%' and removed > is > >> > null; > >> > > > > >> > > > Basically since the network offering is not redundant this error > is > >> > only > >> > > > thrown when there is no router associated with your network. > Usually > >> > > > management server restart tries to implement network again. Please > >> > > restart > >> > > > management server, save and share management server log. > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > On 3/19/19, 3:31 PM, "Jevgeni Zolotarjov" < > j.zolotar...@gmail.com> > >> > > wrote: > >> > > > > >> > > > >>>>Check network_offering table for value in column > >> > > > redundant_router_service for the network offering you use. > >> > > > in table network_offering_table all records have > >> > > > redundant_router_service = > >> > > > 0 > >> > > > > >> > > > Can you also run the following: > >> > > > >>>select name, state, removed from host where name like 'r%' > >> > > > returns zero rows - nothing > >> > > > > >> > > > >>>select * from domain_router; > >> > > > # id, element_id, public_mac_address, public_ip_address, > >> > > > public_netmask, > >> > > > guest_netmask, guest_ip_address, is_redundant_router, > priority, > >> > > > redundant_state, stop_pending, role, template_version, > >> > > scripts_version, > >> > > > vpc_id, update_state > >> > > > '4', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', > >> '0', > >> > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 > 15:37:28 > >> > UTC > >> > > > 2018', > >> > > > '57db7bd8118977a5f2cd3ef1c7503633\n', NULL, NULL > >> > > > '49', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', > >> '0', > >> > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 > 15:37:28 > >> > UTC > >> > > > 2018', > >> > > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL > >> > > > '73', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', > >> '0', > >> > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 > 15:37:28 > >> > UTC > >> > > > 2018', > >> > > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL > >> > > > '74', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', > >> '0', > >> > > > 'VIRTUAL_ROUTER', NULL, NULL, NULL, NULL > >> > > > '75', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', > >> '0', > >> > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 > 15:37:28 > >> > UTC > >> > > > 2018', > >> > > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL > >> > > > '76', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', > >> '0', > >> > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 > 15:37:28 > >> > UTC > >> > > > 2018', > >> > > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL > >> > > > '77', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', > >> '0', > >> > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.1 Fri Jun 22 > 07:52:17 > >> > UTC > >> > > > 2018', > >> > > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, 'UPDATE_FAILED' > >> > > > '80', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', > >> '0', > >> > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.1 Fri Jun 22 > 07:52:17 > >> > UTC > >> > > > 2018', > >> > > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL > >> > > > '85', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', > >> '0', > >> > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.1 Fri Jun 22 > 07:52:17 > >> > UTC > >> > > > 2018', > >> > > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL > >> > > > '86', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', > >> '0', > >> > > > 'VIRTUAL_ROUTER', NULL, NULL, NULL, NULL > >> > > > '87', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', > >> '0', > >> > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.2.0 Mon Nov 12 > >> 15:06:49 > >> > > UTC > >> > > > 2018', '873057731ff2cba4a1f3b2411765407c\n', NULL, NULL > >> > > > > >> > > > > >> > > > >>>select * from router_network_ref; > >> > > > # id, router_id, network_id, guest_type > >> > > > '1', '4', '204', 'Shared' > >> > > > '2', '49', '204', 'Shared' > >> > > > '3', '73', '204', 'Shared' > >> > > > '4', '75', '204', 'Shared' > >> > > > '5', '76', '204', 'Shared' > >> > > > '6', '77', '204', 'Shared' > >> > > > '7', '80', '204', 'Shared' > >> > > > '8', '85', '204', 'Shared' > >> > > > '9', '86', '204', 'Shared' > >> > > > '10', '87', '204', 'Shared' > >> > > > > >> > > > > >> > > > On Wed, Mar 20, 2019 at 12:18 AM Sergey Levitskiy < > >> > > serg...@hotmail.com > >> > > > > > >> > > > wrote: > >> > > > > >> > > > > Check network_offering table for value in column > >> > > > > redundant_router_service for the network offering you use. > >> > > > > Can you also run the following: > >> > > > > select name, state, removed from host where name like 'r%' > >> > > > > select * from domain_router; > >> > > > > select * from router_network_ref; > >> > > > > > >> > > > > Cloudstack is supposed to recreate you VR. If it is not > >> happening > >> > > > there is > >> > > > > something fundamentally wrong. I would advise to destroy > your > >> VR > >> > > > again. > >> > > > > Stop you management server. Rotate management server log and > >> > start > >> > > it > >> > > > > again. If your VR doesn't start in few min, post your > complete > >> > > > management > >> > > > > server log and agent log again. > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > On 3/19/19, 2:56 PM, "Jevgeni Zolotarjov" < > >> > j.zolotar...@gmail.com> > >> > > > wrote: > >> > > > > > >> > > > > >>>Network offering needs to be change to non-redundant > >> > > > > How do I do that? > >> > > > > > >> > > > > On Tue, Mar 19, 2019 at 11:47 PM Sergey Levitskiy < > >> > > > serg...@hotmail.com > >> > > > > > > >> > > > > wrote: > >> > > > > > >> > > > > > Network offering needs to be change to non-redundant. > >> Most > >> > > > likely > >> > > > > old bug > >> > > > > > is resurfaced. > >> > > > > > https://issues.apache.org/jira/browse/CLOUDSTACK-9024 > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > On 3/19/19, 2:19 PM, "Jevgeni Zolotarjov" < > >> > > > j.zolotar...@gmail.com> > >> > > > > wrote: > >> > > > > > > >> > > > > > Hello > >> > > > > > > >> > > > > > I did exactly like you suggested. > >> > > > > > > >> > > > > > After UPDATE on db, I can see the router in > >> cloudstack > >> > > > console. > >> > > > > > But attempt to restart network fails. > >> > > > > > I get error: > >> > > > > > Resource [DataCenter:1] is unreachable: Can't find > >> all > >> > > > necessary > >> > > > > > running > >> > > > > > routers! > >> > > > > > > >> > > > > > I rechecked agents on both servers. They look > >> running > >> > OK. > >> > > > > > > >> > > > > > On Tue, Mar 19, 2019 at 10:00 PM Andrija Panic < > >> > > > > > andrija.pa...@gmail.com> > >> > > > > > wrote: > >> > > > > > > >> > > > > > > Ok, so: > >> > > > > > > > >> > > > > > > 1.BACKUP YOUR DB - in case there is issues, you > >> will > >> > > > want to > >> > > > > restore > >> > > > > > > 2. I tried to reproduce your case on 4.11.2 > >> (though > >> > > > didn't do > >> > > > > any > >> > > > > > > maintenance etc.) - and could not - I have > >> > artificially > >> > > > marked > >> > > > > > existing VR > >> > > > > > > as removed in DB - and tried to restart network > >> and > >> > it > >> > > > worked > >> > > > > just > >> > > > > > fine. > >> > > > > > > > >> > > > > > > Let's try to "undelete" the VR for that network > - > >> > > though > >> > > > I > >> > > > > can't be > >> > > > > > sure if > >> > > > > > > this will work or not. > >> > > > > > > > >> > > > > > > Find VR based on the network UUID: > >> > > > > > > SELECT * FROM vm_instance WHERE id IN > >> (SELECT > >> > > > > instance_id from > >> > > > > > nics > >> > > > > > > where network_id IN (SELECT id FROM networks > WHERE > >> > > > > > > UUID="65ca9a05-ff96-4563-ab9c-ffb610dc8b73")); > >> > > > > > > Obviously, replace this UUID with your network > >> UUID. > >> > > > > > > SQL above will return all VMs (user VMs, VRs, > etc, > >> > > > delete or > >> > > > > not...) > >> > > > > > that > >> > > > > > > have NIC inside the network with that UUID - > look > >> for > >> > > the > >> > > > > r-XX-VM > >> > > > > > being a > >> > > > > > > router. > >> > > > > > > > >> > > > > > > Now that you have the name of the VR (i.e. > >> r-4-VM) - > >> > > you > >> > > > want > >> > > > > to set > >> > > > > > 2 > >> > > > > > > fields to some other value than what it might be > >> > > > already, as > >> > > > > below: > >> > > > > > > > >> > > > > > > Need to set field "state" to Stopped: > >> > > > > > > UPDATE cloud.vm_instance SET > state='Stopped' > >> > > WHERE > >> > > > > > name="r-4-VM"; > >> > > > > > > (what state is set currently ???) > >> > > > > > > > >> > > > > > > Need to set "removed" to NULL value: > >> > > > > > > UPDATE cloud.vm_instance SET removed = > NULL > >> > WHERE > >> > > > > > NAME="r-4-VM"; (is > >> > > > > > > there a removed date already set ???) > >> > > > > > > > >> > > > > > > Obviously make sure that this VR is not running > on > >> > any > >> > > > > hypervisor > >> > > > > > and if > >> > > > > > > running, virsh destroy it... > >> > > > > > > > >> > > > > > > AFTER above has been done (VR is considered to > >> exist > >> > > but > >> > > > > stopped, > >> > > > > > from > >> > > > > > > CloudStack point of view) - try to RESTART the > >> > network > >> > > > (don't > >> > > > > bother > >> > > > > > > deleteing it, since there are VMs in that > network) > >> > > > > > > > >> > > > > > > I'm not 100% positive this will fix your issue, > >> but > >> > > > doesn't > >> > > > > hurt to > >> > > > > > try > >> > > > > > > > >> > > > > > > If above doesn't work - I would still take a > look > >> > into > >> > > > agents > >> > > > > and if > >> > > > > > they > >> > > > > > > are still connected - optionally, restart agents > >> on > >> > > both > >> > > > hosts > >> > > > > once > >> > > > > > more > >> > > > > > > and confirm they are connected and up. > >> > > > > > > > >> > > > > > > Let us know how it goes (and please backup DB > once > >> > more > >> > > > before > >> > > > > any > >> > > > > > actions > >> > > > > > > !) > >> > > > > > > > >> > > > > > > Andrija > >> > > > > > > > >> > > > > > > > >> > > > > > > On Tue, 19 Mar 2019 at 20:41, Jevgeni > Zolotarjov < > >> > > > > > j.zolotar...@gmail.com> > >> > > > > > > wrote: > >> > > > > > > > >> > > > > > > > Yes. Just a single network. > >> > > > > > > > > >> > > > > > > > On Tue, 19 Mar 2019, 21:39 Andrija Panic, < > >> > > > > andrija.pa...@gmail.com > >> > > > > > > > >> > > > > > > wrote: > >> > > > > > > > > >> > > > > > > > > Just one more clarification - this is > Isolate > >> > > single > >> > > > > network (not > >> > > > > > > Shared > >> > > > > > > > > Network, not VPC) ? > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > On Tue, Mar 19, 2019, 19:36 Jevgeni > >> Zolotarjov < > >> > > > > > j.zolotar...@gmail.com > >> > > > > > > > > >> > > > > > > > > wrote: > >> > > > > > > > > > >> > > > > > > > > > Name defaultGuestNetwork > >> > > > > > > > > > ID 4ba834ed-48f3-468f-b667-9bb2d2c258f1 > >> > > > > > > > > > Zone PBT zone 1 > >> > > > > > > > > > Description defaultGuestNetwork > >> > > > > > > > > > Type Shared > >> > > > > > > > > > State Setup > >> > > > > > > > > > VPC ID N/A > >> > > > > > > > > > Persistent No > >> > > > > > > > > > > >> > > > > > > > > > On Tue, Mar 19, 2019 at 8:29 PM Andrija > >> Panic < > >> > > > > > > andrija.pa...@gmail.com > >> > > > > > > > > > >> > > > > > > > > > wrote: > >> > > > > > > > > > > >> > > > > > > > > > > Share the network id and name as seen > from > >> > > GUI... > >> > > > > > > > > > > > >> > > > > > > > > > > On Tue, Mar 19, 2019, 19:27 Jevgeni > >> > Zolotarjov > >> > > < > >> > > > > > > > j.zolotar...@gmail.com > >> > > > > > > > > > > >> > > > > > > > > > > wrote: > >> > > > > > > > > > > > >> > > > > > > > > > > > >>> 1. Confirm please rolling restart > is > >> > set > >> > > > to false > >> > > > > > please - > >> > > > > > > > double > >> > > > > > > > > > > check > >> > > > > > > > > > > > Double checked - It is set to false > >> > > > > > > > > > > > > >> > > > > > > > > > > > >>>>2. If so - do you know the name of > >> VR > >> > > > which you > >> > > > > > deleted ? Is > >> > > > > > > it > >> > > > > > > > > > last > >> > > > > > > > > > > > one > >> > > > > > > > > > > > >>> ever created - if so we can find > it > >> > > > easily... > >> > > > > > > > > > > > I dont know the name > >> > > > > > > > > > > > Is there a way to fetch it from DB? > >> > > > > > > > > > > > > >> > > > > > > > > > > > On Tue, Mar 19, 2019 at 7:58 PM > Andrija > >> > > Panic < > >> > > > > > > > > andrija.pa...@gmail.com > >> > > > > > > > > > > > >> > > > > > > > > > > > wrote: > >> > > > > > > > > > > > > >> > > > > > > > > > > > > Right...still complaining on missing > >> > > running > >> > > > > routers. > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > 1. Confirm please rolling restart is > >> set > >> > to > >> > > > false > >> > > > > please > >> > > > > > - > >> > > > > > > double > >> > > > > > > > > > check > >> > > > > > > > > > > > > 2. If so - do you know the name of > VR > >> > which > >> > > > you > >> > > > > deleted > >> > > > > > ? Is it > >> > > > > > > > > last > >> > > > > > > > > > > one > >> > > > > > > > > > > > > ever created - if so we can find it > >> > > easily... > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > On Tue, Mar 19, 2019, 18:40 Jevgeni > >> > > > Zolotarjov < > >> > > > > > > > > > j.zolotar...@gmail.com > >> > > > > > > > > > > > > >> > > > > > > > > > > > > wrote: > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > Here is management server log > >> > > > > > > > > > > > > > > >> > > > > > > > > > >> > > > > > > >> > > > > https://drive.google.com/open?id=1H2jI0roeiWxtzReB8qV6QxDkNpaki99A > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > On Tue, Mar 19, 2019 at 7:29 PM > >> Andrija > >> > > > Panic < > >> > > > > > > > > > > andrija.pa...@gmail.com > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > wrote: > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > I would also try to "undelete" > VR > >> in > >> > > DB, > >> > > > but > >> > > > > let's > >> > > > > > keep > >> > > > > > > this > >> > > > > > > > as > >> > > > > > > > > > > last > >> > > > > > > > > > > > > > step. > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > On Tue, Mar 19, 2019, 18:24 > >> Andrija > >> > > > Panic < > >> > > > > > > > > > andrija.pa...@gmail.com > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > wrote: > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > Disclaimer: from mobile > device. > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > I don't see any special > failure > >> in > >> > > > agent log, > >> > > > > > except some > >> > > > > > > > > long > >> > > > > > > > > > > > > running > >> > > > > > > > > > > > > > > > migration, timeout for > graceful > >> VM > >> > > > shutdown > >> > > > > etc > >> > > > > > (and > >> > > > > > > agent > >> > > > > > > > > > > restart) > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > You did not send mgmt log > after > >> > last > >> > > > network > >> > > > > > restart > >> > > > > > > > failure > >> > > > > > > > > ? > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > On Tue, Mar 19, 2019, 17:29 > >> Jevgeni > >> > > > > Zolotarjov < > >> > > > > > > > > > > > > j.zolotar...@gmail.com > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > wrote: > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> Under > >> > > > > > > > > > > > > > > >> Infrastructure / Hosts: > >> > > > > > > > > > > > > > > >> both hosts are enabled and > >> green > >> > > > (Unsecure > >> > > > > though) > >> > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > >> agent.log - > >> > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > > > > > > > >> > > > > > >> > https://drive.google.com/open?id=1rATxHKqgNKo2kD23BtlrZy_9gFXC-Bq- > >> > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > >> I set network.rolling.restart > >> to > >> > > > false now. > >> > > > > > Restarted > >> > > > > > > > > > management > >> > > > > > > > > > > > > > server > >> > > > > > > > > > > > > > > - > >> > > > > > > > > > > > > > > >> same problem - cannot restart > >> not > >> > > > delete > >> > > > > network > >> > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > >> On Tue, Mar 19, 2019 at 5:48 > PM > >> > > Boris > >> > > > > Stoyanov < > >> > > > > > > > > > > > > > > >> boris.stoya...@shapeblue.com > > > >> > > > > > > > > > > > > > > >> wrote: > >> > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > >> > Hi, you shouldn’t be > worried > >> > about > >> > > > data > >> > > > > as long > >> > > > > > as you > >> > > > > > > > > use a > >> > > > > > > > > > > > > > separate > >> > > > > > > > > > > > > > > >> > shared storage. > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > >> > What does the cloudstack > >> console > >> > > say > >> > > > > about your > >> > > > > > host? > >> > > > > > > Is > >> > > > > > > > > it > >> > > > > > > > > > > up? > >> > > > > > > > > > > > If > >> > > > > > > > > > > > > > > it’s > >> > > > > > > > > > > > > > > >> up > >> > > > > > > > > > > > > > > >> > then you should be able to > >> > deploy > >> > > a > >> > > > VM on > >> > > > > it. > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > >> > If not, you’ll need to > >> > investigate > >> > > > the > >> > > > > reason > >> > > > > > > > > > cloudstack-agent > >> > > > > > > > > > > > is > >> > > > > > > > > > > > > > not > >> > > > > > > > > > > > > > > >> > connected, is the agent > >> running? > >> > > If > >> > > > not > >> > > > > just > >> > > > > > start it > >> > > > > > > > and > >> > > > > > > > > > see > >> > > > > > > > > > > if > >> > > > > > > > > > > > > it > >> > > > > > > > > > > > > > > >> > connects. > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > >> > If it’s running but still > not > >> > > > connected > >> > > > > please > >> > > > > > share > >> > > > > > > the > >> > > > > > > > > > > > > agent.log. > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > >> > If nothing helps at all you > >> can > >> > > > remove > >> > > > > the host > >> > > > > > and > >> > > > > > > add > >> > > > > > > > it > >> > > > > > > > > > > > again, > >> > > > > > > > > > > > > > but > >> > > > > > > > > > > > > > > >> > let’s try to troubleshoot > >> first. > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > >> > Bobby. > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > >> > > boris.stoya...@shapeblue.com > >> > > > > > > > > > > > > > > >> > www.shapeblue.com > >> > > > > > > > > > > > > > > >> > Amadeus House, Floral > Street, > >> > > London > >> > > > > WC2E 9DPUK > >> > > > > > > > > > > > > > > >> > @shapeblue > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > >> > > On 19 Mar 2019, at 17:29, > >> > > Jevgeni > >> > > > > Zolotarjov < > >> > > > > > > > > > > > > > > j.zolotar...@gmail.com> > >> > > > > > > > > > > > > > > >> > wrote: > >> > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > >> > > Guys, please help with > it. > >> > > > > > > > > > > > > > > >> > > What can be done here? > >> > > > > > > > > > > > > > > >> > > There is too much > valuable > >> > data. > >> > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > >> > > On Tue, Mar 19, 2019 at > >> 4:21 > >> > PM > >> > > > Jevgeni > >> > > > > > Zolotarjov < > >> > > > > > > > > > > > > > > >> > j.zolotar...@gmail.com> > >> > > > > > > > > > > > > > > >> > > wrote: > >> > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > >> > >> Tried that just now and > >> got > >> > > > error: > >> > > > > > > > > > > > > > > >> > >> Resource [DataCenter:1] > is > >> > > > unreachable: > >> > > > > > Can't find > >> > > > > > > > all > >> > > > > > > > > > > > > necessary > >> > > > > > > > > > > > > > > >> running > >> > > > > > > > > > > > > > > >> > >> routers! > >> > > > > > > > > > > > > > > >> > >> > >> > > > > > > > > > > > > > > >> > >> In the log I see: > >> > > > > > > > > > > > > > > >> > >> > >> > > > > ============================================= > >> > > > > > > > > > > > > > > >> > >> > >> > > > > > > > > > > > > > > >> > >> 2019-03-19 14:20:39,644 > >> DEBUG > >> > > > > > > > > > > [o.a.c.e.o.NetworkOrchestrator] > >> > > > > > > > > > > > > > > >> > >> > >> > > (API-Job-Executor-4:ctx-7b6b69eb > >> > > > > job-5093 > >> > > > > > > > ctx-9be30648) > >> > > > > > > > > > > > > > > >> (logid:265a6099) > >> > > > > > > > > > > > > > > >> > >> Restarting network > 204... > >> > > > > > > > > > > > > > > >> > >> 2019-03-19 14:20:39,645 > >> DEBUG > >> > > > > > > > > > > [o.a.c.e.o.NetworkOrchestrator] > >> > > > > > > > > > > > > > > >> > >> > >> > > (API-Job-Executor-4:ctx-7b6b69eb > >> > > > > job-5093 > >> > > > > > > > ctx-9be30648) > >> > > > > > > > > > > > > > > >> (logid:265a6099) > >> > > > > > > > > > > > > > > >> > >> Performing rolling > >> restart of > >> > > > routers > >> > > > > of > >> > > > > > network > >> > > > > > > > > > > > > > Ntwk[204|Guest|6] > >> > > > > > > > > > > > > > > >> > >> 2019-03-19 14:20:39,649 > >> DEBUG > >> > > > > > > > > > > [o.a.c.e.o.NetworkOrchestrator] > >> > > > > > > > > > > > > > > >> > >> > >> > > (API-Job-Executor-4:ctx-7b6b69eb > >> > > > > job-5093 > >> > > > > > > > ctx-9be30648) > >> > > > > > > > > > > > > > > >> (logid:265a6099) > >> > > > > > > > > > > > > > > >> > >> Asking VirtualRouter to > >> > > > implemenet > >> > > > > > > Ntwk[204|Guest|6] > >> > > > > > > > > > > > > > > >> > >> 2019-03-19 14:20:39,658 > >> ERROR > >> > > > > > > > > > [c.c.a.ApiAsyncJobDispatcher] > >> > > > > > > > > > > > > > > >> > >> > >> > > (API-Job-Executor-4:ctx-7b6b69eb > >> > > > > job-5093) > >> > > > > > > > > > (logid:265a6099) > >> > > > > > > > > > > > > > > >> Unexpected > >> > > > > > > > > > > > > > > >> > >> exception while > executing > >> > > > > > > > > > > > > > > >> > >> > >> > > > > > > > > > > > > > >> > > > > > > > >> > > > > > >> org.apache.cloudstack.api.command.user.network.RestartNetworkCmd > >> > > > > > > > > > > > > > > >> > >> > >> > > > > > com.cloud.exception.ResourceUnavailableException: > >> > > > > > > > > > Resource > >> > > > > > > > > > > > > > > >> > [DataCenter:1] > >> > > > > > > > > > > > > > > >> > >> is unreachable: Can't > find > >> > all > >> > > > > necessary > >> > > > > > running > >> > > > > > > > > routers! > >> > > > > > > > > > > > > > > >> > >> at > >> > > > > > > > > > > > > > > >> > >> > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:243) > >> > > > > > > > > > > > > > > >> > >> at > >> > > > > > > > > > > > > > > >> > >> > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElements(NetworkOrchestrator.java:1203) > >> > > > > > > > > > > > > > > >> > >> at > >> > > > > > > > > > > > > > > >> > >> > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.rollingRestartRouters(NetworkOrchestrator.java:2948) > >> > > > > > > > > > > > > > > >> > >> at > >> > > > > > > > > > > > > > > >> > >> > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:2853) > >> > > > > > > > > > > > > > > >> > >> at > >> > > > > > > > > > > > > > > >> > >> > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1883) > >> > > > > > > > > > > > > > > >> > >> at > >> > > > > > > > > > > > >> > > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native > >> > > > > > > > > > > > > > > Method) > >> > > > > > > > > > > > > > > >> > >> at > >> > > > > > > > > > > > >> > > > sun.reflect.NativeMethodAccessorImpl.invoke(Unknown > >> > > > > > > > > > > > > > > Source) > >> > > > > > > > > > > > > > > >> > >> at > >> > > > > > > > > > > > > > >> > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown > >> > > > > > > > > > > > > > > >> > Source) > >> > > > > > > > > > > > > > > >> > >> at > >> > > > > > java.lang.reflect.Method.invoke(Unknown > >> > > > > > > > > Source) > >> > > > > > > > > > > > > > > >> > >> at > >> > > > > > > > > > > > > > > >> > >> > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338) > >> > > > > > > > > > > > > > > >> > >> at > >> > > > > > > > > > > > > > > >> > >> > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197) > >> > > > > > > > > > > > > > > >> > >> at > >> > > > > > > > > > > > > > > >> > >> > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) > >> > > > > > > > > > > > > > > >> > >> at > >> > > > > > > > > > > > > > > >> > >> > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107) > >> > > > > > > > > > > > > > > >> > >> at > >> > > > > > > > > > > > > > > >> > >> > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) > >> > > > > > > > > > > > > > > >> > >> at > >> > > > > > > > > > > > > > > >> > >> > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51) > >> > > > > > > > > > > > > > > >> > >> at > >> > > > > > > > > > > > > > > >> > >> > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) > >> > > > > > > > > > > > > > > >> > >> at > >> > > > > > > > > > > > > > > >> > >> > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) > >> > > > > > > > > > > > > > > >> > >> at > >> > > > > > > > > > > > > > > >> > >> > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) > >> > > > > > > > > > > > > > > >> > >> at > >> > > > > > > > > > > > > > > >> > >> > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) > >> > > > > > > > > > > > > > > >> > >> at > >> > > > > > > > > > com.sun.proxy.$Proxy229.restartNetwork(Unknown > >> > > > > > > > > > > > > Source) > >> > > > > > > > > > > > > > > >> > >> at > >> > > > > > > > > > > > > > > >> > >> > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:99) > >> > > > > > > > > > > > > > > >> > >> at > >> > > > > > > > > > > > > > > >> > >> > > > > > > > > >> > > > com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150) > >> > > > > > > > > > > > > > > >> > >> at > >> > > > > > > > > > > > > > > >> > >> > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108) > >> > > > > > > > > > > > > > > >> > >> at > >> > > > > > > > > > > > > > > >> > >> > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581) > >> > > > > > > > > > > > > > > >> > >> at > >> > > > > > > > > > > > > > > >> > >> > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > >> > > > > > > > > > > > > > > >> > >> at > >> > > > > > > > > > > > > > > >> > >> > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > >> > > > > > > > > > > > > > > >> > >> at > >> > > > > > > > > > > > > > > >> > >> > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > >> > > > > > > > > > > > > > > >> > >> at > >> > > > > > > > > > > > > > > >> > >> > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > >> > > > > > > > > > > > > > > >> > >> at > >> > > > > > > > > > > > > > > >> > >> > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > >> > > > > > > > > > > > > > > >> > >> at > >> > > > > > > > > > > > > > > >> > >> > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529) > >> > > > > > > > > > > > > > > >> > >> at > >> > > > > > > > > > > > > > > > >> > > > > > > >> java.util.concurrent.Executors$RunnableAdapter.call(Unknown > >> > > > > > > > > > > > > > > >> > >> Source) > >> > > > > > > > > > > > > > > >> > >> at > >> > > > > > > java.util.concurrent.FutureTask.run(Unknown > >> > > > > > > > > > > Source) > >> > > > > > > > > > > > > > > >> > >> at > >> > > > > > > > > > > > > > > >> > > > > > > >> java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown > >> > > > > > > > > > > > > > > >> > >> Source) > >> > > > > > > > > > > > > > > >> > >> at > >> > > > > > > > > > > > > > > > >> > > > > > > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown > >> > > > > > > > > > > > > > > >> > >> Source) > >> > > > > > > > > > > > > > > >> > >> at > >> > > > java.lang.Thread.run(Unknown > >> > > > > > Source) > >> > > > > > > > > > > > > > > >> > >> 2019-03-19 14:20:39,659 > >> DEBUG > >> > > > > > > > > > > > [o.a.c.f.j.i.AsyncJobManagerImpl] > >> > > > > > > > > > > > > > > >> > >> > >> > > (API-Job-Executor-4:ctx-7b6b69eb > >> > > > > job-5093) > >> > > > > > > > > > (logid:265a6099) > >> > > > > > > > > > > > > > > Complete > >> > > > > > > > > > > > > > > >> > async > >> > > > > > > > > > > > > > > >> > >> job-5093, jobStatus: > >> FAILED, > >> > > > > resultCode: 530, > >> > > > > > > result: > >> > > > > > > > > > > > > > > >> > >> > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Resource > >> > > > > > > > > > > > > > > >> > >> [DataCenter:1] is un > >> > > > > > > > > > > > > > > >> > >> > >> > > > > > > > > > > > > > > >> > >> On Tue, Mar 19, 2019 at > >> 4:19 > >> > PM > >> > > > Andrija > >> > > > > > Panic < > >> > > > > > > > > > > > > > > >> > > andrija.pa...@shapeblue.com> > >> > > > > > > > > > > > > > > >> > >> wrote: > >> > > > > > > > > > > > > > > >> > >> > >> > > > > > > > > > > > > > > >> > >>> > >> > > > > > > > > > > > > > > >> > >>> Your network can't be > >> > deleted > >> > > > due to > >> > > > > "Can't > >> > > > > > delete > >> > > > > > > > the > >> > > > > > > > > > > > > network, > >> > > > > > > > > > > > > > > not > >> > > > > > > > > > > > > > > >> all > >> > > > > > > > > > > > > > > >> > >>> user vms are expunged. > Vm > >> > > > > > > > > > > > > > > >> > >>> VM[User|i-2-11-VM] is > in > >> > > Stopped > >> > > > > state" - > >> > > > > > which is > >> > > > > > > > > fine. > >> > > > > > > > > > > > > > > >> > >>> > >> > > > > > > > > > > > > > > >> > >>> You should be able to > >> just > >> > > > start the > >> > > > > user > >> > > > > > VM - but > >> > > > > > > > if > >> > > > > > > > > > you > >> > > > > > > > > > > > have > >> > > > > > > > > > > > > > > >> actually > >> > > > > > > > > > > > > > > >> > >>> delete the VR itself, > >> then > >> > > just > >> > > > do > >> > > > > Network > >> > > > > > restart > >> > > > > > > > > with > >> > > > > > > > > > > > > > "cleanup" > >> > > > > > > > > > > > > > > >> and > >> > > > > > > > > > > > > > > >> > it > >> > > > > > > > > > > > > > > >> > >>> will recreate a new VR, > >> > after > >> > > > which > >> > > > > you > >> > > > > > should be > >> > > > > > > > able > >> > > > > > > > > > to > >> > > > > > > > > > > > > start > >> > > > > > > > > > > > > > > the > >> > > > > > > > > > > > > > > >> VM. > >> > > > > > > > > > > > > > > >> > >>> > >> > > > > > > > > > > > > > > >> > >>> Andrija > >> > > > > > > > > > > > > > > >> > >>> > >> > > > > > > > > > > > > > > >> > >>> > >> andrija.pa...@shapeblue.com > >> > > > > > > > > > > > > > > >> > >>> www.shapeblue.com > >> > > > > > > > > > > > > > > >> > >>> Amadeus House, Floral > >> > Street, > >> > > > London > >> > > > > WC2E > >> > > > > > 9DPUK > >> > > > > > > > > > > > > > > >> > >>> @shapeblue > >> > > > > > > > > > > > > > > >> > >>> > >> > > > > > > > > > > > > > > >> > >>> > >> > > > > > > > > > > > > > > >> > >>> > >> > > > > > > > > > > > > > > >> > >>> > >> > > > > > > > > > > > > > > >> > >>> -----Original > >> Message----- > >> > > > > > > > > > > > > > > >> > >>> From: Jevgeni > Zolotarjov > >> < > >> > > > > > j.zolotar...@gmail.com> > >> > > > > > > > > > > > > > > >> > >>> Sent: 19 March 2019 > 15:10 > >> > > > > > > > > > > > > > > >> > >>> To: > >> > > users@cloudstack.apache.org > >> > > > > > > > > > > > > > > >> > >>> Subject: Re: Disaster > >> after > >> > > > > maintenance > >> > > > > > > > > > > > > > > >> > >>> > >> > > > > > > > > > > > > > > >> > >>> I mean I cannot delete > >> > > network: > >> > > > In the > >> > > > > > management > >> > > > > > > > > server > >> > > > > > > > > > > > log I > >> > > > > > > > > > > > > > see > >> > > > > > > > > > > > > > > >> > >>> > >> > > > > > > > > > > > > > > >> > >>> > >> > > > > > > > > > >> > > > >> > > >> > >> > >> -- > >> > >> Andrija Panić > >> > > > -- Andrija Panić
