It started with 4.10 and then gradually upgraded with all stops, when new releases were available.
>>> Why do you have 3 zones in this installation - what is the setup ? >>> SSVM and CPVM (for whatever zone) are failing to be created... Its a result of attempts to create new zone and somehow move VMs to this new zone. These all are unsuccessful attempts. Before problem started there was 1 zone and There should be just 1 zone in reality. >>> yes, the VR can't be started, it get's timeout - in AGENT logs, I see that >>> it attemps to create a volume on primary storage... I guess this is the root cause. I checked, and primary storage is accessible via NFS share on both hosts. How to troubleshoot it? On Wed, Mar 20, 2019 at 6:29 PM Andrija Panic <andrija.pa...@gmail.com> wrote: > Hi, > > 2019-03-20 06:41:50,446 INFO [c.c.u.DatabaseUpgradeChecker] (main:null) > (logid:) DB version = 4.10.0.0 Code Version = 4.11.2.0 > 2019-03-20 06:41:50,447 DEBUG [c.c.u.DatabaseUpgradeChecker] (main:null) > (logid:) Running upgrade Upgrade41000to41100 to upgrade from > 4.10.0.0-4.11.0.0 to 4.11.0.0 > fails due to > java.sql.SQLException: Error on rename of './cloud/ldap_trust_map' to > './cloud/#sql2-2f01-13d' (errno: 152) > > Then later... > > com.cloud.exception.InsufficientServerCapacityException: Unable to create a > deployment for VM[SecondaryStorageVm|s-734-VM]Scope=interface > com.cloud.dc.DataCenter; id=3 > com.cloud.exception.InsufficientServerCapacityException: Unable to create a > deployment for VM[ConsoleProxy|v-733-VM]Scope=interface > com.cloud.dc.DataCenter; id=3 > > 2019-03-20 15:02:39,113 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] > (secstorage-1:ctx-059f87f3) (logid:cf6cf89a) Zone 1 is ready to launch > secondary storage VM > 2019-03-20 15:02:39,117 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] > (secstorage-1:ctx-059f87f3) (logid:cf6cf89a) Zone 2 is not ready to launch > secondary storage VM yet > 2019-03-20 15:02:39,122 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] > (secstorage-1:ctx-059f87f3) (logid:cf6cf89a) Zone 3 is ready to launch > secondary storage VM > > so did you start with clean 4.11.2 install, or was it upgraded one - I see > in logs an upgrade from DB version 4.10 to 4.11 was tried and failed... > Why do you have 3 zones in this installation - what is the setup ? > SSVM and CPVM (for whatever zone) are failing to be created... > > yes, the VR can't be started, it get's timeout - in AGENT logs, I see that > it attemps to create a volume on primary storage... > > > Also, for SSVM I got this one... > 2019-03-20 14:38:09,227 DEBUG [c.c.d.FirstFitPlanner] > (Work-Job-Executor-96:ctx-04c5c9f2 job-5120/job-6960 ctx-fde3d4d7) > (logid:49483c7a) No clusters found having a host with enough capacity, > returning. > > Andrija > > On Wed, 20 Mar 2019 at 16:39, Jevgeni Zolotarjov <j.zolotar...@gmail.com> > wrote: > > > Basic Zone - Yes > > > > router has been actually started/created on KVM side - not created, not > > started. Thats the main problem, I guess > > > > agent.log > > https://drive.google.com/open?id=1rATxHKqgNKo2kD23BtlrZy_9gFXC-Bq- > > > > management log > > https://drive.google.com/open?id=1H2jI0roeiWxtzReB8qV6QxDkNpaki99A > > > > >> Can you confirm your zone/pod/cluster/hosts are all in Enabled state, > > i.e. > > YES, all green > > > > >> Can you connect your both KVM hosts can access/mount both Primary and > > Secondary Storage > > YES. Double checked > > > > >>>Can you also explain your infrastructure - you said you have two hosts > > only, where does CloudStack management run? > > 2 hosts: > > host1: 192.168.1.14 > > host2: 192.168.1.5 > > > > Servers are standing next to each other - connected to the same switch > > Management server runs on the same physical server with host1 > > > > I noticed, that Virtual router gets created after I try to start any of > the > > existing guest VM > > Here are logs > > management: > > https://drive.google.com/open?id=1H2jI0roeiWxtzReB8qV6QxDkNpaki99A > > > > agent on host1: > > https://drive.google.com/open?id=1u8YHYIuyU2MA2UKY7G5z7q8p5XxU1zsy > > > > agent on host2: > > https://drive.google.com/open?id=1YzkCL-FmTgPva-QHHp5vTM5Nb3qAXxz4 > > > > But this virtual router stays in Starting state forever and hence VMs do > > not start either. > > > > On Wed, Mar 20, 2019 at 2:49 PM Andrija Panic <andrija.pa...@gmail.com> > > wrote: > > > > > Just to confirm, you are using Basic Zone in CloudStack, right ? > > > > > > Can you confirm that router has been actually started/created on KVM > > side, > > > again, as requested please post logs (mgmt and agent - and note the > time > > > around which you tried to start VR last time it partially succeeded) - > we > > > can't guess what went wrong without logs. > > > > > > I would push more effort solving this one, instead of reinstalling - > you > > > might hit the issue again and then it's no good. > > > > > > Can you confirm your zone/pod/cluster/hosts are all in Enabled state, > > i.e. > > > not disabled and hosts connected AND both SSVM and CPVM are > > > connectedUP/green > > > Is your dashboard in GUI all green - no issues there ? > > > Can you connect your both KVM hosts can access/mount both Primary and > > > Secondary Storage > > > > > > On Wed, 20 Mar 2019 at 13:15, Jevgeni Zolotarjov < > j.zolotar...@gmail.com > > > > > > wrote: > > > > > > > After dozen of attempts, the Virtual Router could finally be > recreated. > > > But > > > > its in eternal Starting status, and console prompts it required > upgrade > > > and > > > > Version is UNKNOWN > > > > > > > > It does not resolve the problem, I cannot move further form this > point. > > > > Any hints? > > > > > > > > Or I am condemned to do reinstall cloudstack from scratch? > > > > > > > > On Wed, Mar 20, 2019 at 11:08 AM Jevgeni Zolotarjov < > > > > j.zolotar...@gmail.com> > > > > wrote: > > > > > > > > > Under this defaultGuestNetwork, I go to Virtual Appliances. There > is > > no > > > > > VMS - "no data to show" > > > > > > > > > > I dont have any network, other than this single default one. > > > > > > > > > > I've tried adding new network - Add guest network. But I am not > able > > to > > > > do > > > > > so, cause in the wizard popup, it offers empty dropdown with Zones > > > > > selection. And this wizard doesnt not allow to go further without > > > > selecting > > > > > Zone > > > > > > > > > > On Wed, Mar 20, 2019 at 10:28 AM Andrija Panic < > > > andrija.pa...@gmail.com> > > > > > wrote: > > > > > > > > > >> you need to delete/remove all VMs inside this network (tick the > > > > "Expunge" > > > > >> button during VM deletion - if you want to really delete the VMs) > in > > > > order > > > > >> to be able to delete the network - OR simply attach this VM to > > another > > > > >> network, make this new network a DEFAULT one (NIC that is...), and > > > then > > > > >> detach from old network - and then effectively your VM was > "removed" > > > > from > > > > >> old network - after this you should be able to delete the old > > > network. I > > > > >> assume some DB incosistencies perhaps, being the reason you can > not > > > > >> restart > > > > >> the network. > > > > >> > > > > >> Did you try restarting some other Network - or deploying a new > > > network, > > > > >> spin a VM in it, then again try to restart this new network - does > > it > > > > >> work ? > > > > >> > > > > >> Andrija > > > > >> > > > > >> On Wed, 20 Mar 2019 at 08:58, Jevgeni Zolotarjov < > > > > j.zolotar...@gmail.com> > > > > >> wrote: > > > > >> > > > > >> > >>>Stop mgmt, > > > > >> > >>>Stop all agents > > > > >> > >>>Restart libvirtd (and check libvirt logs afterwards) > > > > >> > >>>Start agents > > > > >> > >>>Start mgmt. > > > > >> > > > > > >> > I did that numerous time. Nothing really suspicious > > > > >> > I can see that systems VMs are running - both in cloudstack > > console > > > > and > > > > >> > with virsh list -all > > > > >> > > > > > >> > It is apparently problem with network. > > > > >> > Is there a way to force recreation of defaultGuestNetwork? or > > force > > > > >> > recreation of Virtual Router. > > > > >> > I am unable to delete network, which is supposed to rebuild > > network > > > > with > > > > >> > its router. Thats the issue > > > > >> > > > > > >> > The issue with libvirtd was, that eventually at some point it > was > > > > >> updated > > > > >> > during 4 months of running, and not rebooted. It still worked. > We > > > had > > > > to > > > > >> > add listen_tcp = 1 for libvirtd to start working again. > > > > >> > > > > > >> > On Wed, Mar 20, 2019 at 9:49 AM Andrija Panic < > > > > andrija.pa...@gmail.com> > > > > >> > wrote: > > > > >> > > > > > >> > > As Sergey suggested... but i would also verify no libvirt > issues > > > or > > > > >> > storage > > > > >> > > pool issues - so perhaps: > > > > >> > > > > > > >> > > Stop mgmt, > > > > >> > > Stop all agents > > > > >> > > Restart libvirtd (and check libvirt logs afterwards) > > > > >> > > Start agents > > > > >> > > Start mgmt. > > > > >> > > > > > > >> > > What was originally issue with libvirtd ? > > > > >> > > That sounds fishy to me... > > > > >> > > > > > > >> > > Andrija > > > > >> > > > > > > >> > > On Wed, Mar 20, 2019, 02:15 Sergey Levitskiy < > > serg...@hotmail.com > > > > > > > > >> > wrote: > > > > >> > > > > > > >> > > > select * from networks where removed is null; > > > > >> > > > select * from vm_instance where id=87; > > > > >> > > > select id,name from vm_instance where name like 'r%' and > > removed > > > > is > > > > >> > null; > > > > >> > > > > > > > >> > > > Basically since the network offering is not redundant this > > error > > > > is > > > > >> > only > > > > >> > > > thrown when there is no router associated with your network. > > > > Usually > > > > >> > > > management server restart tries to implement network again. > > > Please > > > > >> > > restart > > > > >> > > > management server, save and share management server log. > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > On 3/19/19, 3:31 PM, "Jevgeni Zolotarjov" < > > > > j.zolotar...@gmail.com> > > > > >> > > wrote: > > > > >> > > > > > > > >> > > > >>>>Check network_offering table for value in column > > > > >> > > > redundant_router_service for the network offering you > > use. > > > > >> > > > in table network_offering_table all records have > > > > >> > > > redundant_router_service = > > > > >> > > > 0 > > > > >> > > > > > > > >> > > > Can you also run the following: > > > > >> > > > >>>select name, state, removed from host where name > like > > > 'r%' > > > > >> > > > returns zero rows - nothing > > > > >> > > > > > > > >> > > > >>>select * from domain_router; > > > > >> > > > # id, element_id, public_mac_address, public_ip_address, > > > > >> > > > public_netmask, > > > > >> > > > guest_netmask, guest_ip_address, is_redundant_router, > > > > priority, > > > > >> > > > redundant_state, stop_pending, role, template_version, > > > > >> > > scripts_version, > > > > >> > > > vpc_id, update_state > > > > >> > > > '4', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, > > > 'UNKNOWN', > > > > >> '0', > > > > >> > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 > > > > 15:37:28 > > > > >> > UTC > > > > >> > > > 2018', > > > > >> > > > '57db7bd8118977a5f2cd3ef1c7503633\n', NULL, NULL > > > > >> > > > '49', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, > > > 'UNKNOWN', > > > > >> '0', > > > > >> > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 > > > > 15:37:28 > > > > >> > UTC > > > > >> > > > 2018', > > > > >> > > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL > > > > >> > > > '73', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, > > > 'UNKNOWN', > > > > >> '0', > > > > >> > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 > > > > 15:37:28 > > > > >> > UTC > > > > >> > > > 2018', > > > > >> > > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL > > > > >> > > > '74', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, > > > 'UNKNOWN', > > > > >> '0', > > > > >> > > > 'VIRTUAL_ROUTER', NULL, NULL, NULL, NULL > > > > >> > > > '75', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, > > > 'UNKNOWN', > > > > >> '0', > > > > >> > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 > > > > 15:37:28 > > > > >> > UTC > > > > >> > > > 2018', > > > > >> > > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL > > > > >> > > > '76', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, > > > 'UNKNOWN', > > > > >> '0', > > > > >> > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 > > > > 15:37:28 > > > > >> > UTC > > > > >> > > > 2018', > > > > >> > > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL > > > > >> > > > '77', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, > > > 'UNKNOWN', > > > > >> '0', > > > > >> > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.1 Fri Jun 22 > > > > 07:52:17 > > > > >> > UTC > > > > >> > > > 2018', > > > > >> > > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, > > 'UPDATE_FAILED' > > > > >> > > > '80', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, > > > 'UNKNOWN', > > > > >> '0', > > > > >> > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.1 Fri Jun 22 > > > > 07:52:17 > > > > >> > UTC > > > > >> > > > 2018', > > > > >> > > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL > > > > >> > > > '85', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, > > > 'UNKNOWN', > > > > >> '0', > > > > >> > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.1 Fri Jun 22 > > > > 07:52:17 > > > > >> > UTC > > > > >> > > > 2018', > > > > >> > > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL > > > > >> > > > '86', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, > > > 'UNKNOWN', > > > > >> '0', > > > > >> > > > 'VIRTUAL_ROUTER', NULL, NULL, NULL, NULL > > > > >> > > > '87', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, > > > 'UNKNOWN', > > > > >> '0', > > > > >> > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.2.0 Mon Nov > 12 > > > > >> 15:06:49 > > > > >> > > UTC > > > > >> > > > 2018', '873057731ff2cba4a1f3b2411765407c\n', NULL, NULL > > > > >> > > > > > > > >> > > > > > > > >> > > > >>>select * from router_network_ref; > > > > >> > > > # id, router_id, network_id, guest_type > > > > >> > > > '1', '4', '204', 'Shared' > > > > >> > > > '2', '49', '204', 'Shared' > > > > >> > > > '3', '73', '204', 'Shared' > > > > >> > > > '4', '75', '204', 'Shared' > > > > >> > > > '5', '76', '204', 'Shared' > > > > >> > > > '6', '77', '204', 'Shared' > > > > >> > > > '7', '80', '204', 'Shared' > > > > >> > > > '8', '85', '204', 'Shared' > > > > >> > > > '9', '86', '204', 'Shared' > > > > >> > > > '10', '87', '204', 'Shared' > > > > >> > > > > > > > >> > > > > > > > >> > > > On Wed, Mar 20, 2019 at 12:18 AM Sergey Levitskiy < > > > > >> > > serg...@hotmail.com > > > > >> > > > > > > > > >> > > > wrote: > > > > >> > > > > > > > >> > > > > Check network_offering table for value in column > > > > >> > > > > redundant_router_service for the network offering you > > > use. > > > > >> > > > > Can you also run the following: > > > > >> > > > > select name, state, removed from host where name like > > > 'r%' > > > > >> > > > > select * from domain_router; > > > > >> > > > > select * from router_network_ref; > > > > >> > > > > > > > > >> > > > > Cloudstack is supposed to recreate you VR. If it is > not > > > > >> happening > > > > >> > > > there is > > > > >> > > > > something fundamentally wrong. I would advise to > destroy > > > > your > > > > >> VR > > > > >> > > > again. > > > > >> > > > > Stop you management server. Rotate management server > log > > > and > > > > >> > start > > > > >> > > it > > > > >> > > > > again. If your VR doesn't start in few min, post your > > > > complete > > > > >> > > > management > > > > >> > > > > server log and agent log again. > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > > >> > > > > On 3/19/19, 2:56 PM, "Jevgeni Zolotarjov" < > > > > >> > j.zolotar...@gmail.com> > > > > >> > > > wrote: > > > > >> > > > > > > > > >> > > > > >>>Network offering needs to be change to > > > non-redundant > > > > >> > > > > How do I do that? > > > > >> > > > > > > > > >> > > > > On Tue, Mar 19, 2019 at 11:47 PM Sergey Levitskiy > < > > > > >> > > > serg...@hotmail.com > > > > >> > > > > > > > > > >> > > > > wrote: > > > > >> > > > > > > > > >> > > > > > Network offering needs to be change to > > > non-redundant. > > > > >> Most > > > > >> > > > likely > > > > >> > > > > old bug > > > > >> > > > > > is resurfaced. > > > > >> > > > > > > > > https://issues.apache.org/jira/browse/CLOUDSTACK-9024 > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > On 3/19/19, 2:19 PM, "Jevgeni Zolotarjov" < > > > > >> > > > j.zolotar...@gmail.com> > > > > >> > > > > wrote: > > > > >> > > > > > > > > > >> > > > > > Hello > > > > >> > > > > > > > > > >> > > > > > I did exactly like you suggested. > > > > >> > > > > > > > > > >> > > > > > After UPDATE on db, I can see the router in > > > > >> cloudstack > > > > >> > > > console. > > > > >> > > > > > But attempt to restart network fails. > > > > >> > > > > > I get error: > > > > >> > > > > > Resource [DataCenter:1] is unreachable: > Can't > > > find > > > > >> all > > > > >> > > > necessary > > > > >> > > > > > running > > > > >> > > > > > routers! > > > > >> > > > > > > > > > >> > > > > > I rechecked agents on both servers. They > look > > > > >> running > > > > >> > OK. > > > > >> > > > > > > > > > >> > > > > > On Tue, Mar 19, 2019 at 10:00 PM Andrija > > Panic < > > > > >> > > > > > andrija.pa...@gmail.com> > > > > >> > > > > > wrote: > > > > >> > > > > > > > > > >> > > > > > > Ok, so: > > > > >> > > > > > > > > > > >> > > > > > > 1.BACKUP YOUR DB - in case there is > issues, > > > you > > > > >> will > > > > >> > > > want to > > > > >> > > > > restore > > > > >> > > > > > > 2. I tried to reproduce your case on > 4.11.2 > > > > >> (though > > > > >> > > > didn't do > > > > >> > > > > any > > > > >> > > > > > > maintenance etc.) - and could not - I have > > > > >> > artificially > > > > >> > > > marked > > > > >> > > > > > existing VR > > > > >> > > > > > > as removed in DB - and tried to restart > > > network > > > > >> and > > > > >> > it > > > > >> > > > worked > > > > >> > > > > just > > > > >> > > > > > fine. > > > > >> > > > > > > > > > > >> > > > > > > Let's try to "undelete" the VR for that > > > network > > > > - > > > > >> > > though > > > > >> > > > I > > > > >> > > > > can't be > > > > >> > > > > > sure if > > > > >> > > > > > > this will work or not. > > > > >> > > > > > > > > > > >> > > > > > > Find VR based on the network UUID: > > > > >> > > > > > > SELECT * FROM vm_instance WHERE id > IN > > > > >> (SELECT > > > > >> > > > > instance_id from > > > > >> > > > > > nics > > > > >> > > > > > > where network_id IN (SELECT id FROM > networks > > > > WHERE > > > > >> > > > > > > > > UUID="65ca9a05-ff96-4563-ab9c-ffb610dc8b73")); > > > > >> > > > > > > Obviously, replace this UUID with your > > network > > > > >> UUID. > > > > >> > > > > > > SQL above will return all VMs (user VMs, > > VRs, > > > > etc, > > > > >> > > > delete or > > > > >> > > > > not...) > > > > >> > > > > > that > > > > >> > > > > > > have NIC inside the network with that > UUID - > > > > look > > > > >> for > > > > >> > > the > > > > >> > > > > r-XX-VM > > > > >> > > > > > being a > > > > >> > > > > > > router. > > > > >> > > > > > > > > > > >> > > > > > > Now that you have the name of the VR (i.e. > > > > >> r-4-VM) - > > > > >> > > you > > > > >> > > > want > > > > >> > > > > to set > > > > >> > > > > > 2 > > > > >> > > > > > > fields to some other value than what it > > might > > > be > > > > >> > > > already, as > > > > >> > > > > below: > > > > >> > > > > > > > > > > >> > > > > > > Need to set field "state" to Stopped: > > > > >> > > > > > > UPDATE cloud.vm_instance SET > > > > state='Stopped' > > > > >> > > WHERE > > > > >> > > > > > name="r-4-VM"; > > > > >> > > > > > > (what state is set currently ???) > > > > >> > > > > > > > > > > >> > > > > > > Need to set "removed" to NULL value: > > > > >> > > > > > > UPDATE cloud.vm_instance SET > removed = > > > > NULL > > > > >> > WHERE > > > > >> > > > > > NAME="r-4-VM"; (is > > > > >> > > > > > > there a removed date already set ???) > > > > >> > > > > > > > > > > >> > > > > > > Obviously make sure that this VR is not > > > running > > > > on > > > > >> > any > > > > >> > > > > hypervisor > > > > >> > > > > > and if > > > > >> > > > > > > running, virsh destroy it... > > > > >> > > > > > > > > > > >> > > > > > > AFTER above has been done (VR is > considered > > to > > > > >> exist > > > > >> > > but > > > > >> > > > > stopped, > > > > >> > > > > > from > > > > >> > > > > > > CloudStack point of view) - try to RESTART > > the > > > > >> > network > > > > >> > > > (don't > > > > >> > > > > bother > > > > >> > > > > > > deleteing it, since there are VMs in that > > > > network) > > > > >> > > > > > > > > > > >> > > > > > > I'm not 100% positive this will fix your > > > issue, > > > > >> but > > > > >> > > > doesn't > > > > >> > > > > hurt to > > > > >> > > > > > try > > > > >> > > > > > > > > > > >> > > > > > > If above doesn't work - I would still > take a > > > > look > > > > >> > into > > > > >> > > > agents > > > > >> > > > > and if > > > > >> > > > > > they > > > > >> > > > > > > are still connected - optionally, restart > > > agents > > > > >> on > > > > >> > > both > > > > >> > > > hosts > > > > >> > > > > once > > > > >> > > > > > more > > > > >> > > > > > > and confirm they are connected and up. > > > > >> > > > > > > > > > > >> > > > > > > Let us know how it goes (and please backup > > DB > > > > once > > > > >> > more > > > > >> > > > before > > > > >> > > > > any > > > > >> > > > > > actions > > > > >> > > > > > > !) > > > > >> > > > > > > > > > > >> > > > > > > Andrija > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > On Tue, 19 Mar 2019 at 20:41, Jevgeni > > > > Zolotarjov < > > > > >> > > > > > j.zolotar...@gmail.com> > > > > >> > > > > > > wrote: > > > > >> > > > > > > > > > > >> > > > > > > > Yes. Just a single network. > > > > >> > > > > > > > > > > > >> > > > > > > > On Tue, 19 Mar 2019, 21:39 Andrija > Panic, > > < > > > > >> > > > > andrija.pa...@gmail.com > > > > >> > > > > > > > > > > >> > > > > > > wrote: > > > > >> > > > > > > > > > > > >> > > > > > > > > Just one more clarification - this is > > > > Isolate > > > > >> > > single > > > > >> > > > > network (not > > > > >> > > > > > > Shared > > > > >> > > > > > > > > Network, not VPC) ? > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > On Tue, Mar 19, 2019, 19:36 Jevgeni > > > > >> Zolotarjov < > > > > >> > > > > > j.zolotar...@gmail.com > > > > >> > > > > > > > > > > > >> > > > > > > > > wrote: > > > > >> > > > > > > > > > > > > >> > > > > > > > > > Name defaultGuestNetwork > > > > >> > > > > > > > > > ID > > 4ba834ed-48f3-468f-b667-9bb2d2c258f1 > > > > >> > > > > > > > > > Zone PBT zone 1 > > > > >> > > > > > > > > > Description defaultGuestNetwork > > > > >> > > > > > > > > > Type Shared > > > > >> > > > > > > > > > State Setup > > > > >> > > > > > > > > > VPC ID N/A > > > > >> > > > > > > > > > Persistent No > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > On Tue, Mar 19, 2019 at 8:29 PM > > Andrija > > > > >> Panic < > > > > >> > > > > > > andrija.pa...@gmail.com > > > > >> > > > > > > > > > > > > >> > > > > > > > > > wrote: > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > Share the network id and name as > > seen > > > > from > > > > >> > > GUI... > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > On Tue, Mar 19, 2019, 19:27 > Jevgeni > > > > >> > Zolotarjov > > > > >> > > < > > > > >> > > > > > > > j.zolotar...@gmail.com > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > wrote: > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > >>> 1. Confirm please rolling > > > restart > > > > is > > > > >> > set > > > > >> > > > to false > > > > >> > > > > > please - > > > > >> > > > > > > > double > > > > >> > > > > > > > > > > check > > > > >> > > > > > > > > > > > Double checked - It is set to > > false > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > >>>>2. If so - do you know the > > name > > > of > > > > >> VR > > > > >> > > > which you > > > > >> > > > > > deleted ? Is > > > > >> > > > > > > it > > > > >> > > > > > > > > > last > > > > >> > > > > > > > > > > > one > > > > >> > > > > > > > > > > > >>> ever created - if so we can > > find > > > > it > > > > >> > > > easily... > > > > >> > > > > > > > > > > > I dont know the name > > > > >> > > > > > > > > > > > Is there a way to fetch it from > > DB? > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > On Tue, Mar 19, 2019 at 7:58 PM > > > > Andrija > > > > >> > > Panic < > > > > >> > > > > > > > > andrija.pa...@gmail.com > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > wrote: > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > Right...still complaining on > > > missing > > > > >> > > running > > > > >> > > > > routers. > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > 1. Confirm please rolling > > restart > > > is > > > > >> set > > > > >> > to > > > > >> > > > false > > > > >> > > > > please > > > > >> > > > > > - > > > > >> > > > > > > double > > > > >> > > > > > > > > > check > > > > >> > > > > > > > > > > > > 2. If so - do you know the > name > > of > > > > VR > > > > >> > which > > > > >> > > > you > > > > >> > > > > deleted > > > > >> > > > > > ? Is it > > > > >> > > > > > > > > last > > > > >> > > > > > > > > > > one > > > > >> > > > > > > > > > > > > ever created - if so we can > find > > > it > > > > >> > > easily... > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > On Tue, Mar 19, 2019, 18:40 > > > Jevgeni > > > > >> > > > Zolotarjov < > > > > >> > > > > > > > > > j.zolotar...@gmail.com > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > wrote: > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > Here is management server > log > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > https://drive.google.com/open?id=1H2jI0roeiWxtzReB8qV6QxDkNpaki99A > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > On Tue, Mar 19, 2019 at 7:29 > > PM > > > > >> Andrija > > > > >> > > > Panic < > > > > >> > > > > > > > > > > andrija.pa...@gmail.com > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > wrote: > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > I would also try to > > "undelete" > > > > VR > > > > >> in > > > > >> > > DB, > > > > >> > > > but > > > > >> > > > > let's > > > > >> > > > > > keep > > > > >> > > > > > > this > > > > >> > > > > > > > as > > > > >> > > > > > > > > > > last > > > > >> > > > > > > > > > > > > > step. > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > On Tue, Mar 19, 2019, > 18:24 > > > > >> Andrija > > > > >> > > > Panic < > > > > >> > > > > > > > > > andrija.pa...@gmail.com > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > wrote: > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > Disclaimer: from mobile > > > > device. > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > I don't see any special > > > > failure > > > > >> in > > > > >> > > > agent log, > > > > >> > > > > > except some > > > > >> > > > > > > > > long > > > > >> > > > > > > > > > > > > running > > > > >> > > > > > > > > > > > > > > > migration, timeout for > > > > graceful > > > > >> VM > > > > >> > > > shutdown > > > > >> > > > > etc > > > > >> > > > > > (and > > > > >> > > > > > > agent > > > > >> > > > > > > > > > > restart) > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > You did not send mgmt > log > > > > after > > > > >> > last > > > > >> > > > network > > > > >> > > > > > restart > > > > >> > > > > > > > failure > > > > >> > > > > > > > > ? > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > On Tue, Mar 19, 2019, > > 17:29 > > > > >> Jevgeni > > > > >> > > > > Zolotarjov < > > > > >> > > > > > > > > > > > > j.zolotar...@gmail.com > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > wrote: > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> Under > > > > >> > > > > > > > > > > > > > > >> Infrastructure / Hosts: > > > > >> > > > > > > > > > > > > > > >> both hosts are enabled > > and > > > > >> green > > > > >> > > > (Unsecure > > > > >> > > > > though) > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > >> agent.log - > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > >> > > > https://drive.google.com/open?id=1rATxHKqgNKo2kD23BtlrZy_9gFXC-Bq- > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > >> I set > > > network.rolling.restart > > > > >> to > > > > >> > > > false now. > > > > >> > > > > > Restarted > > > > >> > > > > > > > > > management > > > > >> > > > > > > > > > > > > > server > > > > >> > > > > > > > > > > > > > > - > > > > >> > > > > > > > > > > > > > > >> same problem - cannot > > > restart > > > > >> not > > > > >> > > > delete > > > > >> > > > > network > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > >> On Tue, Mar 19, 2019 at > > > 5:48 > > > > PM > > > > >> > > Boris > > > > >> > > > > Stoyanov < > > > > >> > > > > > > > > > > > > > > >> > > > boris.stoya...@shapeblue.com > > > > > > > > > >> > > > > > > > > > > > > > > >> wrote: > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > >> > Hi, you shouldn’t be > > > > worried > > > > >> > about > > > > >> > > > data > > > > >> > > > > as long > > > > >> > > > > > as you > > > > >> > > > > > > > > use a > > > > >> > > > > > > > > > > > > > separate > > > > >> > > > > > > > > > > > > > > >> > shared storage. > > > > >> > > > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > > > >> > What does the > > cloudstack > > > > >> console > > > > >> > > say > > > > >> > > > > about your > > > > >> > > > > > host? > > > > >> > > > > > > Is > > > > >> > > > > > > > > it > > > > >> > > > > > > > > > > up? > > > > >> > > > > > > > > > > > If > > > > >> > > > > > > > > > > > > > > it’s > > > > >> > > > > > > > > > > > > > > >> up > > > > >> > > > > > > > > > > > > > > >> > then you should be > able > > > to > > > > >> > deploy > > > > >> > > a > > > > >> > > > VM on > > > > >> > > > > it. > > > > >> > > > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > > > >> > If not, you’ll need > to > > > > >> > investigate > > > > >> > > > the > > > > >> > > > > reason > > > > >> > > > > > > > > > cloudstack-agent > > > > >> > > > > > > > > > > > is > > > > >> > > > > > > > > > > > > > not > > > > >> > > > > > > > > > > > > > > >> > connected, is the > agent > > > > >> running? > > > > >> > > If > > > > >> > > > not > > > > >> > > > > just > > > > >> > > > > > start it > > > > >> > > > > > > > and > > > > >> > > > > > > > > > see > > > > >> > > > > > > > > > > if > > > > >> > > > > > > > > > > > > it > > > > >> > > > > > > > > > > > > > > >> > connects. > > > > >> > > > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > > > >> > If it’s running but > > still > > > > not > > > > >> > > > connected > > > > >> > > > > please > > > > >> > > > > > share > > > > >> > > > > > > the > > > > >> > > > > > > > > > > > > agent.log. > > > > >> > > > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > > > >> > If nothing helps at > all > > > you > > > > >> can > > > > >> > > > remove > > > > >> > > > > the host > > > > >> > > > > > and > > > > >> > > > > > > add > > > > >> > > > > > > > it > > > > >> > > > > > > > > > > > again, > > > > >> > > > > > > > > > > > > > but > > > > >> > > > > > > > > > > > > > > >> > let’s try to > > troubleshoot > > > > >> first. > > > > >> > > > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > > > >> > Bobby. > > > > >> > > > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > > > >> > > > > > boris.stoya...@shapeblue.com > > > > >> > > > > > > > > > > > > > > >> > www.shapeblue.com > > > > >> > > > > > > > > > > > > > > >> > Amadeus House, Floral > > > > Street, > > > > >> > > London > > > > >> > > > > WC2E 9DPUK > > > > >> > > > > > > > > > > > > > > >> > @shapeblue > > > > >> > > > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > > > >> > > On 19 Mar 2019, at > > > 17:29, > > > > >> > > Jevgeni > > > > >> > > > > Zolotarjov < > > > > >> > > > > > > > > > > > > > > j.zolotar...@gmail.com> > > > > >> > > > > > > > > > > > > > > >> > wrote: > > > > >> > > > > > > > > > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > >> > > Guys, please help > > with > > > > it. > > > > >> > > > > > > > > > > > > > > >> > > What can be done > > here? > > > > >> > > > > > > > > > > > > > > >> > > There is too much > > > > valuable > > > > >> > data. > > > > >> > > > > > > > > > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > >> > > On Tue, Mar 19, > 2019 > > at > > > > >> 4:21 > > > > >> > PM > > > > >> > > > Jevgeni > > > > >> > > > > > Zolotarjov < > > > > >> > > > > > > > > > > > > > > >> > > j.zolotar...@gmail.com > > > > > > > >> > > > > > > > > > > > > > > >> > > wrote: > > > > >> > > > > > > > > > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > >> > >> Tried that just > now > > > and > > > > >> got > > > > >> > > > error: > > > > >> > > > > > > > > > > > > > > >> > >> Resource > > > [DataCenter:1] > > > > is > > > > >> > > > unreachable: > > > > >> > > > > > Can't find > > > > >> > > > > > > > all > > > > >> > > > > > > > > > > > > necessary > > > > >> > > > > > > > > > > > > > > >> running > > > > >> > > > > > > > > > > > > > > >> > >> routers! > > > > >> > > > > > > > > > > > > > > >> > >> > > > > >> > > > > > > > > > > > > > > >> > >> In the log I see: > > > > >> > > > > > > > > > > > > > > >> > >> > > > > >> > > > > ============================================= > > > > >> > > > > > > > > > > > > > > >> > >> > > > > >> > > > > > > > > > > > > > > >> > >> 2019-03-19 > > > 14:20:39,644 > > > > >> DEBUG > > > > >> > > > > > > > > > > [o.a.c.e.o.NetworkOrchestrator] > > > > >> > > > > > > > > > > > > > > >> > >> > > > > >> > > (API-Job-Executor-4:ctx-7b6b69eb > > > > >> > > > > job-5093 > > > > >> > > > > > > > ctx-9be30648) > > > > >> > > > > > > > > > > > > > > >> (logid:265a6099) > > > > >> > > > > > > > > > > > > > > >> > >> Restarting network > > > > 204... > > > > >> > > > > > > > > > > > > > > >> > >> 2019-03-19 > > > 14:20:39,645 > > > > >> DEBUG > > > > >> > > > > > > > > > > [o.a.c.e.o.NetworkOrchestrator] > > > > >> > > > > > > > > > > > > > > >> > >> > > > > >> > > (API-Job-Executor-4:ctx-7b6b69eb > > > > >> > > > > job-5093 > > > > >> > > > > > > > ctx-9be30648) > > > > >> > > > > > > > > > > > > > > >> (logid:265a6099) > > > > >> > > > > > > > > > > > > > > >> > >> Performing rolling > > > > >> restart of > > > > >> > > > routers > > > > >> > > > > of > > > > >> > > > > > network > > > > >> > > > > > > > > > > > > > Ntwk[204|Guest|6] > > > > >> > > > > > > > > > > > > > > >> > >> 2019-03-19 > > > 14:20:39,649 > > > > >> DEBUG > > > > >> > > > > > > > > > > [o.a.c.e.o.NetworkOrchestrator] > > > > >> > > > > > > > > > > > > > > >> > >> > > > > >> > > (API-Job-Executor-4:ctx-7b6b69eb > > > > >> > > > > job-5093 > > > > >> > > > > > > > ctx-9be30648) > > > > >> > > > > > > > > > > > > > > >> (logid:265a6099) > > > > >> > > > > > > > > > > > > > > >> > >> Asking > VirtualRouter > > > to > > > > >> > > > implemenet > > > > >> > > > > > > Ntwk[204|Guest|6] > > > > >> > > > > > > > > > > > > > > >> > >> 2019-03-19 > > > 14:20:39,658 > > > > >> ERROR > > > > >> > > > > > > > > > [c.c.a.ApiAsyncJobDispatcher] > > > > >> > > > > > > > > > > > > > > >> > >> > > > > >> > > (API-Job-Executor-4:ctx-7b6b69eb > > > > >> > > > > job-5093) > > > > >> > > > > > > > > > (logid:265a6099) > > > > >> > > > > > > > > > > > > > > >> Unexpected > > > > >> > > > > > > > > > > > > > > >> > >> exception while > > > > executing > > > > >> > > > > > > > > > > > > > > >> > >> > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > >> org.apache.cloudstack.api.command.user.network.RestartNetworkCmd > > > > >> > > > > > > > > > > > > > > >> > >> > > > > >> > > > > > > com.cloud.exception.ResourceUnavailableException: > > > > >> > > > > > > > > > Resource > > > > >> > > > > > > > > > > > > > > >> > [DataCenter:1] > > > > >> > > > > > > > > > > > > > > >> > >> is unreachable: > > Can't > > > > find > > > > >> > all > > > > >> > > > > necessary > > > > >> > > > > > running > > > > >> > > > > > > > > routers! > > > > >> > > > > > > > > > > > > > > >> > >> at > > > > >> > > > > > > > > > > > > > > >> > >> > > > > >> > > > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:243) > > > > >> > > > > > > > > > > > > > > >> > >> at > > > > >> > > > > > > > > > > > > > > >> > >> > > > > >> > > > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElements(NetworkOrchestrator.java:1203) > > > > >> > > > > > > > > > > > > > > >> > >> at > > > > >> > > > > > > > > > > > > > > >> > >> > > > > >> > > > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.rollingRestartRouters(NetworkOrchestrator.java:2948) > > > > >> > > > > > > > > > > > > > > >> > >> at > > > > >> > > > > > > > > > > > > > > >> > >> > > > > >> > > > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:2853) > > > > >> > > > > > > > > > > > > > > >> > >> at > > > > >> > > > > > > > > > > > > > > >> > >> > > > > >> > > > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1883) > > > > >> > > > > > > > > > > > > > > >> > >> at > > > > >> > > > > > > > > > > > > > > >> > > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native > > > > >> > > > > > > > > > > > > > > Method) > > > > >> > > > > > > > > > > > > > > >> > >> at > > > > >> > > > > > > > > > > > > > > >> > > > sun.reflect.NativeMethodAccessorImpl.invoke(Unknown > > > > >> > > > > > > > > > > > > > > Source) > > > > >> > > > > > > > > > > > > > > >> > >> at > > > > >> > > > > > > > > > > > > > > > > >> > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown > > > > >> > > > > > > > > > > > > > > >> > Source) > > > > >> > > > > > > > > > > > > > > >> > >> at > > > > >> > > > > > java.lang.reflect.Method.invoke(Unknown > > > > >> > > > > > > > > Source) > > > > >> > > > > > > > > > > > > > > >> > >> at > > > > >> > > > > > > > > > > > > > > >> > >> > > > > >> > > > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338) > > > > >> > > > > > > > > > > > > > > >> > >> at > > > > >> > > > > > > > > > > > > > > >> > >> > > > > >> > > > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197) > > > > >> > > > > > > > > > > > > > > >> > >> at > > > > >> > > > > > > > > > > > > > > >> > >> > > > > >> > > > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) > > > > >> > > > > > > > > > > > > > > >> > >> at > > > > >> > > > > > > > > > > > > > > >> > >> > > > > >> > > > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107) > > > > >> > > > > > > > > > > > > > > >> > >> at > > > > >> > > > > > > > > > > > > > > >> > >> > > > > >> > > > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) > > > > >> > > > > > > > > > > > > > > >> > >> at > > > > >> > > > > > > > > > > > > > > >> > >> > > > > >> > > > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51) > > > > >> > > > > > > > > > > > > > > >> > >> at > > > > >> > > > > > > > > > > > > > > >> > >> > > > > >> > > > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) > > > > >> > > > > > > > > > > > > > > >> > >> at > > > > >> > > > > > > > > > > > > > > >> > >> > > > > >> > > > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) > > > > >> > > > > > > > > > > > > > > >> > >> at > > > > >> > > > > > > > > > > > > > > >> > >> > > > > >> > > > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) > > > > >> > > > > > > > > > > > > > > >> > >> at > > > > >> > > > > > > > > > > > > > > >> > >> > > > > >> > > > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) > > > > >> > > > > > > > > > > > > > > >> > >> at > > > > >> > > > > > > > > > > > > com.sun.proxy.$Proxy229.restartNetwork(Unknown > > > > >> > > > > > > > > > > > > Source) > > > > >> > > > > > > > > > > > > > > >> > >> at > > > > >> > > > > > > > > > > > > > > >> > >> > > > > >> > > > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:99) > > > > >> > > > > > > > > > > > > > > >> > >> at > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > >> > > > com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150) > > > > >> > > > > > > > > > > > > > > >> > >> at > > > > >> > > > > > > > > > > > > > > >> > >> > > > > >> > > > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108) > > > > >> > > > > > > > > > > > > > > >> > >> at > > > > >> > > > > > > > > > > > > > > >> > >> > > > > >> > > > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581) > > > > >> > > > > > > > > > > > > > > >> > >> at > > > > >> > > > > > > > > > > > > > > >> > >> > > > > >> > > > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > > > > >> > > > > > > > > > > > > > > >> > >> at > > > > >> > > > > > > > > > > > > > > >> > >> > > > > >> > > > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > > > > >> > > > > > > > > > > > > > > >> > >> at > > > > >> > > > > > > > > > > > > > > >> > >> > > > > >> > > > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > > > > >> > > > > > > > > > > > > > > >> > >> at > > > > >> > > > > > > > > > > > > > > >> > >> > > > > >> > > > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > > > > >> > > > > > > > > > > > > > > >> > >> at > > > > >> > > > > > > > > > > > > > > >> > >> > > > > >> > > > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > > > > >> > > > > > > > > > > > > > > >> > >> at > > > > >> > > > > > > > > > > > > > > >> > >> > > > > >> > > > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529) > > > > >> > > > > > > > > > > > > > > >> > >> at > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > >> java.util.concurrent.Executors$RunnableAdapter.call(Unknown > > > > >> > > > > > > > > > > > > > > >> > >> Source) > > > > >> > > > > > > > > > > > > > > >> > >> at > > > > >> > > > > > > > java.util.concurrent.FutureTask.run(Unknown > > > > >> > > > > > > > > > > Source) > > > > >> > > > > > > > > > > > > > > >> > >> at > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > >> java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown > > > > >> > > > > > > > > > > > > > > >> > >> Source) > > > > >> > > > > > > > > > > > > > > >> > >> at > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown > > > > >> > > > > > > > > > > > > > > >> > >> Source) > > > > >> > > > > > > > > > > > > > > >> > >> at > > > > >> > > > java.lang.Thread.run(Unknown > > > > >> > > > > > Source) > > > > >> > > > > > > > > > > > > > > >> > >> 2019-03-19 > > > 14:20:39,659 > > > > >> DEBUG > > > > >> > > > > > > > > > > > > [o.a.c.f.j.i.AsyncJobManagerImpl] > > > > >> > > > > > > > > > > > > > > >> > >> > > > > >> > > (API-Job-Executor-4:ctx-7b6b69eb > > > > >> > > > > job-5093) > > > > >> > > > > > > > > > (logid:265a6099) > > > > >> > > > > > > > > > > > > > > Complete > > > > >> > > > > > > > > > > > > > > >> > async > > > > >> > > > > > > > > > > > > > > >> > >> job-5093, > jobStatus: > > > > >> FAILED, > > > > >> > > > > resultCode: 530, > > > > >> > > > > > > result: > > > > >> > > > > > > > > > > > > > > >> > >> > > > > >> > > > > > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > > > > > > org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Resource > > > > >> > > > > > > > > > > > > > > >> > >> [DataCenter:1] is > un > > > > >> > > > > > > > > > > > > > > >> > >> > > > > >> > > > > > > > > > > > > > > >> > >> On Tue, Mar 19, > 2019 > > > at > > > > >> 4:19 > > > > >> > PM > > > > >> > > > Andrija > > > > >> > > > > > Panic < > > > > >> > > > > > > > > > > > > > > >> > > > > > andrija.pa...@shapeblue.com> > > > > >> > > > > > > > > > > > > > > >> > >> wrote: > > > > >> > > > > > > > > > > > > > > >> > >> > > > > >> > > > > > > > > > > > > > > >> > >>> > > > > >> > > > > > > > > > > > > > > >> > >>> Your network > can't > > be > > > > >> > deleted > > > > >> > > > due to > > > > >> > > > > "Can't > > > > >> > > > > > delete > > > > >> > > > > > > > the > > > > >> > > > > > > > > > > > > network, > > > > >> > > > > > > > > > > > > > > not > > > > >> > > > > > > > > > > > > > > >> all > > > > >> > > > > > > > > > > > > > > >> > >>> user vms are > > > expunged. > > > > Vm > > > > >> > > > > > > > > > > > > > > >> > >>> > VM[User|i-2-11-VM] > > is > > > > in > > > > >> > > Stopped > > > > >> > > > > state" - > > > > >> > > > > > which is > > > > >> > > > > > > > > fine. > > > > >> > > > > > > > > > > > > > > >> > >>> > > > > >> > > > > > > > > > > > > > > >> > >>> You should be > able > > to > > > > >> just > > > > >> > > > start the > > > > >> > > > > user > > > > >> > > > > > VM - but > > > > >> > > > > > > > if > > > > >> > > > > > > > > > you > > > > >> > > > > > > > > > > > have > > > > >> > > > > > > > > > > > > > > >> actually > > > > >> > > > > > > > > > > > > > > >> > >>> delete the VR > > itself, > > > > >> then > > > > >> > > just > > > > >> > > > do > > > > >> > > > > Network > > > > >> > > > > > restart > > > > >> > > > > > > > > with > > > > >> > > > > > > > > > > > > > "cleanup" > > > > >> > > > > > > > > > > > > > > >> and > > > > >> > > > > > > > > > > > > > > >> > it > > > > >> > > > > > > > > > > > > > > >> > >>> will recreate a > new > > > VR, > > > > >> > after > > > > >> > > > which > > > > >> > > > > you > > > > >> > > > > > should be > > > > >> > > > > > > > able > > > > >> > > > > > > > > > to > > > > >> > > > > > > > > > > > > start > > > > >> > > > > > > > > > > > > > > the > > > > >> > > > > > > > > > > > > > > >> VM. > > > > >> > > > > > > > > > > > > > > >> > >>> > > > > >> > > > > > > > > > > > > > > >> > >>> Andrija > > > > >> > > > > > > > > > > > > > > >> > >>> > > > > >> > > > > > > > > > > > > > > >> > >>> > > > > >> andrija.pa...@shapeblue.com > > > > >> > > > > > > > > > > > > > > >> > >>> > www.shapeblue.com > > > > >> > > > > > > > > > > > > > > >> > >>> Amadeus House, > > Floral > > > > >> > Street, > > > > >> > > > London > > > > >> > > > > WC2E > > > > >> > > > > > 9DPUK > > > > >> > > > > > > > > > > > > > > >> > >>> @shapeblue > > > > >> > > > > > > > > > > > > > > >> > >>> > > > > >> > > > > > > > > > > > > > > >> > >>> > > > > >> > > > > > > > > > > > > > > >> > >>> > > > > >> > > > > > > > > > > > > > > >> > >>> > > > > >> > > > > > > > > > > > > > > >> > >>> -----Original > > > > >> Message----- > > > > >> > > > > > > > > > > > > > > >> > >>> From: Jevgeni > > > > Zolotarjov > > > > >> < > > > > >> > > > > > j.zolotar...@gmail.com> > > > > >> > > > > > > > > > > > > > > >> > >>> Sent: 19 March > 2019 > > > > 15:10 > > > > >> > > > > > > > > > > > > > > >> > >>> To: > > > > >> > > users@cloudstack.apache.org > > > > >> > > > > > > > > > > > > > > >> > >>> Subject: Re: > > Disaster > > > > >> after > > > > >> > > > > maintenance > > > > >> > > > > > > > > > > > > > > >> > >>> > > > > >> > > > > > > > > > > > > > > >> > >>> I mean I cannot > > > delete > > > > >> > > network: > > > > >> > > > In the > > > > >> > > > > > management > > > > >> > > > > > > > > server > > > > >> > > > > > > > > > > > log I > > > > >> > > > > > > > > > > > > > see > > > > >> > > > > > > > > > > > > > > >> > >>> > > > > >> > > > > > > > > > > > > > > >> > >>> > > > > >> > > > > > > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > >> -- > > > > >> > > > > >> Andrija Panić > > > > >> > > > > > > > > > > > > > > > > > > -- > > > > > > Andrija Panić > > > > > > > > -- > > Andrija Panić >
