>>>>Check network_offering table for value in column redundant_router_service for the network offering you use. in table network_offering_table all records have redundant_router_service = 0
Can you also run the following: >>>select name, state, removed from host where name like 'r%' returns zero rows - nothing >>>select * from domain_router; # id, element_id, public_mac_address, public_ip_address, public_netmask, guest_netmask, guest_ip_address, is_redundant_router, priority, redundant_state, stop_pending, role, template_version, scripts_version, vpc_id, update_state '4', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0', 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 15:37:28 UTC 2018', '57db7bd8118977a5f2cd3ef1c7503633\n', NULL, NULL '49', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0', 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 15:37:28 UTC 2018', 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL '73', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0', 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 15:37:28 UTC 2018', 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL '74', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0', 'VIRTUAL_ROUTER', NULL, NULL, NULL, NULL '75', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0', 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 15:37:28 UTC 2018', 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL '76', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0', 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 15:37:28 UTC 2018', 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL '77', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0', 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.1 Fri Jun 22 07:52:17 UTC 2018', 'c03a474302d89fa82d345e10fe4cb751\n', NULL, 'UPDATE_FAILED' '80', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0', 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.1 Fri Jun 22 07:52:17 UTC 2018', 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL '85', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0', 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.1 Fri Jun 22 07:52:17 UTC 2018', 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL '86', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0', 'VIRTUAL_ROUTER', NULL, NULL, NULL, NULL '87', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, 'UNKNOWN', '0', 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.2.0 Mon Nov 12 15:06:49 UTC 2018', '873057731ff2cba4a1f3b2411765407c\n', NULL, NULL >>>select * from router_network_ref; # id, router_id, network_id, guest_type '1', '4', '204', 'Shared' '2', '49', '204', 'Shared' '3', '73', '204', 'Shared' '4', '75', '204', 'Shared' '5', '76', '204', 'Shared' '6', '77', '204', 'Shared' '7', '80', '204', 'Shared' '8', '85', '204', 'Shared' '9', '86', '204', 'Shared' '10', '87', '204', 'Shared' On Wed, Mar 20, 2019 at 12:18 AM Sergey Levitskiy <serg...@hotmail.com> wrote: > Check network_offering table for value in column > redundant_router_service for the network offering you use. > Can you also run the following: > select name, state, removed from host where name like 'r%' > select * from domain_router; > select * from router_network_ref; > > Cloudstack is supposed to recreate you VR. If it is not happening there is > something fundamentally wrong. I would advise to destroy your VR again. > Stop you management server. Rotate management server log and start it > again. If your VR doesn't start in few min, post your complete management > server log and agent log again. > > > > > On 3/19/19, 2:56 PM, "Jevgeni Zolotarjov" <j.zolotar...@gmail.com> wrote: > > >>>Network offering needs to be change to non-redundant > How do I do that? > > On Tue, Mar 19, 2019 at 11:47 PM Sergey Levitskiy <serg...@hotmail.com > > > wrote: > > > Network offering needs to be change to non-redundant. Most likely > old bug > > is resurfaced. > > https://issues.apache.org/jira/browse/CLOUDSTACK-9024 > > > > > > > > On 3/19/19, 2:19 PM, "Jevgeni Zolotarjov" <j.zolotar...@gmail.com> > wrote: > > > > Hello > > > > I did exactly like you suggested. > > > > After UPDATE on db, I can see the router in cloudstack console. > > But attempt to restart network fails. > > I get error: > > Resource [DataCenter:1] is unreachable: Can't find all necessary > > running > > routers! > > > > I rechecked agents on both servers. They look running OK. > > > > On Tue, Mar 19, 2019 at 10:00 PM Andrija Panic < > > andrija.pa...@gmail.com> > > wrote: > > > > > Ok, so: > > > > > > 1.BACKUP YOUR DB - in case there is issues, you will want to > restore > > > 2. I tried to reproduce your case on 4.11.2 (though didn't do > any > > > maintenance etc.) - and could not - I have artificially marked > > existing VR > > > as removed in DB - and tried to restart network and it worked > just > > fine. > > > > > > Let's try to "undelete" the VR for that network - though I > can't be > > sure if > > > this will work or not. > > > > > > Find VR based on the network UUID: > > > SELECT * FROM vm_instance WHERE id IN (SELECT > instance_id from > > nics > > > where network_id IN (SELECT id FROM networks WHERE > > > UUID="65ca9a05-ff96-4563-ab9c-ffb610dc8b73")); > > > Obviously, replace this UUID with your network UUID. > > > SQL above will return all VMs (user VMs, VRs, etc, delete or > not...) > > that > > > have NIC inside the network with that UUID - look for the > r-XX-VM > > being a > > > router. > > > > > > Now that you have the name of the VR (i.e. r-4-VM) - you want > to set > > 2 > > > fields to some other value than what it might be already, as > below: > > > > > > Need to set field "state" to Stopped: > > > UPDATE cloud.vm_instance SET state='Stopped' WHERE > > name="r-4-VM"; > > > (what state is set currently ???) > > > > > > Need to set "removed" to NULL value: > > > UPDATE cloud.vm_instance SET removed = NULL WHERE > > NAME="r-4-VM"; (is > > > there a removed date already set ???) > > > > > > Obviously make sure that this VR is not running on any > hypervisor > > and if > > > running, virsh destroy it... > > > > > > AFTER above has been done (VR is considered to exist but > stopped, > > from > > > CloudStack point of view) - try to RESTART the network (don't > bother > > > deleteing it, since there are VMs in that network) > > > > > > I'm not 100% positive this will fix your issue, but doesn't > hurt to > > try > > > > > > If above doesn't work - I would still take a look into agents > and if > > they > > > are still connected - optionally, restart agents on both hosts > once > > more > > > and confirm they are connected and up. > > > > > > Let us know how it goes (and please backup DB once more before > any > > actions > > > !) > > > > > > Andrija > > > > > > > > > On Tue, 19 Mar 2019 at 20:41, Jevgeni Zolotarjov < > > j.zolotar...@gmail.com> > > > wrote: > > > > > > > Yes. Just a single network. > > > > > > > > On Tue, 19 Mar 2019, 21:39 Andrija Panic, < > andrija.pa...@gmail.com > > > > > > wrote: > > > > > > > > > Just one more clarification - this is Isolate single > network (not > > > Shared > > > > > Network, not VPC) ? > > > > > > > > > > > > > > > > > > > > On Tue, Mar 19, 2019, 19:36 Jevgeni Zolotarjov < > > j.zolotar...@gmail.com > > > > > > > > > wrote: > > > > > > > > > > > Name defaultGuestNetwork > > > > > > ID 4ba834ed-48f3-468f-b667-9bb2d2c258f1 > > > > > > Zone PBT zone 1 > > > > > > Description defaultGuestNetwork > > > > > > Type Shared > > > > > > State Setup > > > > > > VPC ID N/A > > > > > > Persistent No > > > > > > > > > > > > On Tue, Mar 19, 2019 at 8:29 PM Andrija Panic < > > > andrija.pa...@gmail.com > > > > > > > > > > > wrote: > > > > > > > > > > > > > Share the network id and name as seen from GUI... > > > > > > > > > > > > > > On Tue, Mar 19, 2019, 19:27 Jevgeni Zolotarjov < > > > > j.zolotar...@gmail.com > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > >>> 1. Confirm please rolling restart is set to false > > please - > > > > double > > > > > > > check > > > > > > > > Double checked - It is set to false > > > > > > > > > > > > > > > > >>>>2. If so - do you know the name of VR which you > > deleted ? Is > > > it > > > > > > last > > > > > > > > one > > > > > > > > >>> ever created - if so we can find it easily... > > > > > > > > I dont know the name > > > > > > > > Is there a way to fetch it from DB? > > > > > > > > > > > > > > > > On Tue, Mar 19, 2019 at 7:58 PM Andrija Panic < > > > > > andrija.pa...@gmail.com > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Right...still complaining on missing running > routers. > > > > > > > > > > > > > > > > > > 1. Confirm please rolling restart is set to false > please > > - > > > double > > > > > > check > > > > > > > > > 2. If so - do you know the name of VR which you > deleted > > ? Is it > > > > > last > > > > > > > one > > > > > > > > > ever created - if so we can find it easily... > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 19, 2019, 18:40 Jevgeni Zolotarjov < > > > > > > j.zolotar...@gmail.com > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Here is management server log > > > > > > > > > > > > > > > > > https://drive.google.com/open?id=1H2jI0roeiWxtzReB8qV6QxDkNpaki99A > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 19, 2019 at 7:29 PM Andrija Panic < > > > > > > > andrija.pa...@gmail.com > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > I would also try to "undelete" VR in DB, but > let's > > keep > > > this > > > > as > > > > > > > last > > > > > > > > > > step. > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 19, 2019, 18:24 Andrija Panic < > > > > > > andrija.pa...@gmail.com > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > Disclaimer: from mobile device. > > > > > > > > > > > > > > > > > > > > > > > > I don't see any special failure in agent log, > > except some > > > > > long > > > > > > > > > running > > > > > > > > > > > > migration, timeout for graceful VM shutdown > etc > > (and > > > agent > > > > > > > restart) > > > > > > > > > > > > > > > > > > > > > > > > You did not send mgmt log after last network > > restart > > > > failure > > > > > ? > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 19, 2019, 17:29 Jevgeni > Zolotarjov < > > > > > > > > > j.zolotar...@gmail.com > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > >> Under > > > > > > > > > > > >> Infrastructure / Hosts: > > > > > > > > > > > >> both hosts are enabled and green (Unsecure > though) > > > > > > > > > > > >> > > > > > > > > > > > >> agent.log - > > > > > > > > > > > >> > > > > > > > > > > > > https://drive.google.com/open?id=1rATxHKqgNKo2kD23BtlrZy_9gFXC-Bq- > > > > > > > > > > > >> > > > > > > > > > > > >> I set network.rolling.restart to false now. > > Restarted > > > > > > management > > > > > > > > > > server > > > > > > > > > > > - > > > > > > > > > > > >> same problem - cannot restart not delete > network > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> On Tue, Mar 19, 2019 at 5:48 PM Boris > Stoyanov < > > > > > > > > > > > >> boris.stoya...@shapeblue.com> > > > > > > > > > > > >> wrote: > > > > > > > > > > > >> > > > > > > > > > > > >> > Hi, you shouldn’t be worried about data > as long > > as you > > > > > use a > > > > > > > > > > separate > > > > > > > > > > > >> > shared storage. > > > > > > > > > > > >> > > > > > > > > > > > > >> > What does the cloudstack console say > about your > > host? > > > Is > > > > > it > > > > > > > up? > > > > > > > > If > > > > > > > > > > > it’s > > > > > > > > > > > >> up > > > > > > > > > > > >> > then you should be able to deploy a VM on > it. > > > > > > > > > > > >> > > > > > > > > > > > > >> > If not, you’ll need to investigate the > reason > > > > > > cloudstack-agent > > > > > > > > is > > > > > > > > > > not > > > > > > > > > > > >> > connected, is the agent running? If not > just > > start it > > > > and > > > > > > see > > > > > > > if > > > > > > > > > it > > > > > > > > > > > >> > connects. > > > > > > > > > > > >> > > > > > > > > > > > > >> > If it’s running but still not connected > please > > share > > > the > > > > > > > > > agent.log. > > > > > > > > > > > >> > > > > > > > > > > > > >> > If nothing helps at all you can remove > the host > > and > > > add > > > > it > > > > > > > > again, > > > > > > > > > > but > > > > > > > > > > > >> > let’s try to troubleshoot first. > > > > > > > > > > > >> > > > > > > > > > > > > >> > Bobby. > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > boris.stoya...@shapeblue.com > > > > > > > > > > > >> > www.shapeblue.com > > > > > > > > > > > >> > Amadeus House, Floral Street, London > WC2E 9DPUK > > > > > > > > > > > >> > @shapeblue > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > On 19 Mar 2019, at 17:29, Jevgeni > Zolotarjov < > > > > > > > > > > > j.zolotar...@gmail.com> > > > > > > > > > > > >> > wrote: > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > Guys, please help with it. > > > > > > > > > > > >> > > What can be done here? > > > > > > > > > > > >> > > There is too much valuable data. > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > On Tue, Mar 19, 2019 at 4:21 PM Jevgeni > > Zolotarjov < > > > > > > > > > > > >> > j.zolotar...@gmail.com> > > > > > > > > > > > >> > > wrote: > > > > > > > > > > > >> > > > > > > > > > > > > > >> > >> Tried that just now and got error: > > > > > > > > > > > >> > >> Resource [DataCenter:1] is unreachable: > > Can't find > > > > all > > > > > > > > > necessary > > > > > > > > > > > >> running > > > > > > > > > > > >> > >> routers! > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > >> In the log I see: > > > > > > > > > > > >> > >> > ============================================= > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > >> 2019-03-19 14:20:39,644 DEBUG > > > > > > > [o.a.c.e.o.NetworkOrchestrator] > > > > > > > > > > > >> > >> (API-Job-Executor-4:ctx-7b6b69eb > job-5093 > > > > ctx-9be30648) > > > > > > > > > > > >> (logid:265a6099) > > > > > > > > > > > >> > >> Restarting network 204... > > > > > > > > > > > >> > >> 2019-03-19 14:20:39,645 DEBUG > > > > > > > [o.a.c.e.o.NetworkOrchestrator] > > > > > > > > > > > >> > >> (API-Job-Executor-4:ctx-7b6b69eb > job-5093 > > > > ctx-9be30648) > > > > > > > > > > > >> (logid:265a6099) > > > > > > > > > > > >> > >> Performing rolling restart of routers > of > > network > > > > > > > > > > Ntwk[204|Guest|6] > > > > > > > > > > > >> > >> 2019-03-19 14:20:39,649 DEBUG > > > > > > > [o.a.c.e.o.NetworkOrchestrator] > > > > > > > > > > > >> > >> (API-Job-Executor-4:ctx-7b6b69eb > job-5093 > > > > ctx-9be30648) > > > > > > > > > > > >> (logid:265a6099) > > > > > > > > > > > >> > >> Asking VirtualRouter to implemenet > > > Ntwk[204|Guest|6] > > > > > > > > > > > >> > >> 2019-03-19 14:20:39,658 ERROR > > > > > > [c.c.a.ApiAsyncJobDispatcher] > > > > > > > > > > > >> > >> (API-Job-Executor-4:ctx-7b6b69eb > job-5093) > > > > > > (logid:265a6099) > > > > > > > > > > > >> Unexpected > > > > > > > > > > > >> > >> exception while executing > > > > > > > > > > > >> > >> > > > > > > > > > > > > > org.apache.cloudstack.api.command.user.network.RestartNetworkCmd > > > > > > > > > > > >> > >> > > com.cloud.exception.ResourceUnavailableException: > > > > > > Resource > > > > > > > > > > > >> > [DataCenter:1] > > > > > > > > > > > >> > >> is unreachable: Can't find all > necessary > > running > > > > > routers! > > > > > > > > > > > >> > >> at > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:243) > > > > > > > > > > > >> > >> at > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElements(NetworkOrchestrator.java:1203) > > > > > > > > > > > >> > >> at > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.rollingRestartRouters(NetworkOrchestrator.java:2948) > > > > > > > > > > > >> > >> at > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:2853) > > > > > > > > > > > >> > >> at > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1883) > > > > > > > > > > > >> > >> at > > > > > > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native > > > > > > > > > > > Method) > > > > > > > > > > > >> > >> at > > > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(Unknown > > > > > > > > > > > Source) > > > > > > > > > > > >> > >> at > > > > > > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown > > > > > > > > > > > >> > Source) > > > > > > > > > > > >> > >> at > > java.lang.reflect.Method.invoke(Unknown > > > > > Source) > > > > > > > > > > > >> > >> at > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338) > > > > > > > > > > > >> > >> at > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197) > > > > > > > > > > > >> > >> at > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) > > > > > > > > > > > >> > >> at > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107) > > > > > > > > > > > >> > >> at > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) > > > > > > > > > > > >> > >> at > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51) > > > > > > > > > > > >> > >> at > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) > > > > > > > > > > > >> > >> at > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) > > > > > > > > > > > >> > >> at > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) > > > > > > > > > > > >> > >> at > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) > > > > > > > > > > > >> > >> at > > > > > com.sun.proxy.$Proxy229.restartNetwork(Unknown > > > > > > > > > Source) > > > > > > > > > > > >> > >> at > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:99) > > > > > > > > > > > >> > >> at > > > > > > > > > > > >> > > > > com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150) > > > > > > > > > > > >> > >> at > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108) > > > > > > > > > > > >> > >> at > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581) > > > > > > > > > > > >> > >> at > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > > > > > > > > > > > >> > >> at > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > > > > > > > > > > > >> > >> at > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > > > > > > > > > > > >> > >> at > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > > > > > > > > > > > >> > >> at > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > > > > > > > > > > > >> > >> at > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529) > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > java.util.concurrent.Executors$RunnableAdapter.call(Unknown > > > > > > > > > > > >> > >> Source) > > > > > > > > > > > >> > >> at > > > java.util.concurrent.FutureTask.run(Unknown > > > > > > > Source) > > > > > > > > > > > >> > >> at > > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown > > > > > > > > > > > >> > >> Source) > > > > > > > > > > > >> > >> at > > > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown > > > > > > > > > > > >> > >> Source) > > > > > > > > > > > >> > >> at java.lang.Thread.run(Unknown > > Source) > > > > > > > > > > > >> > >> 2019-03-19 14:20:39,659 DEBUG > > > > > > > > [o.a.c.f.j.i.AsyncJobManagerImpl] > > > > > > > > > > > >> > >> (API-Job-Executor-4:ctx-7b6b69eb > job-5093) > > > > > > (logid:265a6099) > > > > > > > > > > > Complete > > > > > > > > > > > >> > async > > > > > > > > > > > >> > >> job-5093, jobStatus: FAILED, > resultCode: 530, > > > result: > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Resource > > > > > > > > > > > >> > >> [DataCenter:1] is un > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > >> On Tue, Mar 19, 2019 at 4:19 PM Andrija > > Panic < > > > > > > > > > > > >> > andrija.pa...@shapeblue.com> > > > > > > > > > > > >> > >> wrote: > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > >>> > > > > > > > > > > > >> > >>> Your network can't be deleted due to > "Can't > > delete > > > > the > > > > > > > > > network, > > > > > > > > > > > not > > > > > > > > > > > >> all > > > > > > > > > > > >> > >>> user vms are expunged. Vm > > > > > > > > > > > >> > >>> VM[User|i-2-11-VM] is in Stopped > state" - > > which is > > > > > fine. > > > > > > > > > > > >> > >>> > > > > > > > > > > > >> > >>> You should be able to just start the > user > > VM - but > > > > if > > > > > > you > > > > > > > > have > > > > > > > > > > > >> actually > > > > > > > > > > > >> > >>> delete the VR itself, then just do > Network > > restart > > > > > with > > > > > > > > > > "cleanup" > > > > > > > > > > > >> and > > > > > > > > > > > >> > it > > > > > > > > > > > >> > >>> will recreate a new VR, after which > you > > should be > > > > able > > > > > > to > > > > > > > > > start > > > > > > > > > > > the > > > > > > > > > > > >> VM. > > > > > > > > > > > >> > >>> > > > > > > > > > > > >> > >>> Andrija > > > > > > > > > > > >> > >>> > > > > > > > > > > > >> > >>> andrija.pa...@shapeblue.com > > > > > > > > > > > >> > >>> www.shapeblue.com > > > > > > > > > > > >> > >>> Amadeus House, Floral Street, London > WC2E > > 9DPUK > > > > > > > > > > > >> > >>> @shapeblue > > > > > > > > > > > >> > >>> > > > > > > > > > > > >> > >>> > > > > > > > > > > > >> > >>> > > > > > > > > > > > >> > >>> > > > > > > > > > > > >> > >>> -----Original Message----- > > > > > > > > > > > >> > >>> From: Jevgeni Zolotarjov < > > j.zolotar...@gmail.com> > > > > > > > > > > > >> > >>> Sent: 19 March 2019 15:10 > > > > > > > > > > > >> > >>> To: users@cloudstack.apache.org > > > > > > > > > > > >> > >>> Subject: Re: Disaster after > maintenance > > > > > > > > > > > >> > >>> > > > > > > > > > > > >> > >>> I mean I cannot delete network: In the > > management > > > > > server > > > > > > > > log I > > > > > > > > > > see > > > > > > > > > > > >> > >>> > > > > > > > > > > > >> > >>> > > > > > > > ========================================================== > > > > > > > > > > > >> > >>> 019-03-19 14:06:36,316 DEBUG > > > > > > > > [o.a.c.f.j.i.AsyncJobManagerImpl] > > > > > > > > > > > >> > >>> (API-Job-Executor-1:ctx-1c0fd4dc > job-5090) > > > > > > > (logid:c734edfc) > > > > > > > > > > > >> Executing > > > > > > > > > > > >> > >>> AsyncJobVO {id:5090, userId: 2, > accountId: > > 2, > > > > > > > instanceType: > > > > > > > > > > > Network, > > > > > > > > > > > >> > >>> instanceId: 204, cmd: > > > > > > > > > > > >> > >>> > > > > > > > > > > > > > org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd, > > > > > > > > > > > >> > cmdInfo: > > > > > > > > > > > >> > >>> > > > > > > > > > > > >> > >>> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"2641","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\"interface > > > > > > > > > > > >> > >>> > > > > > > > > > > > >> > >>> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}","ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmdEventType":"NETWORK.DELETE","_":"1553004396247"}, > > > > > > > > > > > >> > >>> cmdVersion: 0, status: IN_PROGRESS, > > processStatus: > > > > 0, > > > > > > > > > > resultCode: > > > > > > > > > > > 0, > > > > > > > > > > > >> > >>> result: null, initMsid: > 264216221068220, > > > > completeMsid: > > > > > > > null, > > > > > > > > > > > >> > lastUpdated: > > > > > > > > > > > >> > >>> null, lastPolled: null, created: null} > > > > > > > > > > > >> > >>> 2019-03-19 14:06:36,351 WARN > > > > > > > > [o.a.c.e.o.NetworkOrchestrator] > > > > > > > > > > > >> > >>> (API-Job-Executor-1:ctx-1c0fd4dc > job-5090 > > > > > ctx-134954fa) > > > > > > > > > > > >> > (logid:c734edfc) > > > > > > > > > > > >> > >>> Can't delete the network, not all > user vms > > are > > > > > expunged. > > > > > > > Vm > > > > > > > > > > > >> > >>> VM[User|i-2-11-VM] is in Stopped state > > > > > > > > > > > >> > >>> 2019-03-19 14:06:36,356 DEBUG > > > > > > > > > [o.a.c.f.j.i.AsyncJobManagerImpl] > > > > > > > > > > > >> > >>> (API-Job-Executor-1:ctx-1c0fd4dc > job-5090) > > > > > > > (logid:c734edfc) > > > > > > > > > > > Complete > > > > > > > > > > > >> > async > > > > > > > > > > > >> > >>> job-5090, jobStatus: FAILED, > resultCode: > > 530, > > > > result: > > > > > > > > > > > >> > >>> > > > > > > > > > > > >> > >>> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed > > > > > > > > > > > >> > >>> to delete network"} > > > > > > > > > > > >> > >>> > > > > > > > ========================================================== > > > > > > > > > > > >> > >>> > > > > > > > > > > > >> > >>> > > > > > > > > > > > >> > >>> I deleted a router, expecting it to be > > recreated > > > on > > > > > > > deleting > > > > > > > > > > > >> network. > > > > > > > > > > > >> > But > > > > > > > > > > > >> > >>> I am unable to delete network because > of > > above > > > error > > > > > > > > > > > >> > >>> > > > > > > > > > > > >> > >>> On Tue, Mar 19, 2019 at 3:58 PM > Jevgeni > > > Zolotarjov < > > > > > > > > > > > >> > >>> j.zolotar...@gmail.com> > > > > > > > > > > > >> > >>> wrote: > > > > > > > > > > > >> > >>> > > > > > > > > > > > >> > >>>> I've managed to make libvirtd running > > > > > > > > > > > >> > >>>> Now cloudstack console shows both > hosts - > > running > > > > > > > > > > > >> > >>>> > > > > > > > > > > > >> > >>>> But now as I have removed network, > VMs are > > unable > > > > to > > > > > > > start. > > > > > > > > > > > >> > >>>> > > > > > > > > > > > >> > >>>> How can I recreate the network now? > > > > > > > > > > > >> > >>>> > > > > > > > > > > > >> > >>>> On Tue, Mar 19, 2019 at 3:14 PM Ivan > > Kudryavtsev > > > > > > > > > > > >> > >>>> <kudryavtsev...@bw-sw.com> > > > > > > > > > > > >> > >>>> wrote: > > > > > > > > > > > >> > >>>> > > > > > > > > > > > >> > >>>>> Jevgeniy, it may be a documentation > bug. > > Take s > > > > > look: > > > > > > > > > > > >> > >>>>> > > > > > > > > > > > > > > > > > https://github.com/apache/cloudstack-documentation/pull/27/files > > > > > > > > > > > >> > >>>>> > > > > > > > > > > > >> > >>>>> вт, 19 мар. 2019 г., 9:09 Jevgeni > > Zolotarjov < > > > > > > > > > > > >> j.zolotar...@gmail.com > > > > > > > > > > > >> > >: > > > > > > > > > > > >> > >>>>> > > > > > > > > > > > >> > >>>>>> That's it - libvirtd failed to > start on > > second > > > > > host. > > > > > > > > > > > >> > >>>>>> Tried restarting, but it does not > start. > > > > > > > > > > > >> > >>>>>> > > > > > > > > > > > >> > >>>>>> > > > > > > > > > > > >> > >>>>>>>> Do you have some NUMA > constraints or > > anything > > > > > which > > > > > > > > > > requires > > > > > > > > > > > >> > >>>>> particular > > > > > > > > > > > >> > >>>>>> RAM configuration? > > > > > > > > > > > >> > >>>>>> No > > > > > > > > > > > >> > >>>>>> > > > > > > > > > > > >> > >>>>>> libvirtd.service - Virtualization > daemon > > > > > > > > > > > >> > >>>>>> Loaded: loaded > > > > > > > > (/usr/lib/systemd/system/libvirtd.service; > > > > > > > > > > > >> > >>>>>> enabled; vendor preset: enabled) > > > > > > > > > > > >> > >>>>>> Active: failed (Result: > start-limit) > > since > > > Tue > > > > > > > > 2019-03-19 > > > > > > > > > > > >> > >>>>>> 13:03:07 > > > > > > > > > > > >> > >>>>> GMT; > > > > > > > > > > > >> > >>>>>> 12s ago > > > > > > > > > > > >> > >>>>>> Docs: man:libvirtd(8) > > > > > > > > > > > >> > >>>>>> https://libvirt.org > > > > > > > > > > > >> > >>>>>> Process: 892 > > ExecStart=/usr/sbin/libvirtd > > > > > > > $LIBVIRTD_ARGS > > > > > > > > > > > >> > >>>>>> (code=exited, > > > > > > > > > > > >> > >>>>>> status=1/FAILURE) > > > > > > > > > > > >> > >>>>>> Main PID: 892 (code=exited, > > status=1/FAILURE) > > > > > > > > > > > >> > >>>>>> Tasks: 19 (limit: 32768) > > > > > > > > > > > >> > >>>>>> CGroup: > /system.slice/libvirtd.service > > > > > > > > > > > >> > >>>>>> ├─11338 > /usr/sbin/libvirtd -d > > -l > > > > > > > > > > > >> > >>>>>> ├─11909 /usr/sbin/dnsmasq > > > > > > > > > > > >> > >>>>>> > > > --conf-file=/var/lib/libvirt/dnsmasq/default.conf > > > > > > > > > > > --leasefile-ro > > > > > > > > > > > >> > >>>>>> > > --dhcp-script=/usr/libexec/libvirt_leaseshelper > > > > > > > > > > > >> > >>>>>> └─11910 /usr/sbin/dnsmasq > > > > > > > > > > > >> > >>>>>> > > > --conf-file=/var/lib/libvirt/dnsmasq/default.conf > > > > > > > > > > > --leasefile-ro > > > > > > > > > > > >> > >>>>>> > > --dhcp-script=/usr/libexec/libvirt_leaseshelper > > > > > > > > > > > >> > >>>>>> > > > > > > > > > > > >> > >>>>>> Mar 19 13:03:07 > > mtl1-apphst04.mt.pbt.com.mt > > > > > > > systemd[1]: > > > > > > > > > > Failed > > > > > > > > > > > >> to > > > > > > > > > > > >> > >>>>>> start Virtualization daemon. > > > > > > > > > > > >> > >>>>>> Mar 19 13:03:07 > > mtl1-apphst04.mt.pbt.com.mt > > > > > > > systemd[1]: > > > > > > > > > Unit > > > > > > > > > > > >> > >>>>>> libvirtd.service entered failed > state. > > > > > > > > > > > >> > >>>>>> Mar 19 13:03:07 > > mtl1-apphst04.mt.pbt.com.mt > > > > > > > systemd[1]: > > > > > > > > > > > >> > >>>>> libvirtd.service > > > > > > > > > > > >> > >>>>>> failed. > > > > > > > > > > > >> > >>>>>> Mar 19 13:03:07 > > mtl1-apphst04.mt.pbt.com.mt > > > > > > > systemd[1]: > > > > > > > > > > > >> > >>>>> libvirtd.service > > > > > > > > > > > >> > >>>>>> holdoff time over, scheduling > restart. > > > > > > > > > > > >> > >>>>>> Mar 19 13:03:07 > > mtl1-apphst04.mt.pbt.com.mt > > > > > > > systemd[1]: > > > > > > > > > > > Stopped > > > > > > > > > > > >> > >>>>>> Virtualization daemon. > > > > > > > > > > > >> > >>>>>> Mar 19 13:03:07 > > mtl1-apphst04.mt.pbt.com.mt > > > > > > > systemd[1]: > > > > > > > > > > start > > > > > > > > > > > >> > >>>>>> request repeated too quickly for > > > libvirtd.service > > > > > Mar > > > > > > > 19 > > > > > > > > > > > 13:03:07 > > > > > > > > > > > >> > >>>>>> mtl1-apphst04.mt.pbt.com.mt > systemd[1]: > > Failed > > > > to > > > > > > > start > > > > > > > > > > > >> > >>>>>> Virtualization daemon. > > > > > > > > > > > >> > >>>>>> Mar 19 13:03:07 > > mtl1-apphst04.mt.pbt.com.mt > > > > > > > systemd[1]: > > > > > > > > > Unit > > > > > > > > > > > >> > >>>>>> libvirtd.service entered failed > state. > > > > > > > > > > > >> > >>>>>> Mar 19 13:03:07 > > mtl1-apphst04.mt.pbt.com.mt > > > > > > > systemd[1]: > > > > > > > > > > > >> > >>>>> libvirtd.service > > > > > > > > > > > >> > >>>>>> failed. > > > > > > > > > > > >> > >>>>>> > > > > > > > > > > > >> > >>>>>> > > > > > > > > > > > >> > >>>>>> On Tue, Mar 19, 2019 at 3:04 PM > Paul > > Angus > > > > > > > > > > > >> > >>>>>> <paul.an...@shapeblue.com> > > > > > > > > > > > >> > >>>>>> wrote: > > > > > > > > > > > >> > >>>>>> > > > > > > > > > > > >> > >>>>>>> Can you check that the cloudstack > agent > > is > > > > running > > > > > > on > > > > > > > > the > > > > > > > > > > host > > > > > > > > > > > >> > >>>>>>> and the agent logs (usual logs > > directory) Also > > > > > worth > > > > > > > > > > checking > > > > > > > > > > > >> > >>>>>>> that libvirt has started ok. Do > you > > have some > > > > > > > > > > > >> > >>>>> NUMA > > > > > > > > > > > >> > >>>>>>> constraints or anything which > requires > > > > particular > > > > > > RAM > > > > > > > > > > > >> > >>> configuration? > > > > > > > > > > > >> > >>>>>>> > > > > > > > > > > > >> > >>>>>>> paul.an...@shapeblue.com > > > > > > > > > > > >> > >>>>>>> www.shapeblue.com > > > > > > > > > > > >> > >>>>>>> Amadeus House, Floral Street, > London > > WC2E > > > 9DPUK > > > > > > > > > @shapeblue > > > > > > > > > > > >> > >>>>>>> > > > > > > > > > > > >> > >>>>>>> > > > > > > > > > > > >> > >>>>>>> > > > > > > > > > > > >> > >>>>>>> > > > > > > > > > > > >> > >>>>>>> -----Original Message----- > > > > > > > > > > > >> > >>>>>>> From: Jevgeni Zolotarjov < > > > > j.zolotar...@gmail.com> > > > > > > > > > > > >> > >>>>>>> Sent: 19 March 2019 14:49 > > > > > > > > > > > >> > >>>>>>> To: users@cloudstack.apache.org > > > > > > > > > > > >> > >>>>>>> Subject: Re: Disaster after > maintenance > > > > > > > > > > > >> > >>>>>>> > > > > > > > > > > > >> > >>>>>>> Can you try migrating a VM to the > > server that > > > > you > > > > > > > > changed > > > > > > > > > > the > > > > > > > > > > > >> RAM > > > > > > > > > > > >> > >>>>> amount? > > > > > > > > > > > >> > >>>>>>> > > > > > > > > > > > >> > >>>>>>> Also: > > > > > > > > > > > >> > >>>>>>> What is the hypervisor version? > > > > > > > > > > > >> > >>>>>>> KVM > > > > > > > > > > > >> > >>>>>>> QEMU Version : 2.0.0 > > > > > > > > > > > >> > >>>>>>> Release : 1.el7.6 > > > > > > > > > > > >> > >>>>>>> > > > > > > > > > > > >> > >>>>>>> > > > > > > > > > > > >> > >>>>>>> Host status in ACS? > > > > > > > > > > > >> > >>>>>>> 1st server: Unsecure > > > > > > > > > > > >> > >>>>>>> 2nd server: Disconnected > > > > > > > > > > > >> > >>>>>>> > > > > > > > > > > > >> > >>>>>>> Did you try to force a VM to > > start/deploy in > > > > this > > > > > > > server > > > > > > > > > > where > > > > > > > > > > > >> you > > > > > > > > > > > >> > >>>>>> changed > > > > > > > > > > > >> > >>>>>>> the RAM? > > > > > > > > > > > >> > >>>>>>> Host status became disconnected. I > > don't know > > > > how > > > > > to > > > > > > > > make > > > > > > > > > it > > > > > > > > > > > >> > >>>>> "connected" > > > > > > > > > > > >> > >>>>>>> again > > > > > > > > > > > >> > >>>>>>> > > > > > > > > > > > >> > >>>>>>> > > > > > > > > > > > >> > >>>>>>> > > > > > > > > > > > >> > >>>>>>> On Tue, Mar 19, 2019 at 2:42 PM > Rafael > > > > > Weingärtner < > > > > > > > > > > > >> > >>>>>>> rafaelweingart...@gmail.com> > wrote: > > > > > > > > > > > >> > >>>>>>> > > > > > > > > > > > >> > >>>>>>>> Can you try migrating a VM to the > > server that > > > > you > > > > > > > > changed > > > > > > > > > > the > > > > > > > > > > > >> RAM > > > > > > > > > > > >> > >>>>>> amount? > > > > > > > > > > > >> > >>>>>>>> > > > > > > > > > > > >> > >>>>>>>> Also: > > > > > > > > > > > >> > >>>>>>>> What is the hypervisor version? > > > > > > > > > > > >> > >>>>>>>> Host status in ACS? > > > > > > > > > > > >> > >>>>>>>> Did you try to force a VM to > > start/deploy in > > > > this > > > > > > > > server > > > > > > > > > > > where > > > > > > > > > > > >> > >>> you > > > > > > > > > > > >> > >>>>>>>> changed the RAM? > > > > > > > > > > > >> > >>>>>>>> > > > > > > > > > > > >> > >>>>>>>> > > > > > > > > > > > >> > >>>>>>>> On Tue, Mar 19, 2019 at 9:39 AM > Jevgeni > > > > > Zolotarjov > > > > > > > > > > > >> > >>>>>>>> <j.zolotar...@gmail.com > > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > > >> > >>>>>>>> wrote: > > > > > > > > > > > >> > >>>>>>>> > > > > > > > > > > > >> > >>>>>>>>> We have Cloudstack 4.11.2 setup > > running fine > > > > for > > > > > > few > > > > > > > > > > months > > > > > > > > > > > >> > >>> (>4) > > > > > > > > > > > >> > >>>>> The > > > > > > > > > > > >> > >>>>>>>>> setup is very simple: 2 hosts We > > decided to > > > > do a > > > > > > > > > > maintenance > > > > > > > > > > > >> to > > > > > > > > > > > >> > >>>>>>>>> increase RAM on both servers > > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > > >> > >>>>>>>>> For this we put first server to > > maintenance. > > > > All > > > > > > VMS > > > > > > > > > moved > > > > > > > > > > > to > > > > > > > > > > > >> > >>>>> second > > > > > > > > > > > >> > >>>>>>>>> host after a while. > > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > > >> > >>>>>>>>> Then first server was shutdown, > RAM > > > increased, > > > > > > > server > > > > > > > > > > turned > > > > > > > > > > > >> > >>> ON. > > > > > > > > > > > >> > >>>>>>>>> Now nothing starts on first > server. > > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > > >> > >>>>>>>>> Tried to delete network, but > this > > fails as > > > > well > > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > > >> > >>>>>>>>> Please help ! > > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > > >> > >>>>>>>>> Here is extract from log: > > > > > > > > > > > >> > >>>>>>>>> > ====================================== > > > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,064 DEBUG > > > > > > > > > > > >> > >>>>> > [o.a.c.s.SecondaryStorageManagerImpl] > > > > > > > > > > > >> > >>>>>>>>> (secstorage-1:ctx-16d6c797) > > (logid:7e3160ce) > > > > > Zone > > > > > > 1 > > > > > > > is > > > > > > > > > > ready > > > > > > > > > > > >> to > > > > > > > > > > > >> > >>>>>>>>> launch secondary storage VM > > > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,125 DEBUG > > > > > > > > > > > [c.c.c.ConsoleProxyManagerImpl] > > > > > > > > > > > >> > >>>>>>>>> (consoleproxy-1:ctx-cbd034b9) > > > (logid:0a8c8bf4) > > > > > > Zone > > > > > > > 1 > > > > > > > > is > > > > > > > > > > > ready > > > > > > > > > > > >> > >>> to > > > > > > > > > > > >> > >>>>>>>>> launch console proxy > > > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,181 DEBUG > > > > [c.c.a.ApiServlet] > > > > > > > > > > > >> > >>>>>>>>> (qtp510113906-285:ctx-6c5e11c3) > > > > (logid:cd8e30be) > > > > > > > > > > ===START=== > > > > > > > > > > > >> > >>>>>>>> 192.168.5.140 > > > > > > > > > > > >> > >>>>>>>>> -- GET > > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > > >> > >>>>>>>> > > > > > > > > > > > >> > >>>>> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > command=deleteNetwork&id=4ba834ed-48f3-468f-b667-9bb2d2c258f1&response > > > > > > > > > > > >> > >>>>>>>> =json&_=1552998473154 > > > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,186 DEBUG > > > > [c.c.a.ApiServer] > > > > > > > > > > > >> > >>>>>>>>> (qtp510113906-285:ctx-6c5e11c3 > > ctx-0cc34dc6) > > > > > > > > > > > (logid:cd8e30be) > > > > > > > > > > > >> > >>>>> CIDRs > > > > > > > > > > > >> > >>>>>>>>> from which account > > > > > > > > > > > >> > >>>>>>>>> > > > > > 'Acct[15863393-8e8d-11e7-8f52-f04da2002bbe-admin]' > > > > > > > is > > > > > > > > > > > >> > >>>>>>>> allowed > > > > > > > > > > > >> > >>>>>>>>> to perform API calls: > 0.0.0.0/0,::/0 > > > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,208 INFO > > > > > > > > > > [o.a.c.f.j.i.AsyncJobMonitor] > > > > > > > > > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 > > job-5081) > > > > > > > > > > (logid:f6751fa7) > > > > > > > > > > > >> Add > > > > > > > > > > > >> > >>>>>>>>> job-5081 into job monitoring > > > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,209 DEBUG > > > > > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl] > > > > > > > > > > > >> > >>>>>>>>> (qtp510113906-285:ctx-6c5e11c3 > > ctx-0cc34dc6) > > > > > > > > > > > (logid:cd8e30be) > > > > > > > > > > > >> > >>>>> submit > > > > > > > > > > > >> > >>>>>>>> async > > > > > > > > > > > >> > >>>>>>>>> job-5081, details: AsyncJobVO > > {id:5081, > > > > userId: > > > > > 2, > > > > > > > > > > > accountId: > > > > > > > > > > > >> > >>> 2, > > > > > > > > > > > >> > >>>>>>>>> instanceType: Network, > instanceId: > > 204, cmd: > > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > > >> > >>> > > > > > > > > > > > > > org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd, > > > > > > > > > > > >> > >>>>>>> cmdInfo: > > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > > >> > >>>>>>>> > > > > > > > > > > > >> > >>>>> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId > > > > > > > > > > > >> > >>>>>>>> > > > > > > > > > > > >> > >>>>> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > ":"2615","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\" > > > > > > > > > > > >> > >>>>>>>> interface > > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > > >> > >>>>>>>> > > > > > > > > > > > >> > >>>>> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}" > > > > > > > > > > > >> > >>>>>>>> > > > > > > > > > > > >> > >>>>> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > ,"ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmd > > > > > > > > > > > >> > >>>>>>>> > > > > EventType":"NETWORK.DELETE","_":"1552998473154"}, > > > > > > > > > > > >> > >>>>>>>>> cmdVersion: 0, status: > IN_PROGRESS, > > > > > processStatus: > > > > > > > 0, > > > > > > > > > > > >> > >>> resultCode: > > > > > > > > > > > >> > >>>>> 0, > > > > > > > > > > > >> > >>>>>>>>> result: null, initMsid: > > 264216221068220, > > > > > > > completeMsid: > > > > > > > > > > null, > > > > > > > > > > > >> > >>>>>>> lastUpdated: > > > > > > > > > > > >> > >>>>>>>>> null, lastPolled: null, > created: null} > > > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,211 DEBUG > > > > [c.c.a.ApiServlet] > > > > > > > > > > > >> > >>>>>>>>> (qtp510113906-285:ctx-6c5e11c3 > > ctx-0cc34dc6) > > > > > > > > > > > (logid:cd8e30be) > > > > > > > > > > > >> > >>>>>>>>> ===END=== > > > > > > > > > > > >> > >>>>>>>>> 192.168.5.140 -- GET > > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > > >> > >>>>>>>> > > > > > > > > > > > >> > >>>>> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > command=deleteNetwork&id=4ba834ed-48f3-468f-b667-9bb2d2c258f1&response > > > > > > > > > > > >> > >>>>>>>> =json&_=1552998473154 > > > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,212 DEBUG > > > > > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl] > > > > > > > > > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 > > job-5081) > > > > > > > > > > (logid:16897ea6) > > > > > > > > > > > >> > >>>>>>>>> Executing AsyncJobVO {id:5081, > > userId: 2, > > > > > > accountId: > > > > > > > > 2, > > > > > > > > > > > >> > >>>>>>>>> instanceType: Network, > > > > > > > > > > > >> > >>>>>>>>> instanceId: 204, cmd: > > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > > >> > >>> > > > > > > > > > > > > > org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd, > > > > > > > > > > > >> > >>>>>>> cmdInfo: > > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > > >> > >>>>>>>> > > > > > > > > > > > >> > >>>>> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId > > > > > > > > > > > >> > >>>>>>>> > > > > > > > > > > > >> > >>>>> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > ":"2615","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\" > > > > > > > > > > > >> > >>>>>>>> interface > > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > > >> > >>>>>>>> > > > > > > > > > > > >> > >>>>> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}" > > > > > > > > > > > >> > >>>>>>>> > > > > > > > > > > > >> > >>>>> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > ,"ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmd > > > > > > > > > > > >> > >>>>>>>> > > > > EventType":"NETWORK.DELETE","_":"1552998473154"}, > > > > > > > > > > > >> > >>>>>>>>> cmdVersion: 0, status: > IN_PROGRESS, > > > > > processStatus: > > > > > > > 0, > > > > > > > > > > > >> > >>> resultCode: > > > > > > > > > > > >> > >>>>> 0, > > > > > > > > > > > >> > >>>>>>>>> result: null, initMsid: > > 264216221068220, > > > > > > > completeMsid: > > > > > > > > > > null, > > > > > > > > > > > >> > >>>>>>> lastUpdated: > > > > > > > > > > > >> > >>>>>>>>> null, lastPolled: null, > created: null} > > > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,257 WARN > > > > > > > > > > > [o.a.c.e.o.NetworkOrchestrator] > > > > > > > > > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 > > job-5081 > > > > > > > > ctx-d5de7979) > > > > > > > > > > > >> > >>>>>>>>> (logid:16897ea6) Can't delete > the > > network, > > > not > > > > > all > > > > > > > > user > > > > > > > > > > vms > > > > > > > > > > > >> are > > > > > > > > > > > >> > >>>>>>>>> expunged. Vm VM[User|i-2-11-VM] > is in > > > Stopped > > > > > > state > > > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,263 DEBUG > > > > > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl] > > > > > > > > > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 > > job-5081) > > > > > > > > > > (logid:16897ea6) > > > > > > > > > > > >> > >>>>> Complete > > > > > > > > > > > >> > >>>>>>>> async > > > > > > > > > > > >> > >>>>>>>>> job-5081, jobStatus: FAILED, > > resultCode: > > > 530, > > > > > > > result: > > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > > >> > >>>>>>>> > > > > > > > > > > > >> > >>>>> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList": > > > > > > > > > > > >> > >>>>>>>> > [],"errorcode":530,"errortext":"Failed > > > > > > > > > > > >> > >>>>>>>>> to delete network"} > > > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,264 DEBUG > > > > > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl] > > > > > > > > > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 > > job-5081) > > > > > > > > > > (logid:16897ea6) > > > > > > > > > > > >> > >>>>> Publish > > > > > > > > > > > >> > >>>>>>>>> async > > > > > > > > > > > >> > >>>>>>>>> job-5081 complete on message bus > > > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,264 DEBUG > > > > > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl] > > > > > > > > > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 > > job-5081) > > > > > > > > > > (logid:16897ea6) > > > > > > > > > > > >> > >>> Wake > > > > > > > > > > > >> > >>>>> up > > > > > > > > > > > >> > >>>>>>>>> jobs related to job-5081 > > > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,264 DEBUG > > > > > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl] > > > > > > > > > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 > > job-5081) > > > > > > > > > > (logid:16897ea6) > > > > > > > > > > > >> > >>> Update > > > > > > > > > > > >> > >>>>>>>>> db status for job-5081 > > > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,265 DEBUG > > > > > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl] > > > > > > > > > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 > > job-5081) > > > > > > > > > > (logid:16897ea6) > > > > > > > > > > > >> > >>> Wake > > > > > > > > > > > >> > >>>>> up > > > > > > > > > > > >> > >>>>>>>>> jobs joined with job-5081 and > disjoin > > all > > > > > subjobs > > > > > > > > > created > > > > > > > > > > > from > > > > > > > > > > > >> > >>>>> job- > > > > > > > > > > > >> > >>>>>>>>> 5081 > > > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,267 DEBUG > > > > > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl] > > > > > > > > > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 > > job-5081) > > > > > > > > > > (logid:16897ea6) > > > > > > > > > > > >> > >>> Done > > > > > > > > > > > >> > >>>>>>>> executing > > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > > >> > > > > > > > org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd > > > > > > > > > > > >> > >>>>> for > > > > > > > > > > > >> > >>>>>>>>> job-5081 > > > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:53,267 INFO > > > > > > > > > > [o.a.c.f.j.i.AsyncJobMonitor] > > > > > > > > > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 > > job-5081) > > > > > > > > > > (logid:16897ea6) > > > > > > > > > > > >> > >>> Remove > > > > > > > > > > > >> > >>>>>>>> job-5081 > > > > > > > > > > > >> > >>>>>>>>> from job monitoring > > > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:56,230 DEBUG > > > > [c.c.a.ApiServlet] > > > > > > > > > > > >> > >>>>>>>>> (qtp510113906-28:ctx-e6c5bc85) > > > > (logid:7fe68f75) > > > > > > > > > > ===START=== > > > > > > > > > > > >> > >>>>>>>> 192.168.5.140 > > > > > > > > > > > >> > >>>>>>>>> -- GET > > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > > >> > >>>>>>>> > > > > > > > > > > > >> > >>>>> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > command=queryAsyncJobResult&jobId=16897ea6-27c3-45b9-a0df-ab217bb5393c > > > > > > > > > > > >> > >>>>>>>> &response=json&_=1552998476202 > > > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:56,238 DEBUG > > > > [c.c.a.ApiServer] > > > > > > > > > > > >> > >>>>>>>>> (qtp510113906-28:ctx-e6c5bc85 > > ctx-da1f4cbd) > > > > > > > > > > (logid:7fe68f75) > > > > > > > > > > > >> > >>> CIDRs > > > > > > > > > > > >> > >>>>>>>>> from which account > > > > > > > > > > > >> > >>>>>>>>> > > > > > 'Acct[15863393-8e8d-11e7-8f52-f04da2002bbe-admin]' > > > > > > > is > > > > > > > > > > > >> > >>>>>>>> allowed > > > > > > > > > > > >> > >>>>>>>>> to perform API calls: > 0.0.0.0/0,::/0 > > > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:27:56,260 DEBUG > > > > [c.c.a.ApiServlet] > > > > > > > > > > > >> > >>>>>>>>> (qtp510113906-28:ctx-e6c5bc85 > > ctx-da1f4cbd) > > > > > > > > > > (logid:7fe68f75) > > > > > > > > > > > >> > >>>>>>>>> ===END=== > > > > > > > > > > > >> > >>>>>>>>> 192.168.5.140 -- GET > > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > > >> > >>>>>>>> > > > > > > > > > > > >> > >>>>> > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > command=queryAsyncJobResult&jobId=16897ea6-27c3-45b9-a0df-ab217bb5393c > > > > > > > > > > > >> > >>>>>>>> &response=json&_=1552998476202 > > > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:28:00,946 INFO > > > > > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl] > > > > > > > > > > > >> > >>>>>>>>> > (AsyncJobMgr-Heartbeat-1:ctx-9b43d1fd) > > > > > > > > (logid:a605267a) > > > > > > > > > > > Begin > > > > > > > > > > > >> > >>>>>>>>> cleanup expired async-jobs > > > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:28:00,951 INFO > > > > > > > > > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl] > > > > > > > > > > > >> > >>>>>>>>> > (AsyncJobMgr-Heartbeat-1:ctx-9b43d1fd) > > > > > > > > (logid:a605267a) > > > > > > > > > > End > > > > > > > > > > > >> > >>>>> cleanup > > > > > > > > > > > >> > >>>>>>>> expired > > > > > > > > > > > >> > >>>>>>>>> async-jobs > > > > > > > > > > > >> > >>>>>>>>> 2019-03-19 12:28:01,142 DEBUG > > > > > > > > > > > >> > >>>>>>>> > > [c.c.n.r.VirtualNetworkApplianceManagerImpl] > > > > > > > > > > > >> > >>>>>>>>> > (RouterStatusMonitor-1:ctx-ad6bbe7e) > > > > > > > (logid:04e4c72b) > > > > > > > > > > Found > > > > > > > > > > > 0 > > > > > > > > > > > >> > >>>>>>>>> routers to update status. > > > > > > > > > > > >> > >>>>>>>>> > > > > > > > > > > > >> > >>>>>>>> > > > > > > > > > > > >> > >>>>>>>> > > > > > > > > > > > >> > >>>>>>>> -- > > > > > > > > > > > >> > >>>>>>>> Rafael Weingärtner > > > > > > > > > > > >> > >>>>>>>> > > > > > > > > > > > >> > >>>>>>> > > > > > > > > > > > >> > >>>>>> > > > > > > > > > > > >> > >>>>> > > > > > > > > > > > >> > >>>> > > > > > > > > > > > >> > >>> > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Andrija Panić > > > > > > > > > > > >
