>>> 1. Confirm please rolling restart is set to false please - double check Double checked - It is set to false
>>>>2. If so - do you know the name of VR which you deleted ? Is it last one >>> ever created - if so we can find it easily... I dont know the name Is there a way to fetch it from DB? On Tue, Mar 19, 2019 at 7:58 PM Andrija Panic <andrija.pa...@gmail.com> wrote: > Right...still complaining on missing running routers. > > 1. Confirm please rolling restart is set to false please - double check > 2. If so - do you know the name of VR which you deleted ? Is it last one > ever created - if so we can find it easily... > > > > On Tue, Mar 19, 2019, 18:40 Jevgeni Zolotarjov <j.zolotar...@gmail.com> > wrote: > > > Here is management server log > > https://drive.google.com/open?id=1H2jI0roeiWxtzReB8qV6QxDkNpaki99A > > > > > > On Tue, Mar 19, 2019 at 7:29 PM Andrija Panic <andrija.pa...@gmail.com> > > wrote: > > > > > I would also try to "undelete" VR in DB, but let's keep this as last > > step. > > > > > > On Tue, Mar 19, 2019, 18:24 Andrija Panic <andrija.pa...@gmail.com> > > wrote: > > > > > > > Disclaimer: from mobile device. > > > > > > > > I don't see any special failure in agent log, except some long > running > > > > migration, timeout for graceful VM shutdown etc (and agent restart) > > > > > > > > You did not send mgmt log after last network restart failure ? > > > > > > > > On Tue, Mar 19, 2019, 17:29 Jevgeni Zolotarjov < > j.zolotar...@gmail.com > > > > > > > wrote: > > > > > > > >> Under > > > >> Infrastructure / Hosts: > > > >> both hosts are enabled and green (Unsecure though) > > > >> > > > >> agent.log - > > > >> https://drive.google.com/open?id=1rATxHKqgNKo2kD23BtlrZy_9gFXC-Bq- > > > >> > > > >> I set network.rolling.restart to false now. Restarted management > > server > > > - > > > >> same problem - cannot restart not delete network > > > >> > > > >> > > > >> > > > >> On Tue, Mar 19, 2019 at 5:48 PM Boris Stoyanov < > > > >> boris.stoya...@shapeblue.com> > > > >> wrote: > > > >> > > > >> > Hi, you shouldn’t be worried about data as long as you use a > > separate > > > >> > shared storage. > > > >> > > > > >> > What does the cloudstack console say about your host? Is it up? If > > > it’s > > > >> up > > > >> > then you should be able to deploy a VM on it. > > > >> > > > > >> > If not, you’ll need to investigate the reason cloudstack-agent is > > not > > > >> > connected, is the agent running? If not just start it and see if > it > > > >> > connects. > > > >> > > > > >> > If it’s running but still not connected please share the > agent.log. > > > >> > > > > >> > If nothing helps at all you can remove the host and add it again, > > but > > > >> > let’s try to troubleshoot first. > > > >> > > > > >> > Bobby. > > > >> > > > > >> > > > > >> > boris.stoya...@shapeblue.com > > > >> > www.shapeblue.com > > > >> > Amadeus House, Floral Street, London WC2E 9DPUK > > > >> > @shapeblue > > > >> > > > > >> > > > > >> > > > > >> > > On 19 Mar 2019, at 17:29, Jevgeni Zolotarjov < > > > j.zolotar...@gmail.com> > > > >> > wrote: > > > >> > > > > > >> > > Guys, please help with it. > > > >> > > What can be done here? > > > >> > > There is too much valuable data. > > > >> > > > > > >> > > On Tue, Mar 19, 2019 at 4:21 PM Jevgeni Zolotarjov < > > > >> > j.zolotar...@gmail.com> > > > >> > > wrote: > > > >> > > > > > >> > >> Tried that just now and got error: > > > >> > >> Resource [DataCenter:1] is unreachable: Can't find all > necessary > > > >> running > > > >> > >> routers! > > > >> > >> > > > >> > >> In the log I see: > > > >> > >> ============================================= > > > >> > >> > > > >> > >> 2019-03-19 14:20:39,644 DEBUG [o.a.c.e.o.NetworkOrchestrator] > > > >> > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093 ctx-9be30648) > > > >> (logid:265a6099) > > > >> > >> Restarting network 204... > > > >> > >> 2019-03-19 14:20:39,645 DEBUG [o.a.c.e.o.NetworkOrchestrator] > > > >> > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093 ctx-9be30648) > > > >> (logid:265a6099) > > > >> > >> Performing rolling restart of routers of network > > Ntwk[204|Guest|6] > > > >> > >> 2019-03-19 14:20:39,649 DEBUG [o.a.c.e.o.NetworkOrchestrator] > > > >> > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093 ctx-9be30648) > > > >> (logid:265a6099) > > > >> > >> Asking VirtualRouter to implemenet Ntwk[204|Guest|6] > > > >> > >> 2019-03-19 14:20:39,658 ERROR [c.c.a.ApiAsyncJobDispatcher] > > > >> > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093) (logid:265a6099) > > > >> Unexpected > > > >> > >> exception while executing > > > >> > >> > org.apache.cloudstack.api.command.user.network.RestartNetworkCmd > > > >> > >> com.cloud.exception.ResourceUnavailableException: Resource > > > >> > [DataCenter:1] > > > >> > >> is unreachable: Can't find all necessary running routers! > > > >> > >> at > > > >> > >> > > > >> > > > > >> > > > > > > com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:243) > > > >> > >> at > > > >> > >> > > > >> > > > > >> > > > > > > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElements(NetworkOrchestrator.java:1203) > > > >> > >> at > > > >> > >> > > > >> > > > > >> > > > > > > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.rollingRestartRouters(NetworkOrchestrator.java:2948) > > > >> > >> at > > > >> > >> > > > >> > > > > >> > > > > > > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:2853) > > > >> > >> at > > > >> > >> > > > >> > > > > >> > > > > > > com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1883) > > > >> > >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > > > Method) > > > >> > >> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown > > > Source) > > > >> > >> at > sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown > > > >> > Source) > > > >> > >> at java.lang.reflect.Method.invoke(Unknown Source) > > > >> > >> at > > > >> > >> > > > >> > > > > >> > > > > > > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338) > > > >> > >> at > > > >> > >> > > > >> > > > > >> > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197) > > > >> > >> at > > > >> > >> > > > >> > > > > >> > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) > > > >> > >> at > > > >> > >> > > > >> > > > > >> > > > > > > org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107) > > > >> > >> at > > > >> > >> > > > >> > > > > >> > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) > > > >> > >> at > > > >> > >> > > > >> > > > > >> > > > > > > com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51) > > > >> > >> at > > > >> > >> > > > >> > > > > >> > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) > > > >> > >> at > > > >> > >> > > > >> > > > > >> > > > > > > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) > > > >> > >> at > > > >> > >> > > > >> > > > > >> > > > > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) > > > >> > >> at > > > >> > >> > > > >> > > > > >> > > > > > > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) > > > >> > >> at com.sun.proxy.$Proxy229.restartNetwork(Unknown > Source) > > > >> > >> at > > > >> > >> > > > >> > > > > >> > > > > > > org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:99) > > > >> > >> at > > > >> com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150) > > > >> > >> at > > > >> > >> > > > >> > > > > >> > > > > > > com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108) > > > >> > >> at > > > >> > >> > > > >> > > > > >> > > > > > > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581) > > > >> > >> at > > > >> > >> > > > >> > > > > >> > > > > > > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > > > >> > >> at > > > >> > >> > > > >> > > > > >> > > > > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > > > >> > >> at > > > >> > >> > > > >> > > > > >> > > > > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > > > >> > >> at > > > >> > >> > > > >> > > > > >> > > > > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > > > >> > >> at > > > >> > >> > > > >> > > > > >> > > > > > > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > > > >> > >> at > > > >> > >> > > > >> > > > > >> > > > > > > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529) > > > >> > >> at > > > java.util.concurrent.Executors$RunnableAdapter.call(Unknown > > > >> > >> Source) > > > >> > >> at java.util.concurrent.FutureTask.run(Unknown Source) > > > >> > >> at > > java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown > > > >> > >> Source) > > > >> > >> at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown > > > >> > >> Source) > > > >> > >> at java.lang.Thread.run(Unknown Source) > > > >> > >> 2019-03-19 14:20:39,659 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > > > >> > >> (API-Job-Executor-4:ctx-7b6b69eb job-5093) (logid:265a6099) > > > Complete > > > >> > async > > > >> > >> job-5093, jobStatus: FAILED, resultCode: 530, result: > > > >> > >> > > > >> > > > > >> > > > > > > org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Resource > > > >> > >> [DataCenter:1] is un > > > >> > >> > > > >> > >> On Tue, Mar 19, 2019 at 4:19 PM Andrija Panic < > > > >> > andrija.pa...@shapeblue.com> > > > >> > >> wrote: > > > >> > >> > > > >> > >>> > > > >> > >>> Your network can't be deleted due to "Can't delete the > network, > > > not > > > >> all > > > >> > >>> user vms are expunged. Vm > > > >> > >>> VM[User|i-2-11-VM] is in Stopped state" - which is fine. > > > >> > >>> > > > >> > >>> You should be able to just start the user VM - but if you have > > > >> actually > > > >> > >>> delete the VR itself, then just do Network restart with > > "cleanup" > > > >> and > > > >> > it > > > >> > >>> will recreate a new VR, after which you should be able to > start > > > the > > > >> VM. > > > >> > >>> > > > >> > >>> Andrija > > > >> > >>> > > > >> > >>> andrija.pa...@shapeblue.com > > > >> > >>> www.shapeblue.com > > > >> > >>> Amadeus House, Floral Street, London WC2E 9DPUK > > > >> > >>> @shapeblue > > > >> > >>> > > > >> > >>> > > > >> > >>> > > > >> > >>> > > > >> > >>> -----Original Message----- > > > >> > >>> From: Jevgeni Zolotarjov <j.zolotar...@gmail.com> > > > >> > >>> Sent: 19 March 2019 15:10 > > > >> > >>> To: users@cloudstack.apache.org > > > >> > >>> Subject: Re: Disaster after maintenance > > > >> > >>> > > > >> > >>> I mean I cannot delete network: In the management server log I > > see > > > >> > >>> > > > >> > >>> ========================================================== > > > >> > >>> 019-03-19 14:06:36,316 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > > > >> > >>> (API-Job-Executor-1:ctx-1c0fd4dc job-5090) (logid:c734edfc) > > > >> Executing > > > >> > >>> AsyncJobVO {id:5090, userId: 2, accountId: 2, instanceType: > > > Network, > > > >> > >>> instanceId: 204, cmd: > > > >> > >>> > org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd, > > > >> > cmdInfo: > > > >> > >>> > > > >> > >>> > > > >> > > > > >> > > > > > > {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"2641","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\"interface > > > >> > >>> > > > >> > >>> > > > >> > > > > >> > > > > > > com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}","ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmdEventType":"NETWORK.DELETE","_":"1553004396247"}, > > > >> > >>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, > > resultCode: > > > 0, > > > >> > >>> result: null, initMsid: 264216221068220, completeMsid: null, > > > >> > lastUpdated: > > > >> > >>> null, lastPolled: null, created: null} > > > >> > >>> 2019-03-19 14:06:36,351 WARN [o.a.c.e.o.NetworkOrchestrator] > > > >> > >>> (API-Job-Executor-1:ctx-1c0fd4dc job-5090 ctx-134954fa) > > > >> > (logid:c734edfc) > > > >> > >>> Can't delete the network, not all user vms are expunged. Vm > > > >> > >>> VM[User|i-2-11-VM] is in Stopped state > > > >> > >>> 2019-03-19 14:06:36,356 DEBUG > [o.a.c.f.j.i.AsyncJobManagerImpl] > > > >> > >>> (API-Job-Executor-1:ctx-1c0fd4dc job-5090) (logid:c734edfc) > > > Complete > > > >> > async > > > >> > >>> job-5090, jobStatus: FAILED, resultCode: 530, result: > > > >> > >>> > > > >> > >>> > > > >> > > > > >> > > > > > > org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed > > > >> > >>> to delete network"} > > > >> > >>> ========================================================== > > > >> > >>> > > > >> > >>> > > > >> > >>> I deleted a router, expecting it to be recreated on deleting > > > >> network. > > > >> > But > > > >> > >>> I am unable to delete network because of above error > > > >> > >>> > > > >> > >>> On Tue, Mar 19, 2019 at 3:58 PM Jevgeni Zolotarjov < > > > >> > >>> j.zolotar...@gmail.com> > > > >> > >>> wrote: > > > >> > >>> > > > >> > >>>> I've managed to make libvirtd running > > > >> > >>>> Now cloudstack console shows both hosts - running > > > >> > >>>> > > > >> > >>>> But now as I have removed network, VMs are unable to start. > > > >> > >>>> > > > >> > >>>> How can I recreate the network now? > > > >> > >>>> > > > >> > >>>> On Tue, Mar 19, 2019 at 3:14 PM Ivan Kudryavtsev > > > >> > >>>> <kudryavtsev...@bw-sw.com> > > > >> > >>>> wrote: > > > >> > >>>> > > > >> > >>>>> Jevgeniy, it may be a documentation bug. Take s look: > > > >> > >>>>> > > > https://github.com/apache/cloudstack-documentation/pull/27/files > > > >> > >>>>> > > > >> > >>>>> вт, 19 мар. 2019 г., 9:09 Jevgeni Zolotarjov < > > > >> j.zolotar...@gmail.com > > > >> > >: > > > >> > >>>>> > > > >> > >>>>>> That's it - libvirtd failed to start on second host. > > > >> > >>>>>> Tried restarting, but it does not start. > > > >> > >>>>>> > > > >> > >>>>>> > > > >> > >>>>>>>> Do you have some NUMA constraints or anything which > > requires > > > >> > >>>>> particular > > > >> > >>>>>> RAM configuration? > > > >> > >>>>>> No > > > >> > >>>>>> > > > >> > >>>>>> libvirtd.service - Virtualization daemon > > > >> > >>>>>> Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; > > > >> > >>>>>> enabled; vendor preset: enabled) > > > >> > >>>>>> Active: failed (Result: start-limit) since Tue 2019-03-19 > > > >> > >>>>>> 13:03:07 > > > >> > >>>>> GMT; > > > >> > >>>>>> 12s ago > > > >> > >>>>>> Docs: man:libvirtd(8) > > > >> > >>>>>> https://libvirt.org > > > >> > >>>>>> Process: 892 ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS > > > >> > >>>>>> (code=exited, > > > >> > >>>>>> status=1/FAILURE) > > > >> > >>>>>> Main PID: 892 (code=exited, status=1/FAILURE) > > > >> > >>>>>> Tasks: 19 (limit: 32768) > > > >> > >>>>>> CGroup: /system.slice/libvirtd.service > > > >> > >>>>>> ├─11338 /usr/sbin/libvirtd -d -l > > > >> > >>>>>> ├─11909 /usr/sbin/dnsmasq > > > >> > >>>>>> --conf-file=/var/lib/libvirt/dnsmasq/default.conf > > > --leasefile-ro > > > >> > >>>>>> --dhcp-script=/usr/libexec/libvirt_leaseshelper > > > >> > >>>>>> └─11910 /usr/sbin/dnsmasq > > > >> > >>>>>> --conf-file=/var/lib/libvirt/dnsmasq/default.conf > > > --leasefile-ro > > > >> > >>>>>> --dhcp-script=/usr/libexec/libvirt_leaseshelper > > > >> > >>>>>> > > > >> > >>>>>> Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: > > Failed > > > >> to > > > >> > >>>>>> start Virtualization daemon. > > > >> > >>>>>> Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: > Unit > > > >> > >>>>>> libvirtd.service entered failed state. > > > >> > >>>>>> Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: > > > >> > >>>>> libvirtd.service > > > >> > >>>>>> failed. > > > >> > >>>>>> Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: > > > >> > >>>>> libvirtd.service > > > >> > >>>>>> holdoff time over, scheduling restart. > > > >> > >>>>>> Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: > > > Stopped > > > >> > >>>>>> Virtualization daemon. > > > >> > >>>>>> Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: > > start > > > >> > >>>>>> request repeated too quickly for libvirtd.service Mar 19 > > > 13:03:07 > > > >> > >>>>>> mtl1-apphst04.mt.pbt.com.mt systemd[1]: Failed to start > > > >> > >>>>>> Virtualization daemon. > > > >> > >>>>>> Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: > Unit > > > >> > >>>>>> libvirtd.service entered failed state. > > > >> > >>>>>> Mar 19 13:03:07 mtl1-apphst04.mt.pbt.com.mt systemd[1]: > > > >> > >>>>> libvirtd.service > > > >> > >>>>>> failed. > > > >> > >>>>>> > > > >> > >>>>>> > > > >> > >>>>>> On Tue, Mar 19, 2019 at 3:04 PM Paul Angus > > > >> > >>>>>> <paul.an...@shapeblue.com> > > > >> > >>>>>> wrote: > > > >> > >>>>>> > > > >> > >>>>>>> Can you check that the cloudstack agent is running on the > > host > > > >> > >>>>>>> and the agent logs (usual logs directory) Also worth > > checking > > > >> > >>>>>>> that libvirt has started ok. Do you have some > > > >> > >>>>> NUMA > > > >> > >>>>>>> constraints or anything which requires particular RAM > > > >> > >>> configuration? > > > >> > >>>>>>> > > > >> > >>>>>>> paul.an...@shapeblue.com > > > >> > >>>>>>> www.shapeblue.com > > > >> > >>>>>>> Amadeus House, Floral Street, London WC2E 9DPUK > @shapeblue > > > >> > >>>>>>> > > > >> > >>>>>>> > > > >> > >>>>>>> > > > >> > >>>>>>> > > > >> > >>>>>>> -----Original Message----- > > > >> > >>>>>>> From: Jevgeni Zolotarjov <j.zolotar...@gmail.com> > > > >> > >>>>>>> Sent: 19 March 2019 14:49 > > > >> > >>>>>>> To: users@cloudstack.apache.org > > > >> > >>>>>>> Subject: Re: Disaster after maintenance > > > >> > >>>>>>> > > > >> > >>>>>>> Can you try migrating a VM to the server that you changed > > the > > > >> RAM > > > >> > >>>>> amount? > > > >> > >>>>>>> > > > >> > >>>>>>> Also: > > > >> > >>>>>>> What is the hypervisor version? > > > >> > >>>>>>> KVM > > > >> > >>>>>>> QEMU Version : 2.0.0 > > > >> > >>>>>>> Release : 1.el7.6 > > > >> > >>>>>>> > > > >> > >>>>>>> > > > >> > >>>>>>> Host status in ACS? > > > >> > >>>>>>> 1st server: Unsecure > > > >> > >>>>>>> 2nd server: Disconnected > > > >> > >>>>>>> > > > >> > >>>>>>> Did you try to force a VM to start/deploy in this server > > where > > > >> you > > > >> > >>>>>> changed > > > >> > >>>>>>> the RAM? > > > >> > >>>>>>> Host status became disconnected. I don't know how to make > it > > > >> > >>>>> "connected" > > > >> > >>>>>>> again > > > >> > >>>>>>> > > > >> > >>>>>>> > > > >> > >>>>>>> > > > >> > >>>>>>> On Tue, Mar 19, 2019 at 2:42 PM Rafael Weingärtner < > > > >> > >>>>>>> rafaelweingart...@gmail.com> wrote: > > > >> > >>>>>>> > > > >> > >>>>>>>> Can you try migrating a VM to the server that you changed > > the > > > >> RAM > > > >> > >>>>>> amount? > > > >> > >>>>>>>> > > > >> > >>>>>>>> Also: > > > >> > >>>>>>>> What is the hypervisor version? > > > >> > >>>>>>>> Host status in ACS? > > > >> > >>>>>>>> Did you try to force a VM to start/deploy in this server > > > where > > > >> > >>> you > > > >> > >>>>>>>> changed the RAM? > > > >> > >>>>>>>> > > > >> > >>>>>>>> > > > >> > >>>>>>>> On Tue, Mar 19, 2019 at 9:39 AM Jevgeni Zolotarjov > > > >> > >>>>>>>> <j.zolotar...@gmail.com > > > >> > >>>>>>>>> > > > >> > >>>>>>>> wrote: > > > >> > >>>>>>>> > > > >> > >>>>>>>>> We have Cloudstack 4.11.2 setup running fine for few > > months > > > >> > >>> (>4) > > > >> > >>>>> The > > > >> > >>>>>>>>> setup is very simple: 2 hosts We decided to do a > > maintenance > > > >> to > > > >> > >>>>>>>>> increase RAM on both servers > > > >> > >>>>>>>>> > > > >> > >>>>>>>>> For this we put first server to maintenance. All VMS > moved > > > to > > > >> > >>>>> second > > > >> > >>>>>>>>> host after a while. > > > >> > >>>>>>>>> > > > >> > >>>>>>>>> Then first server was shutdown, RAM increased, server > > turned > > > >> > >>> ON. > > > >> > >>>>>>>>> Now nothing starts on first server. > > > >> > >>>>>>>>> > > > >> > >>>>>>>>> > > > >> > >>>>>>>>> Tried to delete network, but this fails as well > > > >> > >>>>>>>>> > > > >> > >>>>>>>>> Please help ! > > > >> > >>>>>>>>> > > > >> > >>>>>>>>> Here is extract from log: > > > >> > >>>>>>>>> ====================================== > > > >> > >>>>>>>>> 2019-03-19 12:27:53,064 DEBUG > > > >> > >>>>> [o.a.c.s.SecondaryStorageManagerImpl] > > > >> > >>>>>>>>> (secstorage-1:ctx-16d6c797) (logid:7e3160ce) Zone 1 is > > ready > > > >> to > > > >> > >>>>>>>>> launch secondary storage VM > > > >> > >>>>>>>>> 2019-03-19 12:27:53,125 DEBUG > > > [c.c.c.ConsoleProxyManagerImpl] > > > >> > >>>>>>>>> (consoleproxy-1:ctx-cbd034b9) (logid:0a8c8bf4) Zone 1 is > > > ready > > > >> > >>> to > > > >> > >>>>>>>>> launch console proxy > > > >> > >>>>>>>>> 2019-03-19 12:27:53,181 DEBUG [c.c.a.ApiServlet] > > > >> > >>>>>>>>> (qtp510113906-285:ctx-6c5e11c3) (logid:cd8e30be) > > ===START=== > > > >> > >>>>>>>> 192.168.5.140 > > > >> > >>>>>>>>> -- GET > > > >> > >>>>>>>>> > > > >> > >>>>>>>>> > > > >> > >>>>>>>> > > > >> > >>>>> > > > >> > > > command=deleteNetwork&id=4ba834ed-48f3-468f-b667-9bb2d2c258f1&response > > > >> > >>>>>>>> =json&_=1552998473154 > > > >> > >>>>>>>>> 2019-03-19 12:27:53,186 DEBUG [c.c.a.ApiServer] > > > >> > >>>>>>>>> (qtp510113906-285:ctx-6c5e11c3 ctx-0cc34dc6) > > > (logid:cd8e30be) > > > >> > >>>>> CIDRs > > > >> > >>>>>>>>> from which account > > > >> > >>>>>>>>> 'Acct[15863393-8e8d-11e7-8f52-f04da2002bbe-admin]' is > > > >> > >>>>>>>> allowed > > > >> > >>>>>>>>> to perform API calls: 0.0.0.0/0,::/0 > > > >> > >>>>>>>>> 2019-03-19 12:27:53,208 INFO > > [o.a.c.f.j.i.AsyncJobMonitor] > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081) > > (logid:f6751fa7) > > > >> Add > > > >> > >>>>>>>>> job-5081 into job monitoring > > > >> > >>>>>>>>> 2019-03-19 12:27:53,209 DEBUG > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl] > > > >> > >>>>>>>>> (qtp510113906-285:ctx-6c5e11c3 ctx-0cc34dc6) > > > (logid:cd8e30be) > > > >> > >>>>> submit > > > >> > >>>>>>>> async > > > >> > >>>>>>>>> job-5081, details: AsyncJobVO {id:5081, userId: 2, > > > accountId: > > > >> > >>> 2, > > > >> > >>>>>>>>> instanceType: Network, instanceId: 204, cmd: > > > >> > >>>>>>>>> > > > >> > >>> > org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd, > > > >> > >>>>>>> cmdInfo: > > > >> > >>>>>>>>> > > > >> > >>>>>>>>> > > > >> > >>>>>>>> > > > >> > >>>>> > > > >> > > > {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId > > > >> > >>>>>>>> > > > >> > >>>>> > > > >> > > > ":"2615","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\" > > > >> > >>>>>>>> interface > > > >> > >>>>>>>>> > > > >> > >>>>>>>>> > > > >> > >>>>>>>> > > > >> > >>>>> > > > >> > > > com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}" > > > >> > >>>>>>>> > > > >> > >>>>> > > > >> > > > ,"ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmd > > > >> > >>>>>>>> EventType":"NETWORK.DELETE","_":"1552998473154"}, > > > >> > >>>>>>>>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, > > > >> > >>> resultCode: > > > >> > >>>>> 0, > > > >> > >>>>>>>>> result: null, initMsid: 264216221068220, completeMsid: > > null, > > > >> > >>>>>>> lastUpdated: > > > >> > >>>>>>>>> null, lastPolled: null, created: null} > > > >> > >>>>>>>>> 2019-03-19 12:27:53,211 DEBUG [c.c.a.ApiServlet] > > > >> > >>>>>>>>> (qtp510113906-285:ctx-6c5e11c3 ctx-0cc34dc6) > > > (logid:cd8e30be) > > > >> > >>>>>>>>> ===END=== > > > >> > >>>>>>>>> 192.168.5.140 -- GET > > > >> > >>>>>>>>> > > > >> > >>>>>>>>> > > > >> > >>>>>>>> > > > >> > >>>>> > > > >> > > > command=deleteNetwork&id=4ba834ed-48f3-468f-b667-9bb2d2c258f1&response > > > >> > >>>>>>>> =json&_=1552998473154 > > > >> > >>>>>>>>> 2019-03-19 12:27:53,212 DEBUG > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl] > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081) > > (logid:16897ea6) > > > >> > >>>>>>>>> Executing AsyncJobVO {id:5081, userId: 2, accountId: 2, > > > >> > >>>>>>>>> instanceType: Network, > > > >> > >>>>>>>>> instanceId: 204, cmd: > > > >> > >>>>>>>>> > > > >> > >>> > org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd, > > > >> > >>>>>>> cmdInfo: > > > >> > >>>>>>>>> > > > >> > >>>>>>>>> > > > >> > >>>>>>>> > > > >> > >>>>> > > > >> > > > {"response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId > > > >> > >>>>>>>> > > > >> > >>>>> > > > >> > > > ":"2615","id":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","ctxDetails":"{\" > > > >> > >>>>>>>> interface > > > >> > >>>>>>>>> > > > >> > >>>>>>>>> > > > >> > >>>>>>>> > > > >> > >>>>> > > > >> > > > com.cloud.network.Network\":\"4ba834ed-48f3-468f-b667-9bb2d2c258f1\"}" > > > >> > >>>>>>>> > > > >> > >>>>> > > > >> > > > ,"ctxAccountId":"2","uuid":"4ba834ed-48f3-468f-b667-9bb2d2c258f1","cmd > > > >> > >>>>>>>> EventType":"NETWORK.DELETE","_":"1552998473154"}, > > > >> > >>>>>>>>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, > > > >> > >>> resultCode: > > > >> > >>>>> 0, > > > >> > >>>>>>>>> result: null, initMsid: 264216221068220, completeMsid: > > null, > > > >> > >>>>>>> lastUpdated: > > > >> > >>>>>>>>> null, lastPolled: null, created: null} > > > >> > >>>>>>>>> 2019-03-19 12:27:53,257 WARN > > > [o.a.c.e.o.NetworkOrchestrator] > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081 ctx-d5de7979) > > > >> > >>>>>>>>> (logid:16897ea6) Can't delete the network, not all user > > vms > > > >> are > > > >> > >>>>>>>>> expunged. Vm VM[User|i-2-11-VM] is in Stopped state > > > >> > >>>>>>>>> 2019-03-19 12:27:53,263 DEBUG > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl] > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081) > > (logid:16897ea6) > > > >> > >>>>> Complete > > > >> > >>>>>>>> async > > > >> > >>>>>>>>> job-5081, jobStatus: FAILED, resultCode: 530, result: > > > >> > >>>>>>>>> > > > >> > >>>>>>>>> > > > >> > >>>>>>>> > > > >> > >>>>> > > > >> > > > org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList": > > > >> > >>>>>>>> [],"errorcode":530,"errortext":"Failed > > > >> > >>>>>>>>> to delete network"} > > > >> > >>>>>>>>> 2019-03-19 12:27:53,264 DEBUG > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl] > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081) > > (logid:16897ea6) > > > >> > >>>>> Publish > > > >> > >>>>>>>>> async > > > >> > >>>>>>>>> job-5081 complete on message bus > > > >> > >>>>>>>>> 2019-03-19 12:27:53,264 DEBUG > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl] > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081) > > (logid:16897ea6) > > > >> > >>> Wake > > > >> > >>>>> up > > > >> > >>>>>>>>> jobs related to job-5081 > > > >> > >>>>>>>>> 2019-03-19 12:27:53,264 DEBUG > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl] > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081) > > (logid:16897ea6) > > > >> > >>> Update > > > >> > >>>>>>>>> db status for job-5081 > > > >> > >>>>>>>>> 2019-03-19 12:27:53,265 DEBUG > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl] > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081) > > (logid:16897ea6) > > > >> > >>> Wake > > > >> > >>>>> up > > > >> > >>>>>>>>> jobs joined with job-5081 and disjoin all subjobs > created > > > from > > > >> > >>>>> job- > > > >> > >>>>>>>>> 5081 > > > >> > >>>>>>>>> 2019-03-19 12:27:53,267 DEBUG > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl] > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081) > > (logid:16897ea6) > > > >> > >>> Done > > > >> > >>>>>>>> executing > > > >> > >>>>>>>>> > > > >> org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd > > > >> > >>>>> for > > > >> > >>>>>>>>> job-5081 > > > >> > >>>>>>>>> 2019-03-19 12:27:53,267 INFO > > [o.a.c.f.j.i.AsyncJobMonitor] > > > >> > >>>>>>>>> (API-Job-Executor-1:ctx-d4970c19 job-5081) > > (logid:16897ea6) > > > >> > >>> Remove > > > >> > >>>>>>>> job-5081 > > > >> > >>>>>>>>> from job monitoring > > > >> > >>>>>>>>> 2019-03-19 12:27:56,230 DEBUG [c.c.a.ApiServlet] > > > >> > >>>>>>>>> (qtp510113906-28:ctx-e6c5bc85) (logid:7fe68f75) > > ===START=== > > > >> > >>>>>>>> 192.168.5.140 > > > >> > >>>>>>>>> -- GET > > > >> > >>>>>>>>> > > > >> > >>>>>>>>> > > > >> > >>>>>>>> > > > >> > >>>>> > > > >> > > > command=queryAsyncJobResult&jobId=16897ea6-27c3-45b9-a0df-ab217bb5393c > > > >> > >>>>>>>> &response=json&_=1552998476202 > > > >> > >>>>>>>>> 2019-03-19 12:27:56,238 DEBUG [c.c.a.ApiServer] > > > >> > >>>>>>>>> (qtp510113906-28:ctx-e6c5bc85 ctx-da1f4cbd) > > (logid:7fe68f75) > > > >> > >>> CIDRs > > > >> > >>>>>>>>> from which account > > > >> > >>>>>>>>> 'Acct[15863393-8e8d-11e7-8f52-f04da2002bbe-admin]' is > > > >> > >>>>>>>> allowed > > > >> > >>>>>>>>> to perform API calls: 0.0.0.0/0,::/0 > > > >> > >>>>>>>>> 2019-03-19 12:27:56,260 DEBUG [c.c.a.ApiServlet] > > > >> > >>>>>>>>> (qtp510113906-28:ctx-e6c5bc85 ctx-da1f4cbd) > > (logid:7fe68f75) > > > >> > >>>>>>>>> ===END=== > > > >> > >>>>>>>>> 192.168.5.140 -- GET > > > >> > >>>>>>>>> > > > >> > >>>>>>>>> > > > >> > >>>>>>>> > > > >> > >>>>> > > > >> > > > command=queryAsyncJobResult&jobId=16897ea6-27c3-45b9-a0df-ab217bb5393c > > > >> > >>>>>>>> &response=json&_=1552998476202 > > > >> > >>>>>>>>> 2019-03-19 12:28:00,946 INFO > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl] > > > >> > >>>>>>>>> (AsyncJobMgr-Heartbeat-1:ctx-9b43d1fd) (logid:a605267a) > > > Begin > > > >> > >>>>>>>>> cleanup expired async-jobs > > > >> > >>>>>>>>> 2019-03-19 12:28:00,951 INFO > > > >> [o.a.c.f.j.i.AsyncJobManagerImpl] > > > >> > >>>>>>>>> (AsyncJobMgr-Heartbeat-1:ctx-9b43d1fd) (logid:a605267a) > > End > > > >> > >>>>> cleanup > > > >> > >>>>>>>> expired > > > >> > >>>>>>>>> async-jobs > > > >> > >>>>>>>>> 2019-03-19 12:28:01,142 DEBUG > > > >> > >>>>>>>> [c.c.n.r.VirtualNetworkApplianceManagerImpl] > > > >> > >>>>>>>>> (RouterStatusMonitor-1:ctx-ad6bbe7e) (logid:04e4c72b) > > Found > > > 0 > > > >> > >>>>>>>>> routers to update status. > > > >> > >>>>>>>>> > > > >> > >>>>>>>> > > > >> > >>>>>>>> > > > >> > >>>>>>>> -- > > > >> > >>>>>>>> Rafael Weingärtner > > > >> > >>>>>>>> > > > >> > >>>>>>> > > > >> > >>>>>> > > > >> > >>>>> > > > >> > >>>> > > > >> > >>> > > > >> > >> > > > >> > > > > >> > > > > >> > > > > > > > > > >
