Hi Natalia, Thanks for the update. We missed that context for root causing the issue.
-Suresh On Tue, May 22, 2018 at 4:38 PM, Natalia Costas Lago <nata...@cesga.es> wrote: > > Hi, > > We finally were able to recover the manager :/ > > The problem was due to a database modification made by hand a couple of > months ago in order to eliminate one of the zombie virtual machines > (machines that cannot be expunged because they are dependent on a network > that was deleted before). > > The state of one of the machines was modified and Cloudstack did not like > that change (setting a VM state as Destroyed), this eliminated the VM for > the web interface but we did not detect the problem as we did not restart > the management server. All backups since then were wrong and for some > reason the management server crashed last week. > > Now it is fully operational :) > > Thank you, > Kind regards, > > > El 21/05/2018 a las 12:19, Natalia Costas Lago escribió: > >> >> This is what I can see in localhost log: >> >> cat /var/log/cloudstack/management/localhost.2018-05-21.log >> >> [...] >> >> SEVERE: Exception sending context initialized event to listener instance >> of class org.apache.cloudstack.spring.module.web.CloudStackContextLoa >> derListener >> java.lang.NullPointerException >> at com.cloud.vm.UserVmManagerImpl.loadVmDetailsInMapForExternal >> DhcpIp(UserVmManagerImpl.java:2019) >> at com.cloud.vm.UserVmManagerImpl.start(UserVmManagerImpl.java: >> 1999) >> at org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLif >> eCycle$1.with(CloudStackExtendedLifeCycle.java:75) >> at org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLif >> eCycle.with(CloudStackExtendedLifeCycle.java:154) >> at org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLif >> eCycle.startBeans(CloudStackExtendedLifeCycle.java:72) >> at org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLif >> eCycleStart.run(CloudStackExtendedLifeCycleStart.java:46) >> at org.apache.cloudstack.spring.module.model.impl.DefaultModule >> DefinitionSet$1.with(DefaultModuleDefinitionSet.java:105) >> at org.apache.cloudstack.spring.module.model.impl.DefaultModule >> DefinitionSet.withModule(DefaultModuleDefinitionSet.java:245) >> at org.apache.cloudstack.spring.module.model.impl.DefaultModule >> DefinitionSet.withModule(DefaultModuleDefinitionSet.java:250) >> at org.apache.cloudstack.spring.module.model.impl.DefaultModule >> DefinitionSet.withModule(DefaultModuleDefinitionSet.java:250) >> at org.apache.cloudstack.spring.module.model.impl.DefaultModule >> DefinitionSet.withModule(DefaultModuleDefinitionSet.java:233) >> at org.apache.cloudstack.spring.module.model.impl.DefaultModule >> DefinitionSet.startContexts(DefaultModuleDefinitionSet.java:97) >> at org.apache.cloudstack.spring.module.model.impl.DefaultModule >> DefinitionSet.load(DefaultModuleDefinitionSet.java:80) >> at org.apache.cloudstack.spring.module.factory.ModuleBasedConte >> xtFactory.loadModules(ModuleBasedContextFactory.java:37) >> at org.apache.cloudstack.spring.module.factory.CloudStackSpring >> Context.init(CloudStackSpringContext.java:71) >> at org.apache.cloudstack.spring.module.factory.CloudStackSpring >> Context.<init>(CloudStackSpringContext.java:58) >> at org.apache.cloudstack.spring.module.factory.CloudStackSpring >> Context.<init>(CloudStackSpringContext.java:62) >> at org.apache.cloudstack.spring.module.web.CloudStackContextLoa >> derListener.contextInitialized(CloudStackContextLoaderListener.java:52) >> at org.apache.catalina.core.StandardContext.listenerStart(Stand >> ardContext.java:5118) >> at org.apache.catalina.core.StandardContext.startInternal(Stand >> ardContext.java:5634) >> at org.apache.catalina.util.LifecycleBase.start(LifecycleBase. >> java:145) >> at org.apache.catalina.core.ContainerBase.addChildInternal(Cont >> ainerBase.java:899) >> at org.apache.catalina.core.ContainerBase.addChild(ContainerBas >> e.java:875) >> at org.apache.catalina.core.StandardHost.addChild(StandardHost. >> java:652) >> at org.apache.catalina.startup.HostConfig.deployDirectory(HostC >> onfig.java:1260) >> at org.apache.catalina.startup.HostConfig$DeployDirectory.run( >> HostConfig.java:2002) >> at java.util.concurrent.Executors$RunnableAdapter.call( >> Executors.java:473) >> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool >> Executor.java:1152) >> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo >> lExecutor.java:622) >> at java.lang.Thread.run(Thread.java:748) >> >> >> >> El 21/05/2018 a las 9:21, Natalia Costas Lago escribió: >> >>> >>> It has been working with 1.8 for long... >>> >>> I switched to java 1.7, but java errors persist. >>> >>> We recovered backups from a couple of days prior to the disaster, but we >>> are still unlucky, it seems the problem is also in the backups copy. >>> >>> I installed a fresh cloudstack and copied a recent database and the >>> manager still gives and Exception but to a later point in the execution: >>> >>> >>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO >>> [o.a.c.s.l.CloudStackExtendedLifeCycle] (localhost-startStop-1:null) >>> (logid:) Configuring CloudStack Components >>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO >>> [o.a.c.s.l.CloudStackExtendedLifeCycle] (localhost-startStop-1:null) >>> (logid:) Done Configuring CloudStack Components >>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO >>> [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (localhost-startStop-1:null) >>> (logid:) Loaded module context [ipmitool] in 171 ms >>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO >>> [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (localhost-startStop-1:null) >>> (logid:) Starting module [system] >>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO >>> [o.a.c.s.l.CloudStackExtendedLifeCycle] (localhost-startStop-1:null) >>> (logid:) Starting CloudStack Components >>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO >>> [o.a.c.s.l.CloudStackExtendedLifeCycle] (localhost-startStop-1:null) >>> (logid:) Done Starting CloudStack Components >>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO >>> [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (localhost-startStop-1:null) >>> (logid:) Starting module [core] >>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO >>> [o.a.c.s.l.CloudStackExtendedLifeCycle] (localhost-startStop-1:null) >>> (logid:) Starting CloudStack Components >>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO >>> [c.c.h.h.m.HypervManagerImpl] (localhost-startStop-1:null) (logid:) Cleanup >>> mounted mount points used in previous session >>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO >>> [c.c.u.DatabaseIntegrityChecker] (localhost-startStop-1:null) (logid:) >>> Grabbing lock to check for database integrity. >>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO >>> [c.c.u.DatabaseIntegrityChecker] (localhost-startStop-1:null) (logid:) >>> Performing database integrity check >>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO >>> [c.c.c.ClusterManagerImpl] (localhost-startStop-1:null) (logid:) Starting >>> Cluster manager, msid : 2199193518866 >>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO >>> [c.c.c.ClusterServiceServletContainer] (Thread-10:null) (logid:) >>> Cluster service servlet container listening on port 9090 >>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO >>> [c.c.c.ClusterManagerImpl] (localhost-startStop-1:null) (logid:) Management >>> server 2199193518866, runId 1526887063086 is being started >>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO >>> [c.c.c.ClusterManagerImpl] (localhost-startStop-1:null) (logid:) Management >>> server (host id : 2) is being started at 10.38.3.18:9090 >>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO >>> [c.c.c.ClusterManagerImpl] (localhost-startStop-1:null) (logid:) Cluster >>> manager was started successfully >>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO >>> [c.c.h.HighAvailabilityManagerImpl] (HA-Worker-1:null) (logid:) >>> Starting work >>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO >>> [c.c.h.HighAvailabilityManagerImpl] (HA-Worker-4:null) (logid:) >>> Starting work >>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO >>> [c.c.h.HighAvailabilityManagerImpl] (HA-Worker-3:null) (logid:) >>> Starting work >>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO >>> [c.c.h.HighAvailabilityManagerImpl] (HA-Worker-0:null) (logid:) >>> Starting work >>> May 21 09:19:53 cldmanager.srv.cesga.es server: INFO >>> [c.c.h.HighAvailabilityManagerImpl] (HA-Worker-2:null) (logid:) >>> Starting work >>> May 21 09:19:55 cldmanager.srv.cesga.es server: log4j:WARN No appenders >>> could be found for logger (com.cloud.utils.db.ConnectionConcierge). >>> May 21 09:19:55 cldmanager.srv.cesga.es server: log4j:WARN Please >>> initialize the log4j system properly. >>> May 21 09:19:55 cldmanager.srv.cesga.es server: log4j:WARN See >>> http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. >>> May 21 09:19:55 cldmanager.srv.cesga.es server: Exception in thread >>> "Thread-10" java.lang.NoClassDefFoundError: >>> org/apache/log4j/spi/ThrowableInformation >>> May 21 09:19:55 cldmanager.srv.cesga.es server: at >>> org.apache.log4j.spi.LoggingEvent.<init>(LoggingEvent.java:165) >>> May 21 09:19:55 cldmanager.srv.cesga.es server: at >>> org.apache.log4j.Category.forcedLog(Category.java:391) >>> May 21 09:19:55 cldmanager.srv.cesga.es server: at >>> org.apache.log4j.Category.error(Category.java:322) >>> May 21 09:19:55 cldmanager.srv.cesga.es server: at >>> com.cloud.cluster.ClusterServiceServletContainer$ >>> ListenerThread.run(ClusterServiceServletContainer.java:168) >>> May 21 09:19:55 cldmanager.srv.cesga.es server: Caused by: >>> java.lang.ClassNotFoundException: org.apache.log4j.spi.Throwable >>> Information >>> May 21 09:19:55 cldmanager.srv.cesga.es server: at >>> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(W >>> ebappClassLoaderBase.java:1892) >>> May 21 09:19:55 cldmanager.srv.cesga.es server: at >>> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(W >>> ebappClassLoaderBase.java:1735) >>> May 21 09:19:55 cldmanager.srv.cesga.es server: ... 4 more >>> >>> >>> >>> El 17/05/2018 a las 7:39, ilya musayev escribió: >>> >>>> You must be running java 1.7 - I assume Suresh was alluding to this. >>>> We’ve seen this error with 1.8... >>>> >>>> On Wed, May 16, 2018 at 10:36 PM Suresh Kumar Anaparti < >>>> sureshkumar.anapa...@gmail.com> wrote: >>>> >>>> Hi Natalia, >>>>> >>>>> Any changes in the java version or system configuration? >>>>> >>>>> -Suresh >>>>> >>>>> On Thu, May 17, 2018 at 4:14 AM, Dag Sonstebo < >>>>> dag.sonst...@shapeblue.com> >>>>> wrote: >>>>> >>>>> Hi Natalia, >>>>>> >>>>>> My guess is you have some sort of corruption on your original host. >>>>>> Since >>>>>> the management service itself is stateless it’s generally much quicker >>>>>> >>>>> just >>>>> >>>>>> building a new one than trying to recover the old one. >>>>>> >>>>>> So my suggestions would be: >>>>>> - Recover your DB to a dedicated DB host if you haven’t done so >>>>>> already. >>>>>> - Build a new management server and just point this to the original >>>>>> DB as >>>>>> described in http://docs.cloudstack.apache.org/projects/cloudstack- >>>>>> installation/en/4.9/management-server/index.html# >>>>>> additional-management-servers >>>>>> NOTE: do not use the “–deploy-as” option as this will write a new >>>>>> blank >>>>>> database. If you used encryption keys during the original setup you >>>>>> also >>>>>> need to specify these. >>>>>> >>>>>> Once you have these two speaking you should in theory be back up and >>>>>> running again. >>>>>> >>>>>> Regards, >>>>>> Dag Sonstebo >>>>>> Cloud Architect >>>>>> ShapeBlue >>>>>> >>>>>> >>>>>> dag.sonst...@shapeblue.com >>>>>> www.shapeblue.com >>>>>> 53 Chandos Place, Covent Garden, London WC2N 4HSUK >>>>>> @shapeblue >>>>>> >>>>>> >>>>>> >>>>>> On 16/05/2018, 16:44, "Natalia Costas Lago" <nata...@cesga.es> wrote: >>>>>> >>>>>> >>>>>> Dear all, >>>>>> >>>>>> For some reason our cloudstack manager died and now we are not >>>>>> able >>>>>> >>>>> to >>>>> >>>>>> make it work. Our cloudstack version is 4.9.0. >>>>>> >>>>>> We tried to do a clean install of the manager and recover the >>>>>> >>>>> database >>>>> >>>>>> from the last backup, but we were not able to recover the >>>>>> service. >>>>>> >>>>>> At present we can see in the logs that it almost boot, but at >>>>>> the end >>>>>> we >>>>>> get this error: >>>>>> >>>>>> May 16 17:33:51 cldmanager.srv.cesga.es server: INFO >>>>>> [o.a.c.e.o.NetworkOrchestrator] (localhost-startStop-1:null) >>>>>> (logid:) >>>>>> Network Manager will run the NetworkGarbageCollector every '600' >>>>>> seconds. >>>>>> May 16 17:33:51 cldmanager.srv.cesga.es server: INFO >>>>>> [c.c.a.ApiServer] >>>>>> (Thread-12:null) (logid:) ApiServer listening on port 8096 >>>>>> May 16 17:33:51 cldmanager.srv.cesga.es server: INFO >>>>>> [o.a.c.s.SecondaryStorageManagerImpl] >>>>>> (localhost-startStop-1:null) >>>>>> (logid:) Start secondary storage vm manager >>>>>> May 16 17:33:52 cldmanager.srv.cesga.es server: log4j:WARN No >>>>>> appenders >>>>>> could be found for logger (com.cloud.utils.db.Connection >>>>>> Concierge). >>>>>> May 16 17:33:52 cldmanager.srv.cesga.es server: log4j:WARN >>>>>> Please >>>>>> initialize the log4j system properly. >>>>>> May 16 17:33:52 cldmanager.srv.cesga.es server: log4j:WARN See >>>>>> http://logging.apache.org/log4j/1.2/faq.html#noconfig for more >>>>>> info. >>>>>> May 16 17:34:20 cldmanager.srv.cesga.es server: Exception in >>>>>> thread >>>>>> "CapacityChecker" java.lang.NoClassDefFoundError: >>>>>> org/apache/cloudstack/managed/context/ManagedContextTimerTask$1 >>>>>> May 16 17:34:20 cldmanager.srv.cesga.es server: at >>>>>> org.apache.cloudstack.managed.context.ManagedContextTimerTask.run( >>>>>> ManagedContextTimerTask.java:27) >>>>>> May 16 17:34:20 cldmanager.srv.cesga.es server: at >>>>>> java.util.TimerThread.mainLoop(Timer.java:555) >>>>>> May 16 17:34:20 cldmanager.srv.cesga.es server: at >>>>>> java.util.TimerThread.run(Timer.java:505) >>>>>> May 16 17:34:20 cldmanager.srv.cesga.es server: Caused by: >>>>>> java.lang.ClassNotFoundException: >>>>>> org.apache.cloudstack.managed.context.ManagedContextTimerTask$1 >>>>>> May 16 17:34:20 cldmanager.srv.cesga.es server: at >>>>>> org.apache.catalina.loader.WebappClassLoaderBase.loadClass( >>>>>> WebappClassLoaderBase.java:1892) >>>>>> May 16 17:34:20 cldmanager.srv.cesga.es server: at >>>>>> org.apache.catalina.loader.WebappClassLoaderBase.loadClass( >>>>>> WebappClassLoaderBase.java:1735) >>>>>> May 16 17:34:20 cldmanager.srv.cesga.es server: ... 3 more >>>>>> May 16 17:34:24 cldmanager.srv.cesga.es server: Exception in >>>>>> thread >>>>>> "Timer-1" java.lang.NoClassDefFoundError: >>>>>> org/apache/cloudstack/managed/context/ManagedContextTimerTask$1 >>>>>> May 16 17:34:24 cldmanager.srv.cesga.es server: at >>>>>> org.apache.cloudstack.managed.context.ManagedContextTimerTask.run( >>>>>> ManagedContextTimerTask.java:27) >>>>>> May 16 17:34:24 cldmanager.srv.cesga.es server: at >>>>>> java.util.TimerThread.mainLoop(Timer.java:555) >>>>>> May 16 17:34:24 cldmanager.srv.cesga.es server: at >>>>>> java.util.TimerThread.run(Timer.java:505) >>>>>> >>>>>> Any help is appreciated.. >>>>>> >>>>>> Kind regards, >>>>>> >>>>>> NOTA: Para cualquier incidencia/consulta/petición que requiera >>>>>> seguimiento por favor enviar a helpdesk_comunicacio...@cesga.es >>>>>> >>>>>> -- >>>>>> ==================================================== >>>>>> Natalia Costas Lago >>>>>> Senior Communications Technician >>>>>> Galicia Supercomputing Centre (CESGA) >>>>>> (CESGA on Twitter | CESGA on Facebook) >>>>>> >>>>>> Avenida de Vigo, s/n (Campus Vida) >>>>>> 15705 Santiago de Compostela - SPAIN >>>>>> >>>>>> E-mail: nata...@cesga.es >>>>>> Cell: +34 981 56 98 10 (ext. 237) >>>>>> Fax: +34 981 59 46 16 >>>>>> Web: https://www.cesga.es/ >>>>>> ==================================================== >>>>>> [IMPORTANTE] La información contenida en este mensaje y >>>>>> sus posibles documentos adjuntos es privada y confidencial >>>>>> y está dirigida únicamente a su destinatario/a. Si usted no >>>>>> es el/la destinatario/a original de este mensaje, por favor >>>>>> elimínelo. La distribución o copia de este mensaje no está >>>>>> autorizada. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>> NOTA: Para cualquier incidencia/consulta/petición que requiera >>> seguimiento por favor enviar a helpdesk_comunicacio...@cesga.es >>> >>> >> NOTA: Para cualquier incidencia/consulta/petición que requiera >> seguimiento por favor enviar a helpdesk_comunicacio...@cesga.es >> >> > NOTA: Para cualquier incidencia/consulta/petición que requiera > seguimiento por favor enviar a helpdesk_comunicacio...@cesga.es > > -- > ==================================================== > Natalia Costas Lago > Senior Communications Technician > Galicia Supercomputing Centre (CESGA) > (CESGA on Twitter | CESGA on Facebook) > > Avenida de Vigo, s/n (Campus Vida) > 15705 Santiago de Compostela - SPAIN > > E-mail: nata...@cesga.es > Cell: +34 981 56 98 10 (ext. 237) > Fax: +34 981 59 46 16 > Web: https://www.cesga.es/ > ==================================================== > [IMPORTANTE] La información contenida en este mensaje y > sus posibles documentos adjuntos es privada y confidencial > y está dirigida únicamente a su destinatario/a. Si usted no > es el/la destinatario/a original de este mensaje, por favor > elimínelo. La distribución o copia de este mensaje no está > autorizada. > >
