Hi,
We finally were able to recover the manager :/
The problem was due to a database modification made by hand a couple of
months ago in order to eliminate one of the zombie virtual machines
(machines that cannot be expunged because they are dependent on a
network that was deleted before).
The state of one of the machines was modified and Cloudstack did not
like that change (setting a VM state as Destroyed), this eliminated the
VM for the web interface but we did not detect the problem as we did not
restart the management server. All backups since then were wrong and for
some reason the management server crashed last week.
Now it is fully operational :)
Thank you,
Kind regards,
El 21/05/2018 a las 12:19, Natalia Costas Lago escribió:
This is what I can see in localhost log:
cat /var/log/cloudstack/management/localhost.2018-05-21.log
[...]
SEVERE: Exception sending context initialized event to listener
instance of class
org.apache.cloudstack.spring.module.web.CloudStackContextLoaderListener
java.lang.NullPointerException
at
com.cloud.vm.UserVmManagerImpl.loadVmDetailsInMapForExternalDhcpIp(UserVmManagerImpl.java:2019)
at
com.cloud.vm.UserVmManagerImpl.start(UserVmManagerImpl.java:1999)
at
org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle$1.with(CloudStackExtendedLifeCycle.java:75)
at
org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.with(CloudStackExtendedLifeCycle.java:154)
at
org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycle.startBeans(CloudStackExtendedLifeCycle.java:72)
at
org.apache.cloudstack.spring.lifecycle.CloudStackExtendedLifeCycleStart.run(CloudStackExtendedLifeCycleStart.java:46)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet$1.with(DefaultModuleDefinitionSet.java:105)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:245)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:250)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:250)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.withModule(DefaultModuleDefinitionSet.java:233)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.startContexts(DefaultModuleDefinitionSet.java:97)
at
org.apache.cloudstack.spring.module.model.impl.DefaultModuleDefinitionSet.load(DefaultModuleDefinitionSet.java:80)
at
org.apache.cloudstack.spring.module.factory.ModuleBasedContextFactory.loadModules(ModuleBasedContextFactory.java:37)
at
org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.init(CloudStackSpringContext.java:71)
at
org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:58)
at
org.apache.cloudstack.spring.module.factory.CloudStackSpringContext.<init>(CloudStackSpringContext.java:62)
at
org.apache.cloudstack.spring.module.web.CloudStackContextLoaderListener.contextInitialized(CloudStackContextLoaderListener.java:52)
at
org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:5118)
at
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5634)
at
org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
at
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
at
org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:875)
at
org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
at
org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1260)
at
org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:2002)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:473)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
at java.lang.Thread.run(Thread.java:748)
El 21/05/2018 a las 9:21, Natalia Costas Lago escribió:
It has been working with 1.8 for long...
I switched to java 1.7, but java errors persist.
We recovered backups from a couple of days prior to the disaster, but
we are still unlucky, it seems the problem is also in the backups copy.
I installed a fresh cloudstack and copied a recent database and the
manager still gives and Exception but to a later point in the execution:
May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
[o.a.c.s.l.CloudStackExtendedLifeCycle] (localhost-startStop-1:null)
(logid:) Configuring CloudStack Components
May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
[o.a.c.s.l.CloudStackExtendedLifeCycle] (localhost-startStop-1:null)
(logid:) Done Configuring CloudStack Components
May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
[o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
(localhost-startStop-1:null) (logid:) Loaded module context
[ipmitool] in 171 ms
May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
[o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
(localhost-startStop-1:null) (logid:) Starting module [system]
May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
[o.a.c.s.l.CloudStackExtendedLifeCycle] (localhost-startStop-1:null)
(logid:) Starting CloudStack Components
May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
[o.a.c.s.l.CloudStackExtendedLifeCycle] (localhost-startStop-1:null)
(logid:) Done Starting CloudStack Components
May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
[o.a.c.s.m.m.i.DefaultModuleDefinitionSet]
(localhost-startStop-1:null) (logid:) Starting module [core]
May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
[o.a.c.s.l.CloudStackExtendedLifeCycle] (localhost-startStop-1:null)
(logid:) Starting CloudStack Components
May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
[c.c.h.h.m.HypervManagerImpl] (localhost-startStop-1:null) (logid:)
Cleanup mounted mount points used in previous session
May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
[c.c.u.DatabaseIntegrityChecker] (localhost-startStop-1:null)
(logid:) Grabbing lock to check for database integrity.
May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
[c.c.u.DatabaseIntegrityChecker] (localhost-startStop-1:null)
(logid:) Performing database integrity check
May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
[c.c.c.ClusterManagerImpl] (localhost-startStop-1:null) (logid:)
Starting Cluster manager, msid : 2199193518866
May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
[c.c.c.ClusterServiceServletContainer] (Thread-10:null) (logid:)
Cluster service servlet container listening on port 9090
May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
[c.c.c.ClusterManagerImpl] (localhost-startStop-1:null) (logid:)
Management server 2199193518866, runId 1526887063086 is being started
May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
[c.c.c.ClusterManagerImpl] (localhost-startStop-1:null) (logid:)
Management server (host id : 2) is being started at 10.38.3.18:9090
May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
[c.c.c.ClusterManagerImpl] (localhost-startStop-1:null) (logid:)
Cluster manager was started successfully
May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
[c.c.h.HighAvailabilityManagerImpl] (HA-Worker-1:null) (logid:)
Starting work
May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
[c.c.h.HighAvailabilityManagerImpl] (HA-Worker-4:null) (logid:)
Starting work
May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
[c.c.h.HighAvailabilityManagerImpl] (HA-Worker-3:null) (logid:)
Starting work
May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
[c.c.h.HighAvailabilityManagerImpl] (HA-Worker-0:null) (logid:)
Starting work
May 21 09:19:53 cldmanager.srv.cesga.es server: INFO
[c.c.h.HighAvailabilityManagerImpl] (HA-Worker-2:null) (logid:)
Starting work
May 21 09:19:55 cldmanager.srv.cesga.es server: log4j:WARN No
appenders could be found for logger
(com.cloud.utils.db.ConnectionConcierge).
May 21 09:19:55 cldmanager.srv.cesga.es server: log4j:WARN Please
initialize the log4j system properly.
May 21 09:19:55 cldmanager.srv.cesga.es server: log4j:WARN See
http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
May 21 09:19:55 cldmanager.srv.cesga.es server: Exception in thread
"Thread-10" java.lang.NoClassDefFoundError:
org/apache/log4j/spi/ThrowableInformation
May 21 09:19:55 cldmanager.srv.cesga.es server: at
org.apache.log4j.spi.LoggingEvent.<init>(LoggingEvent.java:165)
May 21 09:19:55 cldmanager.srv.cesga.es server: at
org.apache.log4j.Category.forcedLog(Category.java:391)
May 21 09:19:55 cldmanager.srv.cesga.es server: at
org.apache.log4j.Category.error(Category.java:322)
May 21 09:19:55 cldmanager.srv.cesga.es server: at
com.cloud.cluster.ClusterServiceServletContainer$ListenerThread.run(ClusterServiceServletContainer.java:168)
May 21 09:19:55 cldmanager.srv.cesga.es server: Caused by:
java.lang.ClassNotFoundException:
org.apache.log4j.spi.ThrowableInformation
May 21 09:19:55 cldmanager.srv.cesga.es server: at
org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1892)
May 21 09:19:55 cldmanager.srv.cesga.es server: at
org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1735)
May 21 09:19:55 cldmanager.srv.cesga.es server: ... 4 more
El 17/05/2018 a las 7:39, ilya musayev escribió:
You must be running java 1.7 - I assume Suresh was alluding to this.
We’ve seen this error with 1.8...
On Wed, May 16, 2018 at 10:36 PM Suresh Kumar Anaparti <
sureshkumar.anapa...@gmail.com> wrote:
Hi Natalia,
Any changes in the java version or system configuration?
-Suresh
On Thu, May 17, 2018 at 4:14 AM, Dag Sonstebo
<dag.sonst...@shapeblue.com>
wrote:
Hi Natalia,
My guess is you have some sort of corruption on your original
host. Since
the management service itself is stateless it’s generally much
quicker
just
building a new one than trying to recover the old one.
So my suggestions would be:
- Recover your DB to a dedicated DB host if you haven’t done so
already.
- Build a new management server and just point this to the
original DB as
described in http://docs.cloudstack.apache.org/projects/cloudstack-
installation/en/4.9/management-server/index.html#
additional-management-servers
NOTE: do not use the “–deploy-as” option as this will write a new
blank
database. If you used encryption keys during the original setup
you also
need to specify these.
Once you have these two speaking you should in theory be back up and
running again.
Regards,
Dag Sonstebo
Cloud Architect
ShapeBlue
dag.sonst...@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue
On 16/05/2018, 16:44, "Natalia Costas Lago" <nata...@cesga.es> wrote:
Dear all,
For some reason our cloudstack manager died and now we are
not able
to
make it work. Our cloudstack version is 4.9.0.
We tried to do a clean install of the manager and recover the
database
from the last backup, but we were not able to recover the
service.
At present we can see in the logs that it almost boot, but at
the end
we
get this error:
May 16 17:33:51 cldmanager.srv.cesga.es server: INFO
[o.a.c.e.o.NetworkOrchestrator] (localhost-startStop-1:null)
(logid:)
Network Manager will run the NetworkGarbageCollector every '600'
seconds.
May 16 17:33:51 cldmanager.srv.cesga.es server: INFO
[c.c.a.ApiServer]
(Thread-12:null) (logid:) ApiServer listening on port 8096
May 16 17:33:51 cldmanager.srv.cesga.es server: INFO
[o.a.c.s.SecondaryStorageManagerImpl]
(localhost-startStop-1:null)
(logid:) Start secondary storage vm manager
May 16 17:33:52 cldmanager.srv.cesga.es server: log4j:WARN No
appenders
could be found for logger
(com.cloud.utils.db.ConnectionConcierge).
May 16 17:33:52 cldmanager.srv.cesga.es server: log4j:WARN
Please
initialize the log4j system properly.
May 16 17:33:52 cldmanager.srv.cesga.es server: log4j:WARN See
http://logging.apache.org/log4j/1.2/faq.html#noconfig for
more info.
May 16 17:34:20 cldmanager.srv.cesga.es server: Exception in
thread
"CapacityChecker" java.lang.NoClassDefFoundError:
org/apache/cloudstack/managed/context/ManagedContextTimerTask$1
May 16 17:34:20 cldmanager.srv.cesga.es server: at
org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(
ManagedContextTimerTask.java:27)
May 16 17:34:20 cldmanager.srv.cesga.es server: at
java.util.TimerThread.mainLoop(Timer.java:555)
May 16 17:34:20 cldmanager.srv.cesga.es server: at
java.util.TimerThread.run(Timer.java:505)
May 16 17:34:20 cldmanager.srv.cesga.es server: Caused by:
java.lang.ClassNotFoundException:
org.apache.cloudstack.managed.context.ManagedContextTimerTask$1
May 16 17:34:20 cldmanager.srv.cesga.es server: at
org.apache.catalina.loader.WebappClassLoaderBase.loadClass(
WebappClassLoaderBase.java:1892)
May 16 17:34:20 cldmanager.srv.cesga.es server: at
org.apache.catalina.loader.WebappClassLoaderBase.loadClass(
WebappClassLoaderBase.java:1735)
May 16 17:34:20 cldmanager.srv.cesga.es server: ... 3 more
May 16 17:34:24 cldmanager.srv.cesga.es server: Exception in
thread
"Timer-1" java.lang.NoClassDefFoundError:
org/apache/cloudstack/managed/context/ManagedContextTimerTask$1
May 16 17:34:24 cldmanager.srv.cesga.es server: at
org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(
ManagedContextTimerTask.java:27)
May 16 17:34:24 cldmanager.srv.cesga.es server: at
java.util.TimerThread.mainLoop(Timer.java:555)
May 16 17:34:24 cldmanager.srv.cesga.es server: at
java.util.TimerThread.run(Timer.java:505)
Any help is appreciated..
Kind regards,
NOTA: Para cualquier incidencia/consulta/petición que requiera
seguimiento por favor enviar a helpdesk_comunicacio...@cesga.es
--
====================================================
Natalia Costas Lago
Senior Communications Technician
Galicia Supercomputing Centre (CESGA)
(CESGA on Twitter | CESGA on Facebook)
Avenida de Vigo, s/n (Campus Vida)
15705 Santiago de Compostela - SPAIN
E-mail: nata...@cesga.es
Cell: +34 981 56 98 10 (ext. 237)
Fax: +34 981 59 46 16
Web: https://www.cesga.es/
====================================================
[IMPORTANTE] La información contenida en este mensaje y
sus posibles documentos adjuntos es privada y confidencial
y está dirigida únicamente a su destinatario/a. Si usted no
es el/la destinatario/a original de este mensaje, por favor
elimínelo. La distribución o copia de este mensaje no está
autorizada.
NOTA: Para cualquier incidencia/consulta/petición que requiera
seguimiento por favor enviar a helpdesk_comunicacio...@cesga.es
NOTA: Para cualquier incidencia/consulta/petición que requiera
seguimiento por favor enviar a helpdesk_comunicacio...@cesga.es
NOTA: Para cualquier incidencia/consulta/petición que requiera
seguimiento por favor enviar a helpdesk_comunicacio...@cesga.es
--
====================================================
Natalia Costas Lago
Senior Communications Technician
Galicia Supercomputing Centre (CESGA)
(CESGA on Twitter | CESGA on Facebook)
Avenida de Vigo, s/n (Campus Vida)
15705 Santiago de Compostela - SPAIN
E-mail: nata...@cesga.es
Cell: +34 981 56 98 10 (ext. 237)
Fax: +34 981 59 46 16
Web: https://www.cesga.es/
====================================================
[IMPORTANTE] La información contenida en este mensaje y
sus posibles documentos adjuntos es privada y confidencial
y está dirigida únicamente a su destinatario/a. Si usted no
es el/la destinatario/a original de este mensaje, por favor
elimínelo. La distribución o copia de este mensaje no está
autorizada.
