For the past 6 months or so we've had intermittent crashes; at this point we
seem to be going down once every 1-3 weeks. When this happens, tomcat is
still running and postgres is stopped. We restart both and everything is
fine until it happens again. We finally decided to move our monitoring cron
from daily to hourly, which makes reading the error logs a little less
confusing. Below please find tomcat / catalina error messages (these seem
to be the most informative). I've removed tracebacks for the most part,
except for the first severe null pointer error, to make them more legible.
We're running DSpace 1.5.1. If anything looks familiar here, or more
significant than anything else, I appreciate any suggestions as to what the
problem could be!
Many thanks,
Anna
TriCollege Library Consortium
Dspace went down between 6pm and 7pm on dec 13th. As usual, postgres was
down.
The site is back up now but we seem to be out of theories.
The logs contain many warnings like:
Dec 13, 2010 4:38:34 PM org.apache.jk.core.MsgContext action
WARNING: Error sending end packet
java.net.SocketException: Broken pipe
and
Dec 13, 2010 4:38:34 PM org.apache.jk.common.ChannelSocket processConnection
WARNING: processCallbacks status 2
***** Suspicious errors begin at 6:58 pm on the day in
question! (see catalina logs for full tracebacks)
Dec 13, 2010 6:58:30 PM org.apache.catalina.connector.CoyoteAdapter service
SEVERE: An exception or error occurred in the container during the request
processing java.lang.NullPointerException
at
org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:1995)
at
org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:1790)
at
org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:873)
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1326)
at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1205)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:336)
at org.apache.log4j.spi.LoggingEvent.<init>(LoggingEvent.java:154)
at org.apache.log4j.Category.forcedLog(Category.java:388)
at org.apache.log4j.Category.log(Category.java:853)
at
org.apache.commons.logging.impl.Log4JLogger.error(Log4JLogger.java:193)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:250)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
at
org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200)
at
org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:775)
at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:704)
at
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:897)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:685)
at java.lang.Thread.run(Thread.java:636)
Dec 13, 2010 6:58:37 PM org.apache.naming.resources.FileDirContext list
WARNING: Could not get dir listing for /var/dspace/webapps/sword/WEB-INF/lib
Dec 13, 2010 6:58:37 PM org.apache.catalina.loader.WebappClassLoader
modified
INFO: Additional JARs have been added
Dec 13, 2010 6:58:37 PM org.apache.catalina.core.StandardContext reload
INFO: Reloading this Context has started
Dec 13, 2010 6:58:37 PM
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor
processChildren
SEVERE: Exception invoking periodic operation:
java.lang.NoClassDefFoundError: org/apache/log4j/spi/ThrowableInformation
Dec 13, 2010 6:58:37 PM org.apache.naming.resources.FileDirContext list
WARNING: Could not get dir listing for /var/dspace/webapps/jspui/WEB-INF/lib
Dec 13, 2010 6:58:37 PM org.apache.catalina.loader.WebappClassLoader
modified
INFO: Additional JARs have been added
Dec 13, 2010 6:58:37 PM org.apache.catalina.core.StandardContext reload
INFO: Reloading this Context has started
Dec 13, 2010 6:58:37 PM org.apache.catalina.startup.ContextConfig
defaultWebConfig
SEVERE: Missing default web.xml, using application web.xml only conf/web.xml
/usr/share/tomcat5/conf/web.xml
java.io.FileNotFoundException: /usr/share/tomcat5/conf/web.xml (Too many
open
files)
Dec 13, 2010 6:58:37 PM org.apache.naming.resources.FileDirContext list
WARNING: Could not get dir listing for /var/dspace/webapps/jspui/WEB-INF
Dec 13, 2010 6:58:37 PM org.apache.catalina.startup.TldConfig lifecycleEvent
SEVERE: Error processing TLD files for context path /dspace-jspui
***** Similar errors repeat a few times, until we get to:
Dec 13, 2010 6:58:57 PM org.apache.catalina.core.ContainerBase stop
INFO: Container
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/dspace-sword].[jsp]
has not been started
Dec 13, 2010 6:58:57 PM org.apache.catalina.core.ContainerBase stop
INFO: Container
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/dspace-sword].[default]
has not been started
Dec 13, 2010 6:58:57 PM org.apache.catalina.core.ContainerBase stop
INFO: Container
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/dspace-sword].[deposit]
has not been started
Dec 13, 2010 6:58:57 PM org.apache.catalina.core.ContainerBase stop
INFO: Container
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/dspace-sword].[servicedocument]
has not been started
Dec 13, 2010 6:58:57 PM
org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor
processChildren
SEVERE: Exception invoking periodic operation:
java.lang.NoClassDefFoundError: org/apache/log4j/spi/ThrowableInformation
***** Then we come to the exciting:
Dec 13, 2010 6:58:57 PM org.apache.catalina.loader.WebappClassLoader
loadClass
INFO: Illegal access: this web application instance has been stopped
already.
Could not load org.apache.log4j.spi.ThrowableInformation. The eventual
following stack trace is caused by an error thrown for debugging purposes as
well as to attempt to terminate the thread which caused the illegal access,
and
has no functional impact.
java.lang.IllegalStateException
***** This one is similar to the jspui one, except for oai:
Dec 13, 2010 6:59:18 PM org.apache.naming.resources.FileDirContext list
WARNING: Could not get dir listing for /var/dspace/webapps/oai/WEB-INF
Dec 13, 2010 6:59:18 PM org.apache.catalina.startup.TldConfig lifecycleEvent
SEVERE: Error processing TLD files for context path /dspace-oai
java.io.FileNotFoundException:
/usr/share/tomcat5/work/Catalina/localhost/dspace-oai/tldCache.ser (Too many
open files)
***** Later we come to a place where jspui has already been started:
Dec 13, 2010 6:59:38 PM org.apache.catalina.core.StandardContext start
INFO: Container
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/dspace-jspui]
has already been started
Dec 13, 2010 6:59:38 PM org.apache.catalina.loader.WebappClassLoader
modified
INFO: Additional JARs have been added
Dec 13, 2010 6:59:38 PM org.apache.catalina.core.StandardContext reload
INFO: Reloading this Context has started
Dec 13, 2010 6:59:38 PM org.apache.catalina.startup.ContextConfig
defaultWebConfig
SEVERE: Missing default web.xml, using application web.xml only conf/web.xml
/usr/share/tomcat5/conf/web.xml
------------------------------------------------------------------------------
Lotusphere 2011
Register now for Lotusphere 2011 and learn how
to connect the dots, take your collaborative environment
to the next level, and enter the era of Social Business.
http://p.sf.net/sfu/lotusphere-d2d
_______________________________________________
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech
