Hi all We upgraded to 1.7.2 and everything seems fine except that at least once per day the web service becomes unresponsive. Also, when stopping the tomcat the processors shoots up to 102% utilisation.
This then can only be fixed by killing the process and restarting the instance
again. That is obviously not sustainable. When looking at the catalina.out I
see:
Nov 5, 2011 12:39:53 PM org.apache.solr.update.SolrIndexWriter finalize
87877 SEVERE: SolrIndexWriter was not closed prior to finalize(), indicates a
bug -- POSSIBLE RESOURCE LEAK!!!
87878 Nov 5, 2011 12:39:58 PM org.apache.solr.common.SolrException log
87879 SEVERE: java.lang.OutOfMemoryError: PermGen space
87880
87881 Nov 5, 2011 1:31:46 PM
org.apache.solr.update.processor.LogUpdateProcessor finish
87882 INFO: {} 0 1001
87883 Nov 5, 2011 1:31:47 PM org.apache.solr.update.SolrIndexWriter finalize
87884 SEVERE: SolrIndexWriter was not closed prior to finalize(), indicates a
bug -- POSSIBLE RESOURCE LEAK!!!
87885 Nov 5, 2011 1:31:49 PM org.apache.solr.common.SolrException log
87886 SEVERE: java.lang.OutOfMemoryError: PermGen space
87887
87888 Exception in thread "http-80-38" java.lang.NullPointerException
87889 Exception in thread "http-80-37" java.lang.NullPointerException
87890 Exception in thread "http-80-29" java.lang.OutOfMemoryError: PermGen space
87891 Exception in thread "http-80-10" java.lang.OutOfMemoryError: PermGen space
87892 Exception in thread "http-80-4" java.lang.OutOfMemoryError: PermGen space
87893 Exception in thread "http-80-8" java.lang.OutOfMemoryError: PermGen space
87894 Exception in thread "http-80-23" java.lang.OutOfMemoryError: PermGen space
87895 Exception in thread "http-80-27" java.lang.OutOfMemoryError: PermGen space
87896 Exception in thread "http-80-20" java.lang.OutOfMemoryError: PermGen space
87897 Exception in thread "http-80-7" java.lang.OutOfMemoryError: PermGen space
87898 Exception in thread "http-80-11" java.lang.OutOfMemoryError: PermGen space
87899 Exception in thread "http-80-24" java.lang.OutOfMemoryError: PermGen space
87900 Exception in thread "http-80-15" java.lang.OutOfMemoryError: PermGen space
87901 Exception in thread "http-80-5" java.lang.OutOfMemoryError: PermGen space
87902 Exception in thread "http-80-3" java.lang.OutOfMemoryError: PermGen space
87903 Exception in thread "http-80-1" java.lang.OutOfMemoryError: PermGen space
87904 Exception in thread "http-80-28" java.lang.OutOfMemoryError: PermGen space
87905 Exception in thread "http-80-33" java.lang.OutOfMemoryError: PermGen space
87906 Exception in thread "http-80-35" java.lang.OutOfMemoryError: PermGen space
87907 Exception in thread "http-80-31" java.lang.OutOfMemoryError: PermGen space
87908 Exception in thread "http-80-36" java.lang.OutOfMemoryError: PermGen space
87909 Exception in thread "http-80-6" java.lang.OutOfMemoryError: PermGen space
87910 Exception in thread "http-80-14" java.lang.OutOfMemoryError: PermGen space
87911 Exception in thread "http-80-25" java.lang.OutOfMemoryError: PermGen space
87912 Exception in thread "http-80-9" java.lang.OutOfMemoryError: PermGen space
87913 Exception in thread "http-80-12" java.lang.OutOfMemoryError: PermGen space
87914 Exception in thread "http-80-32" java.lang.OutOfMemoryError: PermGen space
87915 Exception in thread "http-80-21" java.lang.OutOfMemoryError: PermGen space
87916 Exception in thread "http-80-22" java.lang.OutOfMemoryError: PermGen space
87917 Exception in thread "http-80-19" java.lang.OutOfMemoryError: PermGen space
87918 Exception in thread "http-80-17" java.lang.OutOfMemoryError: PermGen space
87919 Exception in thread "http-80-2" java.lang.OutOfMemoryError: PermGen space
87920 Exception in thread "http-80-26" java.lang.OutOfMemoryError: PermGen space
87921 Exception in thread "http-80-34" java.lang.OutOfMemoryError: PermGen space
87922 Exception in thread "http-80-18" java.lang.OutOfMemoryError: PermGen space
87923 Exception in thread "http-80-16" java.lang.OutOfMemoryError: PermGen space
87924 Exception in thread "http-80-55" java.lang.OutOfMemoryError: PermGen space
87925 Exception in thread "http-80-52" java.lang.OutOfMemoryError: PermGen space
87926 Exception in thread "http-80-48" java.lang.OutOfMemoryError: PermGen space
87927 Exception in thread "http-80-47" java.lang.OutOfMemoryError: PermGen space
87928 Exception in thread "http-80-58" java.lang.OutOfMemoryError: PermGen space
87929 Exception in thread "http-80-42" java.lang.OutOfMemoryError: PermGen space
87930 Exception in thread "http-80-59" java.lang.OutOfMemoryError: PermGen space
87931 Exception in thread "http-80-49" java.lang.OutOfMemoryError: PermGen space
And later on after trying to start the service:
Cannot open PID file /var/run/jsvc.pid, PID is 1098
88001 Service exit with a return value of 255
88002 Cannot open PID file /var/run/jsvc.pid, PID is 1162
88003 Service exit with a return value of 255
88004 Service killed by signal 9
88005 Nov 7, 2011 7:36:53 AM org.apache.catalina.core.AprLifecycleListener init
88006 INFO: The APR based Apache Tomcat Native library which allows optimal
performance in production environments was not found on the java.library.path:
/home/jfclere/jakarta-tomcat-connectors/jni/native/.libs
88007 Nov 7, 2011 7:36:53 AM org.apache.coyote.http11.Http11Protocol init
88008 INFO: Initializing Coyote HTTP/1.1 on http-80
88009 Nov 7, 2011 7:36:53 AM org.apache.catalina.startup.Catalina load
88010 INFO: Initialization processed in 362 ms
88011 Nov 7, 2011 7:36:53 AM org.apache.catalina.core.StandardService start
88012 INFO: Starting service Catalina
88013 Nov 7, 2011 7:36:53 AM org.apache.catalina.core.StandardEngine start
88014 INFO: Starting Servlet Engine: Apache Tomcat/6.0.32
88015 Nov 7, 2011 7:36:53 AM org.apache.catalina.startup.HostConfig
deployDescriptor
88016 INFO: Deploying configuration descriptor manager.xml
88017 Nov 7, 2011 7:36:53 AM org.apache.catalina.core.StandardContext
resourcesStart
88018 SEVERE: Error starting static Resources
88019 java.lang.IllegalArgumentException: Document base
/app/dspace/webapps/manager does not exist or is not a readable directory
88020 at
org.apache.naming.resources.FileDirContext.setDocBase(FileDirContext.java:142)
88021 at
org.apache.catalina.core.StandardContext.resourcesStart(StandardContext.java:4319)
88022 at
org.apache.catalina.core.StandardContext.start(StandardContext.java:4488)
88023 at
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
88024 at
org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
88025 at
org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
88026 at
org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:675)
88027 at
org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:601)
88028 at
org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:502)
88029 at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1315)
88030 at
org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
88031 at
org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
88032 at
org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1061)
88033 at org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
88034 at
org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053)
88035 at
org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
88036 at
org.apache.catalina.core.StandardService.start(StandardService.java:525)
88037 at
org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
88038 at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
88039 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
88040 at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
88041 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
88042 at java.lang.reflect.Method.invoke(Method.java:597)
88043 at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
We use Rhel 6 64 bit on a dell R710 with 24 Gb Ram. I changed the catalina.sh
file in /tomcat/bin to:
CATALINA_OPTS=$CATALINA_OPTS"
-Djava.library.path=/home/jfclere/jakarta-tomcat-connectors/jni/native/.libs
-Xmx12288M -Xms12288M -Dfile.encoding=UTF-8 -XX:+UseConcMarkSweepGC
-XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabl ed
-XX:MaxPermSize=1024M"
Is this the right place ? It doesn't seem to make a difference.
My thoughts are that solr is causing issues? How do I disable it totally to see
if this is indeed the case?
Any help/ideas will be appreciated.
Thanks
Barrie Swanepoel
Systems Administrator
University of Johannesburg
Library and Information Centre
PO Box 524
Auckland Park
Johannesburg
Republic of South Africa
Tel: +27 11 559 2347
Cell: 082 565 7743
[Description: UJ_emailer_01]
________________________________
This email and all contents are subject to the following disclaimer:
http://disclaimer.uj.ac.za
<<inline: image003.jpg>>
------------------------------------------------------------------------------ RSA(R) Conference 2012 Save $700 by Nov 18 Register now http://p.sf.net/sfu/rsa-sfdev2dev1
_______________________________________________ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
