Hello!
First of all, forgive me as I'm new to DSpace, nor am I the one who
set up the system.
That being said, we are consistently receiving errors referring to
"Too many files open", in which we must restart the Tomcat service
two or more times a day (which has been happening for nearly a month
now). Searching on our system is fine, it occurs when our users are
attempting to edit the metadata and/or uploading/removing bitstreams
- or when the system attempts to send email alerts. At this point,
other than searching, the system is unusable until the services are
restarted.
I have done several searches in Google as well as in the Dspace
mailing lists, but I haven't found an actual solution to the problem,
although I have found discussions on the topic.
In relation to all the discussions I've read, I have provided most of
the information requested by them in a hope that others may have an
answer, or at least speed things up a bit.
DSpace version:
1.4.1
Operating System:
OS Version:
# cat /etc/redhat-release
Red Hat Enterprise Linux AS release 4 (Nahant Update 4)
# uname -a
Linux <node name removed> 2.6.9-42.0.10.ELsmp #1 SMP Fri Feb 16
17:17:21 EST 2007 i686 i686 i386 GNU/Linux
Cron Jobs:
dspace cron jobs
]# crontab -u dspace -l
# Send out subscription emails at 1:00 everyday
0 1 * * * /opt/dspace/bin/sub-daily
#Run the media filter at 2:00 every day
0 2 * * * /opt/dspace/bin/filter-media
#Run the checksum checker at 3:00
0 3 * * * /opt/dspace/bin/checker -lp
#Mail the results to the sysadmin at 4:00
0 4 * * * /opt/dspace/bin/dsrun org.dspace.checker.DailyReportEmailer
-c
#Run stat analyses
0 5 * * * /opt/dspace/bin/stat-general
10 5 * * * /opt/dspace/bin/stat-monthly
20 5 * * * /opt/dspace/bin/stat-report-general
30 5 * * * /opt/dspace/bin/stat-report-monthly
Notice that index-all is not included, as my understand is that
filter-media includes a rebuild of the indexes. I have manually
rebuilt the indexes which does not resolve the issue.
INodes:
# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/cciss/c0d0p1 524288 111084 413204 22% /
none 214062 1 214061 1% /dev/shm
/dev/cciss/c0d0p3 17137664 644078 16493586 4% /var
files-nr Information:
# cat /proc/sys/fs/file-nr
5120 0 580170
Example Error from Tomcat's catalina.out:
Feb 15, 2008 10:28:05 AM org.apache.tomcat.util.net.PoolTcpEndpoint
acceptSocket
SEVERE: Endpoint
ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=8080] ignored
exception: java.net.SocketException: Too many open files
java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353)
at java.net.ServerSocket.implAccept(ServerSocket.java:448)
at java.net.ServerSocket.accept(ServerSocket.java:419)
at
org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(Def
aultServerSocketFactory.java:61)
at
org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoin
t.java:408)
at
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFoll
owerWorkerThread.java:71)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPo
ol.java:685)
at java.lang.Thread.run(Thread.java:534)
Feb 15, 2008 10:28:05 AM org.apache.tomcat.util.net.PoolTcpEndpoint
acceptSocket
Example corresponding errors in dspace.log:
java.io.FileNotFoundException:
/var/opt/dspace/assetstore/58/21/34/5821340475210663413811699708242116
0566 (Too many open files)
va.io.FileNotFoundException:
/var/opt/dspace/assetstore/58/21/34/5821340475210663413811699708242116
0566 (Too many open files)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:106)
at
edu.sdsc.grid.io.local.LocalFileInputStream.open(LocalFileInputStream.
java:171)
at
edu.sdsc.grid.io.GeneralFileInputStream.<init>(GeneralFileInputStream.
java:145)
at
edu.sdsc.grid.io.local.LocalFileInputStream.<init>(LocalFileInputStrea
m.java:139)
at
edu.sdsc.grid.io.FileFactory.newFileInputStream(FileFactory.java:630)
at
org.dspace.storage.bitstore.BitstreamStorageManager.retrieve(Bitstream
StorageManager.java:553)
at org.dspace.content.Bitstream.retrieve(Bitstream.java:512)
at
org.dspace.app.webui.servlet.BitstreamServlet.doDSGet(BitstreamServlet
.java:205)
at
org.dspace.app.webui.servlet.DSpaceServlet.processRequest(DSpaceServle
t.java:159)
at
org.dspace.app.webui.servlet.DSpaceServlet.doGet(DSpaceServlet.java:10
0)
at
javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
at
javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appli
cationFilterChain.java:269)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFi
lterChain.java:188)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperVa
lve.java:210)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextVa
lve.java:174)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.ja
va:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.ja
va:117)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValv
e.java:108)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java
:151)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:
870)
at
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.pr
ocessConnection(Http11BaseProtocol.java:665)
at
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoi
nt.java:528)
at
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFoll
owerWorkerThread.java:81)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPo
ol.java:685)
at java.lang.Thread.run(Thread.java:534)
2008-02-15 10:28:06,006 WARN org.dspace.app.webui.util.UIUtil @
Unable to send email alert
java.io.FileNotFoundException:
/opt/dspace/config/emails/internal_error (Too many open files)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:106)
at java.io.FileInputStream.<init>(FileInputStream.java:66)
at java.io.FileReader.<init>(FileReader.java:41)
at
org.dspace.core.ConfigurationManager.getEmail(ConfigurationManager.jav
a:282)
at org.dspace.app.webui.util.UIUtil.sendAlert(UIUtil.java:533)
at
org.dspace.app.webui.servlet.InternalErrorServlet.doGet(InternalErrorS
ervlet.java:86)
at
javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
at
javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appli
cationFilterChain.java:269)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFi
lterChain.java:188)
at
org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispa
tcher.java:691)
at
org.apache.catalina.core.ApplicationDispatcher.processRequest(Applicat
ionDispatcher.java:471)
at
org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDi
spatcher.java:403)
at
org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDisp
atcher.java:301)
at
org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.ja
va:364)
at
org.apache.catalina.core.StandardHostValve.throwable(StandardHostValve
.java:213)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.ja
va:135)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.ja
va:117)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValv
e.java:108)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java
:151)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:
870)
at
org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.pr
ocessConnection(Http11BaseProtocol.java:665)
at
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoi
nt.java:528)
at
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFoll
owerWorkerThread.java:81)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPo
ol.java:685)
at java.lang.Thread.run(Thread.java:534)
Any assistance would be greatly appreciated.
Thanks!
Paul Marlow
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
DSpace-tech mailing list
DSpace-tech@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-tech
