Randall, To answer your questions, Tomcat is running under the tomcat user, with ulimits as follows:
ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited file size (blocks, -f) unlimited pending signals (-i) 1024 max locked memory (kbytes, -l) 32 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 110591 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited I also read another post, indicating that there could be memory problems with indexing a large repository, such that "the org.dspace.core.Context object cache, which in the case of a full re-index will currently try to get an in-memory copy of every object in the DSpace instance." It was suggested to add: target.decache(); in the org.dspace.search.DSIndexer.indexAllItems(), as well as in org.dspace.browse.Browse.indexAll(). Now we currently aren't experiencing memory problems while running our indexing, so I'm not sure if this could be a potential cause or not. Does anyone have any info on this suggestion, and if it could be related to our issue? Thanks, Paul ---- Original Message ---- From: [EMAIL PROTECTED] To: [EMAIL PROTECTED] Subject: Re: [Dspace-tech] Error: Too many files open Date: Fri, 15 Feb 2008 17:08:01 -0500 >Paul, > >Who is the Tomcat container running as and what are their ulimits? >If >you become that user and do a 'ulimit -a' you will see a line like: > >open files (-n) 2048 > >Maybe the defaults on your system and the user that you are running >Tomcat as is too small. > >Quoting Paul Marlow <[EMAIL PROTECTED]>: > >> Hello! >> >> First of all, forgive me as I'm new to DSpace, nor am I the one who >> set up the system. >> >> That being said, we are consistently receiving errors referring to >> "Too many files open", in which we must restart the Tomcat service >> two or more times a day (which has been happening for nearly a >month >> now). Searching on our system is fine, it occurs when our users >are >> attempting to edit the metadata and/or uploading/removing >bitstreams >> - or when the system attempts to send email alerts. At this point, >> other than searching, the system is unusable until the services are >> restarted. >> >> I have done several searches in Google as well as in the Dspace >> mailing lists, but I haven't found an actual solution to the >problem, >> although I have found discussions on the topic. >> >> In relation to all the discussions I've read, I have provided most >of >> the information requested by them in a hope that others may have an >> answer, or at least speed things up a bit. >> >> DSpace version: >> 1.4.1 >> >> Operating System: >> OS Version: >> # cat /etc/redhat-release >> Red Hat Enterprise Linux AS release 4 (Nahant Update 4) >> # uname -a >> Linux <node name removed> 2.6.9-42.0.10.ELsmp #1 SMP Fri Feb 16 >> 17:17:21 EST 2007 i686 i686 i386 GNU/Linux >> >> Cron Jobs: >> dspace cron jobs >> ]# crontab -u dspace -l >> # Send out subscription emails at 1:00 everyday >> 0 1 * * * /opt/dspace/bin/sub-daily >> #Run the media filter at 2:00 every day >> 0 2 * * * /opt/dspace/bin/filter-media >> #Run the checksum checker at 3:00 >> 0 3 * * * /opt/dspace/bin/checker -lp >> #Mail the results to the sysadmin at 4:00 >> 0 4 * * * /opt/dspace/bin/dsrun >org.dspace.checker.DailyReportEmailer >> -c >> #Run stat analyses >> 0 5 * * * /opt/dspace/bin/stat-general >> 10 5 * * * /opt/dspace/bin/stat-monthly >> 20 5 * * * /opt/dspace/bin/stat-report-general >> 30 5 * * * /opt/dspace/bin/stat-report-monthly >> >> Notice that index-all is not included, as my understand is that >> filter-media includes a rebuild of the indexes. I have manually >> rebuilt the indexes which does not resolve the issue. >> >> INodes: >> # df -i >> Filesystem Inodes IUsed IFree IUse% Mounted on >> /dev/cciss/c0d0p1 524288 111084 413204 22% / >> none 214062 1 214061 1% /dev/shm >> /dev/cciss/c0d0p3 17137664 644078 16493586 4% /var >> >> files-nr Information: >> # cat /proc/sys/fs/file-nr >> 5120 0 580170 >> >> Example Error from Tomcat's catalina.out: >> Feb 15, 2008 10:28:05 AM org.apache.tomcat.util.net.PoolTcpEndpoint >> acceptSocket >> SEVERE: Endpoint >> ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=8080] ignored >> exception: java.net.SocketException: Too many open files >> java.net.SocketException: Too many open files >> at java.net.PlainSocketImpl.socketAccept(Native Method) >> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353) >> at java.net.ServerSocket.implAccept(ServerSocket.java:448) >> at java.net.ServerSocket.accept(ServerSocket.java:419) >> at >> >org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(De >f >> aultServerSocketFactory.java:61) >> at >> >org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoi >n >> t.java:408) >> at >> >org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFol >l >> owerWorkerThread.java:71) >> at >> >org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadP >o >> ol.java:685) >> at java.lang.Thread.run(Thread.java:534) >> Feb 15, 2008 10:28:05 AM org.apache.tomcat.util.net.PoolTcpEndpoint >> acceptSocket >> >> Example corresponding errors in dspace.log: >> java.io.FileNotFoundException: >> >/var/opt/dspace/assetstore/58/21/34/582134047521066341381169970824211 >6 >> 0566 (Too many open files) >> va.io.FileNotFoundException: >> >/var/opt/dspace/assetstore/58/21/34/582134047521066341381169970824211 >6 >> 0566 (Too many open files) >> at java.io.FileInputStream.open(Native Method) >> at java.io.FileInputStream.<init>(FileInputStream.java:106) >> at >> >edu.sdsc.grid.io.local.LocalFileInputStream.open(LocalFileInputStream >. >> java:171) >> at >> >edu.sdsc.grid.io.GeneralFileInputStream.<init>(GeneralFileInputStream >. >> java:145) >> at >> >edu.sdsc.grid.io.local.LocalFileInputStream.<init>(LocalFileInputStre >a >> m.java:139) >> at >> >edu.sdsc.grid.io.FileFactory.newFileInputStream(FileFactory.java:630) >> at >> >org.dspace.storage.bitstore.BitstreamStorageManager.retrieve(Bitstrea >m >> StorageManager.java:553) >> at org.dspace.content.Bitstream.retrieve(Bitstream.java:512) >> at >> >org.dspace.app.webui.servlet.BitstreamServlet.doDSGet(BitstreamServle >t >> .java:205) >> at >> >org.dspace.app.webui.servlet.DSpaceServlet.processRequest(DSpaceServl >e >> t.java:159) >> at >> >org.dspace.app.webui.servlet.DSpaceServlet.doGet(DSpaceServlet.java:1 >0 >> 0) >> at >> javax.servlet.http.HttpServlet.service(HttpServlet.java:689) >> at >> javax.servlet.http.HttpServlet.service(HttpServlet.java:802) >> at >> >org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appl >i >> cationFilterChain.java:269) >> at >> >org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationF >i >> lterChain.java:188) >> at >> >org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperV >a >> lve.java:210) >> at >> >org.apache.catalina.core.StandardContextValve.invoke(StandardContextV >a >> lve.java:174) >> at >> >org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.j >a >> va:127) >> at >> >org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.j >a >> va:117) >> at >> >org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineVal >v >> e.java:108) >> at >> >org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.jav >a >> :151) >> at >> >org.apache.coyote.http11.Http11Processor.process(Http11Processor.java >: >> 870) >> at >> >org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.p >r >> ocessConnection(Http11BaseProtocol.java:665) >> at >> >org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpo >i >> nt.java:528) >> at >> >org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFol >l >> owerWorkerThread.java:81) >> at >> >org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadP >o >> ol.java:685) >> at java.lang.Thread.run(Thread.java:534) >> 2008-02-15 10:28:06,006 WARN org.dspace.app.webui.util.UIUtil @ >> Unable to send email alert >> java.io.FileNotFoundException: >> /opt/dspace/config/emails/internal_error (Too many open files) >> at java.io.FileInputStream.open(Native Method) >> at java.io.FileInputStream.<init>(FileInputStream.java:106) >> at java.io.FileInputStream.<init>(FileInputStream.java:66) >> at java.io.FileReader.<init>(FileReader.java:41) >> at >> >org.dspace.core.ConfigurationManager.getEmail(ConfigurationManager.ja >v >> a:282) >> at >org.dspace.app.webui.util.UIUtil.sendAlert(UIUtil.java:533) >> at >> >org.dspace.app.webui.servlet.InternalErrorServlet.doGet(InternalError >S >> ervlet.java:86) >> at >> javax.servlet.http.HttpServlet.service(HttpServlet.java:689) >> at >> javax.servlet.http.HttpServlet.service(HttpServlet.java:802) >> at >> >org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Appl >i >> cationFilterChain.java:269) >> at >> >org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationF >i >> lterChain.java:188) >> at >> >org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDisp >a >> tcher.java:691) >> at >> >org.apache.catalina.core.ApplicationDispatcher.processRequest(Applica >t >> ionDispatcher.java:471) >> at >> >org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationD >i >> spatcher.java:403) >> at >> >org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDis >p >> atcher.java:301) >> at >> >org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.j >a >> va:364) >> at >> >org.apache.catalina.core.StandardHostValve.throwable(StandardHostValv >e >> .java:213) >> at >> >org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.j >a >> va:135) >> at >> >org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.j >a >> va:117) >> at >> >org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineVal >v >> e.java:108) >> at >> >org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.jav >a >> :151) >> at >> >org.apache.coyote.http11.Http11Processor.process(Http11Processor.java >: >> 870) >> at >> >org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.p >r >> ocessConnection(Http11BaseProtocol.java:665) >> at >> >org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpo >i >> nt.java:528) >> at >> >org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFol >l >> owerWorkerThread.java:81) >> at >> >org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadP >o >> ol.java:685) >> at java.lang.Thread.run(Thread.java:534) >> >> >> Any assistance would be greatly appreciated. >> >> Thanks! >> >> Paul Marlow >> >> >> >> >--------------------------------------------------------------------- >---- >> This SF.net email is sponsored by: Microsoft >> Defy all challenges. Microsoft(R) Visual Studio 2008. >> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ >> _______________________________________________ >> DSpace-tech mailing list >> DSpace-tech@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/dspace-tech >> > > > ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
