OutOfMemoryError on search in large, simple index
-------------------------------------------------
Key: LUCENE-1053
URL: https://issues.apache.org/jira/browse/LUCENE-1053
Project: Lucene - Java
Issue Type: Bug
Components: Search
Affects Versions: 2.0.0
Environment: Red Hat Enterprise Linux ES release 3 (Taroon Update 9)
Linux sb-test-acs-001 2.4.21-47.0.1.ELsmp #1 SMP Fri Oct 13 17:56:20 EDT 2006
i686 i686 i386 GNU/Linux
2 GB RAM
Java version 1.5.0_13
Reporter: Lars Clausen
We get OutOfMemoryError when performing a one-term search in our index. The
search, if completed, should give only a few thousand hits, but from inspecting
a heap dump it appears that many more documents in the index get stored in
Lucene during the search. Our index consists of eight fields per document,
fairly regularly sized, the total index size is 170GB, spread over about 400
million documents (425 bytes per document). The search is a simple TermQuery,
the search term a trivial string, the code in question looks like this (cut
together for conciseness):
public static final String FIELD_URL = "url";
...
luceneSearcher = new IndexSearcher(indexDir.getAbsolutePath());
Query query = new TermQuery(new Term(DigestIndexer.FIELD_URL, uri));
try {
Hits hits = luceneSearcher.search(query);
Stack trace:
Oct 11, 2007 4:02:19 PM org.slf4j.impl.JCLLoggerAdapter error
SEVERE: EXCEPTION
java.lang.OutOfMemoryError: Java heap space
at
org.apache.lucene.index.SegmentReader.getNorms(SegmentReader.java:384)
at org.apache.lucene.index.SegmentReader.norms(SegmentReader.java:393)
at
org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:68)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:129)
at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:99)
at org.apache.lucene.search.Hits.getMoreDocs(Hits.java:65)
at org.apache.lucene.search.Hits.(Hits.java:44)
at org.apache.lucene.search.Searcher.search(Searcher.java:44)
at org.apache.lucene.search.Searcher.search(Searcher.java:36)
at
dk.netarkivet.common.distribute.arcrepository.ARCLookup.luceneLookup(ARCLookup.java:166)
at
dk.netarkivet.common.distribute.arcrepository.ARCLookup.lookup(ARCLookup.java:130)
at
dk.netarkivet.viewerproxy.ARCArchiveAccess.lookup(ARCArchiveAccess.java:126)
at
dk.netarkivet.viewerproxy.NotifyingURIResolver.lookup(NotifyingURIResolver.java:72)
at
dk.netarkivet.viewerproxy.CommandResolver.lookup(CommandResolver.java:80)
at
dk.netarkivet.viewerproxy.CommandResolver.lookup(CommandResolver.java:80)
at
dk.netarkivet.viewerproxy.CommandResolver.lookup(CommandResolver.java:80)
at dk.netarkivet.viewerproxy.WebProxy.handle(WebProxy.java:129)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
at org.mortbay.jetty.Server.handle(Server.java:285)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:457)
at
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:751)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:500)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:209)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:357)
at
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:217)
at
org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
