I have a collection of indices with a total of about 7,000,000 documents between them all. When I attempt to run a search over these indices, the searching process's memory usage increases to ~1.7GB if I allow java to use that much memory. If I don't (my normal memory cap is 512MB), I get the following exception:
Exception in thread "Thread-2" java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOfRange(Arrays.java:3209) at java.lang.String.<init>(String.java:216) at org.apache.lucene.index.TermBuffer.toTerm(TermBuffer.java:104) at org.apache.lucene.index.SegmentTermEnum.term(SegmentTermEnum.java:159) at org.apache.lucene.index.TermInfosReader.ensureIndexIsRead(TermInfosReader.java:119) at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:157) at org.apache.lucene.index.SegmentReader.docFreq(SegmentReader.java:419) at org.apache.lucene.search.IndexSearcher.docFreq(IndexSearcher.java:87) at org.apache.lucene.search.Searcher.docFreqs(Searcher.java:178) at org.apache.lucene.search.MultiSearcher.createWeight(MultiSearcher.java:311) at org.apache.lucene.search.Searcher.search(Searcher.java:118) at org.apache.lucene.search.Searcher.search(Searcher.java:97) at SearchThread.run(SearchThread.java:54) So, it looks like simply attempting to read the .tii files from the indices is taking huge amounts of RAM. This is only happening on one machine; other machines with similar data run just with 256-512MB memory restrictions, so I'm trying to figure out what could cause the .tii files to become so bloated. Is there anything I can do to fix these indices? Searching is also very slow on this machine; many machines with tens of millions of documents can do searches with subsecond responses, whereas this machine takes many seconds to call its HitCollector's collect function for the first time. Any suggestions about how to slim down the .tii files on this machine (or any workarounds) would be much appreciated. --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]