I am updating status of the documents frequently. There will be huge number of deletes. I do optimize the index once in a day.
I want to know the usage for setTermInfosIndexDivisor. Directory dir = FSDirectory.getDirectory(indexPath); IndexReader reader = IndexReader.open(dir, true); reader.setTermInfosIndexDivisor(5); I reopen the IndexReader whenever there is any document added to Index. Do i need to set setTermInfosIndexDivisor(5); during re-opening of the index also. I tried this, first time it accepted and second time onwards it throws "terms already loaded" expection. >Loaded terms might not dominate your memory consumption in side >lucene. Again, you should provide more information of indexing, the >environment and the situation where the error occurs. I do indexing with no norms with all default values. As per the documentation, it should subsample the terms loaded in to memory. Regards Ganesh ----- Original Message ----- From: "Simon Willnauer" <simon.willna...@googlemail.com> To: <java-user@lucene.apache.org> Sent: Thursday, June 25, 2009 2:10 PM Subject: Re: setTermInfosIndexDivisor Hey there, On Thu, Jun 25, 2009 at 9:10 AM, Ganesh<emailg...@yahoo.co.in> wrote: > Hello all, > > I am using Lucene v2.4.1 > > 1) > I have build multiple indexes of total 30 million documents. My memory limit > is 512 MB. In this case i am getting frequently OOME. If i increased the > memory limit to 1 GB / 1.5 GB then it is working fine. My point is it will > also will get exhausted when it reaches 60 / 90 million documents. > > I thought to use setTermInfosIndexDivisor, but even then the memory > consumption is same. This parameter has no effect. Whether this parameter > should be set while building index? I build the index using default value. > After hitting OOME i am setting this. I would be curious what you do to your index. do you have a lot of pending deletes? do you call optimize frequently? In which situations do you hit the OOM? > > Directory dir = FSDirectory.getDirectory(indexPath); > IndexReader reader = IndexReader.open(dir, true); > reader.setTermInfosIndexDivisor(5); Loaded terms might not dominate your memory consumption in side lucene. Again, you should provide more information of indexing, the environment and the situation where the error occurs. simon > > 2) > IndexWriter.setTermIndexInterval should be set while creating the index? If i > build the index with default value, After some time if i use this parameter, > Whether there will be some effect? > > Regards > Ganesh > > Send instant messages to your online friends http://in.messenger.yahoo.com > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > > --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org Send instant messages to your online friends http://in.messenger.yahoo.com --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
