Hi Doug, I have experimented with a mergeFactor of 5 or 10 (default) but it didn't help matters once I reached the ulimit. I understand how the mergeFactor affects Lucene's performance.
I am actually not doing any searches with IndexReader right now, just indexing. Yes, I do store and reuse the IndexReader per index, but my test right now is purely indexing. Setting my ulimit to 1M seems to do the trick for now, but I'm sure I could do better by tweaking Lucene's API. I will continue to investigate. Thanks, Nick. Doug Cutting wrote: > Are you changing the default mergeFactor or other settings? If so, > how? Large mergeFactors are generally a bad idea: they don't make > things faster in the long run and they chew up file handles. > > Are all searches reusing a single IndexReader? They should. This is > the other most common reason folks run out of file handles: they open > too many IndexReaders. The exception may be thrown when merging, but > the root cause might be something else. > > Doug > > Nick Atkins wrote: >> Hi, >> >> What's the best way to manage the number of open files used by Lucene >> when it's running under Tomcat? I have a indexing application running >> as a web app and I index a huge number of mail messages (upwards of >> 40000 in some cases). Lucene's merging routine always craps out >> eventually with the "too many open files" regardless of how large I set >> ulimit to. lsof tells me they are all "deleted" but they still seem to >> count as open files. I don't want to set ulimit to some enormous value >> just to solve this (because it will never be large enough). What's the >> best strategy here? >> >> I have tried setting various parameters on the IndexWriter such as the >> MergeFactor, MaxMergeDocs and MaxBufferedDocs but they seem to only >> affect the merge timing algorithm wrt memory usage. The number of files >> used seems to be unaffected by anything I can set on the IndexWriter. >> >> Any hints much appreciated. >> >> Cheers, >> >> Nick. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [EMAIL PROTECTED] >> For additional commands, e-mail: [EMAIL PROTECTED] >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
