Hi, why are you doing this? Lucene's IndexWriter can handle addDocuments in multiple threads. And, since Lucene 4, it will process them almost completely parallel! If you do the addDocuments single-threaded you are adding an additional bottleneck in your application. If you are doing a synchronization on IndexWriter (which I hope you will not do), things will go wrong, too.
Uwe ----- Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: u...@thetaphi.de > -----Original Message----- > From: Igor Shalyminov [mailto:ishalymi...@yandex-team.ru] > Sent: Thursday, November 21, 2013 4:45 PM > To: java-user@lucene.apache.org > Subject: Lucene multithreaded indexing problems > > Hello! > > I tried to perform indexing multithreadedly, with a FixedThreadPool of > Callable workers. > The main operation - parsing a single document and addDocument() to the > index - is done by a single worker. > After parsing a document, a lot (really a lot) of Strings appears, and at the > end of the worker's call() all of them goes to the indexWriter. > I use no merging, the resourses are flushed on disk when the segment size > limit is reached. > > The problem is, after a little while (when the most of the heap memory is > used) indexer makes no progress, and CPU load is constant 100% (no > difference if there are 2 threads or 32). So I think at some point garbage > collection takes the whole indexing process down. > > Could you please give some advices on the proper concurrent indexing with > Lucene? > Can there be "memory leaks" somewhere in the indexWriter? Maybe I must > perform some operations with writer to release unused resourses from time > to time? > > > -- > Best Regards, > Igor > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org