Hi,

why are you doing this? Lucene's IndexWriter can handle addDocuments in 
multiple threads. And, since Lucene 4, it will process them almost completely 
parallel!
If you do the addDocuments single-threaded you are adding an additional 
bottleneck in your application. If you are doing a synchronization on 
IndexWriter (which I hope you will not do), things will go wrong, too.

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: u...@thetaphi.de


> -----Original Message-----
> From: Igor Shalyminov [mailto:ishalymi...@yandex-team.ru]
> Sent: Thursday, November 21, 2013 4:45 PM
> To: java-user@lucene.apache.org
> Subject: Lucene multithreaded indexing problems
> 
> Hello!
> 
> I tried to perform indexing multithreadedly, with a FixedThreadPool of
> Callable workers.
> The main operation - parsing a single document and addDocument() to the
> index - is done by a single worker.
> After parsing a document, a lot (really a lot) of Strings appears, and at the
> end of the worker's call() all of them goes to the indexWriter.
> I use no merging, the resourses are flushed on disk when the segment size
> limit is reached.
> 
> The problem is, after a little while (when the most of the heap memory is
> used) indexer makes no progress, and CPU load is constant 100% (no
> difference if there are 2 threads or 32). So I think at some point garbage
> collection takes the whole indexing process down.
> 
> Could you please give some advices on the proper concurrent indexing with
> Lucene?
> Can there be "memory leaks" somewhere in the indexWriter? Maybe I must
> perform some operations with writer to release unused resourses from time
> to time?
> 
> 
> --
> Best Regards,
> Igor
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
> For additional commands, e-mail: java-user-h...@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org

Reply via email to