RE: indexer threading?

2010-04-27 Thread Wawok, Brian
- DocumentsWriter.getThreadState - SolrIndexWriter.ensureOpen I don't know the guts of Solr/Lucene in such details so can't make any conclusions. Are there any configuration techniques to improve indexing performance in multiple threads scenario? Alex On Mon, Apr 26, 2010 at 6:52 PM, Wawok, Brian wrote: >

indexer threading?

2010-04-26 Thread Wawok, Brian
Hi, I was wondering about how the multi-threading of the indexer works? I am using SolrJ to stream documents to a server. As I add more threads on the client side, I slowly see both speed and CPU usage go up on the indexer side. Once I hit about 4 threads, my indexer is at 100% cpu usage (of 1

RE: solr best practice to submit many documents

2010-04-08 Thread Wawok, Brian
TTP. On Wed, Apr 7, 2010 at 9:18 AM, Wawok, Brian wrote: > I don't think I want to stream from Java, text munging in Java is a PITA. > Would rather stream from a script, so need a more general solution. > > The Streaming document interface looks interesting, let me see if I can

RE: solr best practice to submit many documents

2010-04-07 Thread Wawok, Brian
index a large number of documents. Paolo Wawok, Brian wrote: > Hello, > > I am using SOLR for some proof of concept work, and was wondering if anyone > has some guidance on a best practice. > > Background: > Nightly get a delivery of a few 1000 reports. Each report is betw

solr best practice to submit many documents

2010-04-07 Thread Wawok, Brian
Hello, I am using SOLR for some proof of concept work, and was wondering if anyone has some guidance on a best practice. Background: Nightly get a delivery of a few 1000 reports. Each report is between 1 and 500,000 pages. For my proof of concept I am using a single 100,000 page report. I want