RE: Batch Indexing - best practice?

2010-03-17 Thread Murdoch, Paul
ks again, Paul -Original Message- From: java-user-return-45439-paul.b.murdoch=saic@lucene.apache.org [mailto:java-user-return-45439-paul.b.murdoch=saic@lucene.apache.org ] On Behalf Of Erick Erickson Sent: Monday, March 15, 2010 12:45 PM To: java-user@lucene.apache.org Subject: Re: Batch

Re: Batch Indexing - best practice?

2010-03-15 Thread Erick Erickson
-Original Message- > From: java-user-return-45433-paul.b.murdoch=saic@lucene.apache.org > [mailto:java-user-return-45433-paul.b.murdoch=saic@lucene.apache.org > ] On Behalf Of Mark Miller > Sent: Monday, March 15, 2010 10:48 AM > To: java-user@lucene.apache.org > Subject:

Re: Batch Indexing - best practice?

2010-03-15 Thread Mark Miller
:48 AM To: java-user@lucene.apache.org Subject: Re: Batch Indexing - best practice? On 03/15/2010 10:41 AM, Murdoch, Paul wrote: Hi, I'm using Lucene 2.9.2. Currently, when creating my index, I'm calling indexWriter.addDocument(doc) for each Document I want to ind

RE: Batch Indexing - best practice?

2010-03-15 Thread Murdoch, Paul
:java-user-return-45433-paul.b.murdoch=saic@lucene.apache.org ] On Behalf Of Mark Miller Sent: Monday, March 15, 2010 10:48 AM To: java-user@lucene.apache.org Subject: Re: Batch Indexing - best practice? On 03/15/2010 10:41 AM, Murdoch, Paul wrote: > Hi, > > > > I'm using

Re: Batch Indexing - best practice?

2010-03-15 Thread Ian Lea
See http://wiki.apache.org/lucene-java/ImproveIndexingSpeed for plenty of tips. Suggested by Mike just a few hours ago in another thread ... -- Ian. On Mon, Mar 15, 2010 at 2:41 PM, Murdoch, Paul wrote: > Hi, > > > > I'm using Lucene 2.9.2.  Currently, when creating my index, I'm calling > ind

Re: Batch Indexing - best practice?

2010-03-15 Thread Mark Miller
On 03/15/2010 10:41 AM, Murdoch, Paul wrote: Hi, I'm using Lucene 2.9.2. Currently, when creating my index, I'm calling indexWriter.addDocument(doc) for each Document I want to index. The Documents aren't large and I'm averaging indexing about 500 documents every 90 seconds. I'd like to try

Re: batch indexing

2007-05-02 Thread Erick Erickson
For some intermediate period of time when indexing, the document are buffered in RAM. There is a complex interplay between several of the parameters to an IndexWriter that govern how many documents are indexed in RAM before being flushed to the FSDirectory. Of course, if you specify a RAMdirector

Re: batch indexing

2007-05-02 Thread Chris
Sorry , ask a question. You say FSDirectory is RADdirectory as least until it flushes. I cannot understand your means . May you please teach me what it means? FSDirectory stored in filesystem , and RADdirectory stored in RAM. MergeFactor and MaxBufferedDocs settings are limited and controlled

Re: batch indexing

2007-04-29 Thread Erick Erickson
MAIL PROTECTED]> wrote: Thanks Erik , so FSDirectory seems better option than RAMDirectory ? Also I think O.S can cache files in which case FSDirectory may not be bad , your thoughts ? -Original Message- From: Erick Erickson [mailto:[EMAIL PROTECTED] Sent: Sunday, April 29, 2007 7:07

RE: batch indexing

2007-04-29 Thread Chandan Tamrakar
@lucene.apache.org Subject: Re: batch indexing As I understand it, FSDirectory *is* RAMdirectory, at least until it flushes. There have been several discussions of this, search the mail archive for things like MergeFactor, MaxBufferedDocs and the like. You'll find quite a bit of information about how

Re: batch indexing

2007-04-29 Thread Erick Erickson
As I understand it, FSDirectory *is* RAMdirectory, at least until it flushes. There have been several discussions of this, search the mail archive for things like MergeFactor, MaxBufferedDocs and the like. You'll find quite a bit of information about how these parameters interact. Particularly, s

Re: batch indexing using RAMDirectory

2006-06-28 Thread James Pine
Hey Eric, I think you want: fsWriter.addIndexes(Directory[] {ramDir}); to be: fsWriter.addIndexes(new Directory[]{ramDir}); JAMES --- zheng <[EMAIL PROTECTED]> wrote: > I am a novice in lucene. I write some code to do > batch indexing using > RAMDirectory according to the code provided in >