On upgrading to 2.3, it's best to flush by RAM
(writer.setRAMBufferSizeMB) instead of document count.
Generally, the more RAM the better, to a point. Though you should
also be sure not to use so much RAM that your JVM must GC too often
or hits OOM error, or your machine starts swapping.
Mike
vivek sar wrote:
my maxBufferedDocs is 1000, do you recommend bigger than that? What's
a good number for this for a very high indexing rate (10K new
documents every min).
On Jan 19, 2008 10:30 PM, Otis Gospodnetic
<[EMAIL PROTECTED]> wrote:
In addition to what Mike already said:
maxMergeDocs=99999 -- do you really mean maxMergeDocs and not
maxBufferedDocs?
Larg(er) maxBufferedDocs will speed up indexing.
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
----- Original Message ----
From: vivek sar <[EMAIL PROTECTED]>
To: java-user@lucene.apache.org
Sent: Friday, January 18, 2008 4:31:26 AM
Subject: Optimize for large index size
Hi,
We are using Lucene 2.2. We have an index of size 70G (within 3-4
days) and growing. We run optimize pretty frequently (once every hour
- due to large number of index updates every min - can be up to 100K
new documents every min). I have seen every now and then the optimize
takes 3-4 hours to complete and up to 8 G memory (our limit). This
makes the whole system slow. Few questions,
1) Is there any alternative to optimize? That is, can we do without
optimize and still have our search fast?
2) What's the best way to use optimize, i.e. how can we make the
optimize much faster and use lesser memory?
3) Is there a way to partition the indexes using Lucene? Let's say we
partition daily, so we have to optimize only the daily indexes and
not
the whole thing.
Our mergefactor=200 and maxMergeDocs=99999
Thanks,
-vivek
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
