The data is loaded in chunks of up to 100K docs in separate runs of the program if that helps answer the first question. All buffers have default values, docs are small but not tiny, JVM is running with default settings.
Answers to previous questions, and infostream, will follow once the job has failed again. -- Ian. On Tue, Mar 18, 2008 at 1:24 PM, Michael McCandless <[EMAIL PROTECTED]> wrote: > > One question: do you know whether 67,861 docs "feels like" a newly > flushed segment, or, the result of a merge? > > Ie, roughly how many docs are you buffering in IndexWriter before it > flushes? Are they very small documents and your RAM buffer is large? > > Mike > > > > Ian Lea wrote: > > > Hi > > > > > > When bulk loading into a new index I'm seeing this exception > > > > Exception in thread "Thread-1" > > org.apache.lucene.index.MergePolicy$MergeException: > > org.apache.lucene.index.CorruptIndexException: doc counts differ for > > segment _4l: fieldsReader shows 67861 but segmentInfo shows 67862 > > at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run > > (ConcurrentMergeScheduler.java:271) > > Caused by: org.apache.lucene.index.CorruptIndexException: doc counts > > differ for segment _4l: fieldsReader shows 67861 but segmentInfo shows > > 67862 > > at org.apache.lucene.index.SegmentReader.initialize > > (SegmentReader.java:313) > > at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:262) > > at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:221) > > at org.apache.lucene.index.IndexWriter.mergeMiddle > > (IndexWriter.java:3093) > > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2834) > > at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run > > (ConcurrentMergeScheduler.java:240) > > > > when use java version 1.6.0_05-b13 or 1.6.0_04-b12 on linux, with > > lucene 2.3.0 or 2.3.1 or lucene-core-2.3-SNAPSHOT from yesterday. > > > > With java version 1.6.0_03-b05 things work fine. > > > > The exception happens a few hundred thousand documents into the load. > > > > A different program updating a different index with different data on > > a different server gave a similar error on version 1.6.0_05-b13 and > > lucene 2.3.0. > > > > > > Any ideas? Is this maybe a known issue or am I missing something > > obvious? > > > > > > > > -- > > Ian. > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
