Documents are biblio records. All have title, author etc. stored, some have a few extra fields as well. Typically around 25 fields per doc. The index is created with compound format, everything else as default.
I've rerun the job until failure. Different numbers this time, but basically the same exception. In the failing pass it was trying to load 100K docs. Exception in thread "Thread-1" org.apache.lucene.index.MergePolicy$MergeException: org.apache.lucene.index.CorruptIndexException: doc counts differ for segment _cb: fieldsReader shows 71663 but segmentInfo shows 71664 at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:271) Caused by: org.apache.lucene.index.CorruptIndexException: doc counts differ for segment _cb: fieldsReader shows 71663 but segmentInfo shows 71664 at org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:313) at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:262) at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:221) at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3099) at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2834) at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:240) As mentioned, the index is loaded in chunks - the infostream from the failing pass is attached. All infostreams from all 19 odd runs leading up to the failure available as well if that would help. Running with -ea doesn't seem to have made any difference. -- Ian. On Tue, Mar 18, 2008 at 12:09 PM, Michael McCandless <[EMAIL PROTECTED]> wrote: > > Can you call IndexWriter.setInfoStream(...) and get the error to > happen and post back the resulting output? And, turn on assertions > (java -ea) since that may catch the issue sooner. > > Can you describe you are setting up IndexWriter (autoCommit, > compound, etc.), and what your documents are like? Do your documents > have a fixed schema (same fields every time), or it varies such that > some documents have no stored fields and some do? > > Mike > > > > Ian Lea wrote: > > > Hi > > > > > > When bulk loading into a new index I'm seeing this exception > > > > Exception in thread "Thread-1" > > org.apache.lucene.index.MergePolicy$MergeException: > > org.apache.lucene.index.CorruptIndexException: doc counts differ for > > segment _4l: fieldsReader shows 67861 but segmentInfo shows 67862 > > at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run > > (ConcurrentMergeScheduler.java:271) > > Caused by: org.apache.lucene.index.CorruptIndexException: doc counts > > differ for segment _4l: fieldsReader shows 67861 but segmentInfo shows > > 67862 > > at org.apache.lucene.index.SegmentReader.initialize > > (SegmentReader.java:313) > > at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:262) > > at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:221) > > at org.apache.lucene.index.IndexWriter.mergeMiddle > > (IndexWriter.java:3093) > > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2834) > > at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run > > (ConcurrentMergeScheduler.java:240) > > > > when use java version 1.6.0_05-b13 or 1.6.0_04-b12 on linux, with > > lucene 2.3.0 or 2.3.1 or lucene-core-2.3-SNAPSHOT from yesterday. > > > > With java version 1.6.0_03-b05 things work fine. > > > > The exception happens a few hundred thousand documents into the load. > > > > A different program updating a different index with different data on > > a different server gave a similar error on version 1.6.0_05-b13 and > > lucene 2.3.0. > > > > > > Any ideas? Is this maybe a known issue or am I missing something > > obvious? > > > > > > > > -- > > Ian. > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > >
--------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]