On Tue, Mar 18, 2008 at 7:38 AM, Ian Lea <[EMAIL PROTECTED]> wrote: > Hi > > > When bulk loading into a new index I'm seeing this exception > > Exception in thread "Thread-1" > org.apache.lucene.index.MergePolicy$MergeException: > org.apache.lucene.index.CorruptIndexException: doc counts differ for > segment _4l: fieldsReader shows 67861 but segmentInfo shows 67862 > at > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:271) > Caused by: org.apache.lucene.index.CorruptIndexException: doc counts > differ for segment _4l: fieldsReader shows 67861 but segmentInfo shows > 67862 > at > org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:313) > at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:262) > at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:221) > at > org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3093) > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2834) > at > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:240) > > when use java version 1.6.0_05-b13 or 1.6.0_04-b12 on linux, with > lucene 2.3.0 or 2.3.1 or lucene-core-2.3-SNAPSHOT from yesterday. > > With java version 1.6.0_03-b05 things work fine. > > The exception happens a few hundred thousand documents into the load. > > A different program updating a different index with different data on > a different server gave a similar error on version 1.6.0_05-b13 and > lucene 2.3.0. > > Any ideas? Is this maybe a known issue or am I missing something obvious?
My guess is perhaps a thread safety bug, more likely in Lucene indexing code (less likely in the JVM or specific libc). What Linux version are you using? What hardware are you running on (specifically, the CPU)? If possible, it would be great if you could check out Lucene trunk, crank up the iterations by modifying the TestStressIndexing2 and maybe fiddle with some of the other parameters in TestStressIndexing2.testMultiConfig(), and see if you can get it to fail. -Yonik --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
