Different ramBufferSizeMB during indexing should never cause corruption! Can you try setting the ram buffer to 256 MB in your test env and see if that makes the corruption go away?
This could also be a hardware issue in your test env. If you run CheckIndex on the corrupt index does it always fail in the same way? Mike On Fri, Jan 14, 2011 at 6:43 AM, Li Li <[email protected]> wrote: > hi all, > we have confronted this problem 3 times when testing > The exception stack is > Exception in thread "Lucene Merge Thread #2" > org.apache.lucene.index.MergePolicy$MergeException: > org.apache.lucene.index.CorruptIndexException: docs out of order (7286 > <= 7286 ) > at > org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:355) > at > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:319) > Caused by: org.apache.lucene.index.CorruptIndexException: docs out of > order (7286 <= 7286 ) > at > org.apache.lucene.index.FormatPostingsDocsWriter.addDoc(FormatPostingsDocsWriter.java:75) > at > org.apache.lucene.index.SegmentMerger.appendPostings(SegmentMerger.java:880) > at > org.apache.lucene.index.SegmentMerger.mergeTermInfos(SegmentMerger.java:818) > at > org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:756) > at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:187) > at > org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5354) > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4937) > > Or > Exception in thread "Lucene Merge Thread #0" > org.apache.lucene.index.MergePolicy$MergeException: > java.lang.ArrayIndexOutOfBoundsException: 330 > at > org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:355) > at > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:319) > Caused by: java.lang.ArrayIndexOutOfBoundsException: 330 > at org.apache.lucene.util.BitVector.get(BitVector.java:102) > at > org.apache.lucene.index.SegmentTermDocs.next(SegmentTermDocs.java:238) > at > org.apache.lucene.index.SegmentTermDocs.next(SegmentTermDocs.java:168) > at > org.apache.lucene.index.SegmentTermPositions.next(SegmentTermPositions.java:98) > at > org.apache.lucene.index.SegmentMerger.appendPostings(SegmentMerger.java:870) > at > org.apache.lucene.index.SegmentMerger.mergeTermInfos(SegmentMerger.java:818) > at > org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:756) > at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:187) > at > org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5354) > > > We did some minor modification based on lucene 2.9.1 and solr > 1.4.0. we modified frq file to store 4 bytes for the positions of the > term occured > in these document(Accessing full postions in prx is time consuming > that can't meed our needs). I can't tell it's our bug or lucene's own > bug. > I searched the mail list and found the mail "problem during index > merge" posted in 2010-10-21. It's similar to our case. > It seems the docList in frq file is wrongly stored. When Merging, > when it's decoded, the wrong docID many larger than maxDocs(BitVector > deletedDocs) > which cause the second exception. Or docID delta is less than 0(it > reads wrongly) which cause the first exception > we are still continueing testing to turn off our modification and > open infoStream in solr-config.xml > > We found a strange phenomenon. when we test, it sometimes hited > exceptions but in our production environment, it never hit any. > the hardware and software environments are the same. We checked > carefully and find the only difference is this line in solr-config.xml > <ramBufferSizeMB>32</ramBufferSizeMB> in testing environment > <ramBufferSizeMB>256</ramBufferSizeMB>in production environment > The indexed documents number for each machine is also roughly the > same. 10M+ documents. > I can't make sure the indice in production env are correct because > even there are some terms' docList are wrong, if the doc delta >0 and > don't have > some deleted documents, it will not hit the 2 exceptions. > The search results in production env and we don't find any strange results. > > Will when the ramBufferSizeMB is too small results in index corruption? > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
