Hi,
We ran into the same issue (corrupted index) using Lucene 2.4.0.
There was no outage or system reboot - not sure how could it get
corrupted. Here is the exception,
Caused by: java.io.IOException: background merge hit exception:
_io5:c66777491 _nh9:c10656736 _taq:c2021563 _s8m:c1421051
_uh5:c2065961 _r0y:c1124653 _s4s:c2477731 _t6w:c4340938 _ucx:c8018451
_xkb:c13842776 _xkd:c3394 _xke:c3379 _xkg:c1231 _xkh:c1252 _xkj:c1680
_xkk:c1689 into _xkl [optimize]
at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2258)
at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2194)
at
com.packetmotion.manager.query.fulltext.index.Indexer.optimizeMasterIndex(Indexer.java:887)
at
com.packetmotion.manager.query.fulltext.index.Indexer.createNewIndexPartition(Indexer.java:783)
at
com.packetmotion.manager.query.fulltext.index.Indexer.indexByDevice(Indexer.java:582)
at
com.packetmotion.manager.query.fulltext.index.Indexer.indexData(Indexer.java:440)
at
com.packetmotion.manager.query.fulltext.index.Indexer$$FastClassByCGLIB$$97fb7e9b.invoke(<generated>)
at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:149)
at
org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:628)
at
com.packetmotion.manager.query.fulltext.index.Indexer$$EnhancerByCGLIB$$ebbe3914.indexData(<generated>)
at
com.packetmotion.manager.query.fulltext.index.IndexerJob$1.run(IndexerJob.java:38)
... 8 more
Caused by: java.lang.IndexOutOfBoundsException: Index: 51, Size: 26
at java.util.ArrayList.RangeCheck(ArrayList.java:547)
at java.util.ArrayList.get(ArrayList.java:322)
at org.apache.lucene.index.FieldInfos.fieldInfo(FieldInfos.java:265)
at org.apache.lucene.index.FieldsReader.doc(FieldsReader.java:185)
at
org.apache.lucene.index.SegmentReader.document(SegmentReader.java:729)
at
org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:359)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:139)
at
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4226)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3877)
at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205)
at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260)
I ran the checkIndex tool to fix the corrupted index. Here is its output,
Opening index @ /opt/ps_manager/apps/conf/index/MasterIndex
Segments file=segments_1587 numSegments=18 version=FORMAT_HAS_PROX [Lucene 2.4]
1 of 18: name=_io5 docCount=66777491
compound=true
hasProx=true
numFiles=1
size (MB)=6,680.574
no deletions
test: open reader.........OK
test: fields, norms.......OK [30 fields]
test: terms, freq, prox...OK [368415 terms; 1761057989 terms/docs
pairs; 2095636359 tokens]
test: stored fields.......OK [66777491 total field count; avg 1
fields per doc]
test: term vectors........OK [0 total vector count; avg 0
term/freq vector fields per doc]
2 of 18: name=_nh9 docCount=10656736
compound=true
hasProx=true
numFiles=1
size (MB)=1,058.869
no deletions
test: open reader.........OK
test: fields, norms.......OK [30 fields]
test: terms, freq, prox...OK [118964 terms; 278176718 terms/docs
pairs; 329825350 tokens]
test: stored fields.......OK [10656736 total field count; avg 1
fields per doc]
test: term vectors........OK [0 total vector count; avg 0
term/freq vector fields per doc]
3 of 18: name=_taq docCount=2021563
compound=true
hasProx=true
numFiles=1
size (MB)=208.544
no deletions
test: open reader.........OK
test: fields, norms.......OK [30 fields]
test: terms, freq, prox...OK [145494 terms; 54131697 terms/docs
pairs; 65411701 tokens]
test: stored fields.......OK [2021563 total field count; avg 1
fields per doc]
test: term vectors........OK [0 total vector count; avg 0
term/freq vector fields per doc]
4 of 18: name=_s8m docCount=1421051
compound=true
hasProx=true
numFiles=1
size (MB)=162.443
no deletions
test: open reader.........OK
test: fields, norms.......OK [30 fields]
test: terms, freq, prox...OK [210276 terms; 42491363 terms/docs
pairs; 53054214 tokens]
test: stored fields.......OK [1421051 total field count; avg 1
fields per doc]
test: term vectors........OK [0 total vector count; avg 0
term/freq vector fields per doc]
5 of 18: name=_uh5 docCount=2065961
compound=true
hasProx=true
numFiles=1
size (MB)=229.394
no deletions
test: open reader.........OK
test: fields, norms.......OK [30 fields]
test: terms, freq, prox...OK [126789 terms; 60416663 terms/docs
pairs; 75120265 tokens]
test: stored fields.......OK [2065964 total field count; avg 1
fields per doc]
test: term vectors........OK [0 total vector count; avg 0
term/freq vector fields per doc]
6 of 18: name=_r0y docCount=1124653
compound=true
hasProx=true
numFiles=1
size (MB)=115.792
no deletions
test: open reader.........OK
test: fields, norms.......OK [26 fields]
test: terms, freq, prox...FAILED
WARNING: fixIndex() would remove reference to this segment; full exception:
java.lang.RuntimeException: term desthost:wir docFreq=6 != num docs
seen 0 + num docs deleted 0
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:475)
at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:676)
7 of 18: name=_s4s docCount=2477731
compound=true
hasProx=true
numFiles=1
size (MB)=230.698
no deletions
test: open reader.........PuTTYPuTTYOK
test: fields, norms.......OK [25 fields]
test: terms, freq, prox...OK [90327 terms; 60212096 terms/docs
pairs; 72740383 tokens]
test: stored fields.......OK [2477731 total field count; avg 1
fields per doc]
test: term vectors........OK [0 total vector count; avg 0
term/freq vector fields per doc]
8 of 18: name=_t6w docCount=4340938
compound=true
hasProx=true
numFiles=1
size (MB)=451.389
no deletions
test: open reader.........OK
test: fields, norms.......OK [26 fields]
test: terms, freq, prox...OK [157817 terms; 121753990 terms/docs
pairs; 151141568 tokens]
test: stored fields.......OK [4340938 total field count; avg 1
fields per doc]
test: term vectors........OK [0 total vector count; avg 0
term/freq vector fields per doc]
9 of 18: name=_ucx docCount=8018451
compound=true
hasProx=true
numFiles=1
size (MB)=845.968
no deletions
test: open reader.........OK
test: fields, norms.......OK [26 fields]
test: terms, freq, prox...OK [410057 terms; 227617455 terms/docs
pairs; 283398975 tokens]
test: stored fields.......OK [8018451 total field count; avg 1
fields per doc]
test: term vectors........OK [0 total vector count; avg 0
term/freq vector fields per doc]
10 of 18: name=_xl9 docCount=13891933
compound=true
hasProx=true
numFiles=1
size (MB)=1,408.881
no deletions
test: open reader.........OK
test: fields, norms.......OK [26 fields]
test: terms, freq, prox...OK [535226 terms; 376394493 terms/docs
pairs; 465003060 tokens]
test: stored fields.......OK [13891933 total field count; avg 1
fields per doc]
test: term vectors........OK [0 total vector count; avg 0
term/freq vector fields per doc]
11 of 18: name=_xlb docCount=3521
compound=true
hasProx=true
numFiles=1
size (MB)=0.411
no deletions
test: open reader.........OK
test: fields, norms.......OK [25 fields]
test: terms, freq, prox...OK [2070 terms; 108496 terms/docs pairs;
136365 tokens]
test: stored fields.......OK [3521 total field count; avg 1 fields per doc]
test: term vectors........OK [0 total vector count; avg 0
term/freq vector fields per doc]
12 of 18: name=_xlc docCount=3529
compound=true
hasProx=true
numFiles=1
size (MB)=0.412
no deletions
test: open reader.........OK
test: fields, norms.......OK [22 fields]
test: terms, freq, prox...OK [2214 terms; 108633 terms/docs pairs;
136384 tokens]
test: stored fields.......OK [3529 total field count; avg 1 fields per doc]
test: term vectors........OK [0 total vector count; avg 0
term/freq vector fields per doc]
13 of 18: name=_xle docCount=1401
compound=true
hasProx=true
numFiles=1
size (MB)=0.188
no deletions
test: open reader.........OK
test: fields, norms.......OK [21 fields]
test: terms, freq, prox...OK [1854 terms; 48703 terms/docs pairs;
62221 tokens]
test: stored fields.......OK [1401 total field count; avg 1 fields per doc]
test: term vectors........OK [0 total vector count; avg 0
term/freq vector fields per doc]
14 of 18: name=_xlf docCount=1399
compound=true
hasProx=true
numFiles=1
size (MB)=0.188
no deletions
test: open reader.........OK
test: fields, norms.......OK [21 fields]
test: terms, freq, prox...OK [1763 terms; 48910 terms/docs pairs;
63035 tokens]
test: stored fields.......OK [1399 total field count; avg 1 fields per doc]
test: term vectors........OK [0 total vector count; avg 0
term/freq vector fields per doc]
15 of 18: name=_xlh docCount=1727
compound=true
hasProx=true
numFiles=1
size (MB)=0.24
no deletions
test: open reader.........OK
test: fields, norms.......OK [21 fields]
test: terms, freq, prox...OK [2318 terms; 62596 terms/docs pairs;
80688 tokens]
test: stored fields.......OK [1727 total field count; avg 1 fields per doc]
test: term vectors........OK [0 total vector count; avg 0
term/freq vector fields per doc]
16 of 18: name=_xli docCount=1716
compound=true
hasProx=true
numFiles=1
size (MB)=0.237
no deletions
test: open reader.........OK
test: fields, norms.......OK [21 fields]
test: terms, freq, prox...OK [2264 terms; 61867 terms/docs pairs;
79497 tokens]
test: stored fields.......OK [1716 total field count; avg 1 fields per doc]
test: term vectors........OK [0 total vector count; avg 0
term/freq vector fields per doc]
17 of 18: name=_xlk docCount=2921
compound=true
hasProx=true
numFiles=1
size (MB)=0.364
no deletions
test: open reader.........OK
test: fields, norms.......OK [25 fields]
test: terms, freq, prox...OK [2077 terms; 96536 terms/docs pairs;
123166 tokens]
test: stored fields.......OK [2921 total field count; avg 1 fields per doc]
test: term vectors........OK [0 total vector count; avg 0
term/freq vector fields per doc]
18 of 18: name=_xll docCount=3876
compound=true
hasProx=true
numFiles=1
size (MB)=0.476
no deletions
test: open reader.........OK
test: fields, norms.......OK [21 fields]
test: terms, freq, prox...OK [2261 terms; 130104 terms/docs pairs;
166867 tokens]
test: stored fields.......OK [3876 total field count; avg 1 fields per doc]
test: term vectors........OK [0 total vector count; avg 0
term/freq vector fields per doc]
WARNING: 1 broken segments (containing 1124653 documents) detected
WARNING: 1124653 documents will be lost
NOTE: will write new segments file in 5 seconds; this will remove
1124653 docs from the index. THIS IS YOUR LAST CHANCE TO CTRL+C!
5...
4...
3...
2...
1...
Writing...
OK
Wrote new segments file "segments_1588"
Any ideas how would the index get corrupted?
Thanks,
-vivek
On Sat, Jan 3, 2009 at 8:04 AM, Brian Whitman <br...@echonest.com> wrote:
>>
>>
>> It's very strange that CheckIndex -fix did not resolve the issue. After
>> fixing it, if you re-run CheckIndex on the index do you still see that
>> original one broken segment present? CheckIndex should have removed
>> reference to that one segment.
>>
>
> I just ran it again, and it detected the same error and claimed to fix it. I
> then shut down the solr server (I wasn't sure if this would be an issue),
> ran it a third time (where it again found and claimed to fix the error),
> then a fourth where it did not find any problems, and now the optimize()
> call on the running server does not throw the merge exception.
>
> Did this corruption happen only once? (You mentioned hitting dups in
> the past...
>> but did you also see corruption too?)
>
>
> Not that we know of, but it's very likely we never noticed. (The only reason
> I discovered this was our commits were taking 20-40x longer on this index
> than others)
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
