Thanks Mike and Uwe.
I already reindexed in production, my goal is to get to the root cause to
make sure it doesn't happen again.
Will remove the flush(). No idea why it's there.
Attaching checkIndex.Main() output (why did I bother writing my own output
:#)
*Output:*
Opening index @ C:\\customers\\SC\\corrupt catalog index E3 -
Nov\\WDPP29715_ap03\\opt\\WAS\\LotusConnections\\Data\\catalog\\index\\Places\\index
Segments file=segments_2bsy numSegments=10 version=FORMAT_3_1 [Lucene 3.1]
userData={VERSION=9460}
1 of 10: name=_9n docCount=4141
compound=false
hasProx=true
numFiles=9
size (MB)=3.89
diagnostics = {os.arch=amd64, java.vendor=IBM Corporation, os=Linux,
mergeFactor=10, source=merge, java.version=1.6.0, lucene.version=3.1.0
1085809 - 2011-03-26 17:59:57, optimize=false,
os.version=2.6.18-348.12.1.el5}
has deletions [delFileName=_9n_2t.del]
test: open reader.........OK [209 deleted docs]
test: fields..............OK [27 fields]
test: field norms.........OK [4 fields]
test: terms, freq, prox...OK [82966 terms; 295120 terms/docs pairs;
300750 tokens]
test: stored fields.......OK [72872 total field count; avg 18.533
fields per doc]
test: term vectors........OK [0 total vector count; avg 0 term/freq
vector fields per doc]
2 of 10: name=_9o docCount=19999
compound=false
hasProx=true
numFiles=9
size (MB)=21.487
diagnostics = {os.arch=amd64, java.vendor=IBM Corporation, os=Linux,
mergeFactor=10, source=merge, java.version=1.6.0, lucene.version=3.1.0
1085809 - 2011-03-26 17:59:57, optimize=false,
os.version=2.6.18-348.12.1.el5}
has deletions [delFileName=_9o_lg.del]
test: open reader.........OK [1396 deleted docs]
test: fields..............OK [27 fields]
test: field norms.........OK [4 fields]
test: terms, freq, prox...OK [318090 terms; 1773898 terms/docs pairs;
1888318 tokens]
test: stored fields.......OK [390466 total field count; avg 20.989
fields per doc]
test: term vectors........OK [0 total vector count; avg 0 term/freq
vector fields per doc]
3 of 10: name=_6k docCount=2000
compound=true
hasProx=true
numFiles=2
size (MB)=2.386
diagnostics = {os.arch=amd64, java.vendor=IBM Corporation, os=Linux,
source=addIndexes(IndexReader...), java.version=1.6.0, lucene.version=3.1.0
1085809 - 2011-03-26 17:59:57, os.version=2.6.18-348.12.1.el5}
has deletions [delFileName=_6k_62.del]
test: open reader.........OK [389 deleted docs]
test: fields..............OK [27 fields]
test: field norms.........OK [4 fields]
test: terms, freq, prox...OK [46699 terms; 193013 terms/docs pairs;
178450 tokens]
test: stored fields.......OK [35965 total field count; avg 22.325
fields per doc]
test: term vectors........OK [0 total vector count; avg 0 term/freq
vector fields per doc]
4 of 10: name=_6l docCount=2000
compound=true
hasProx=true
numFiles=2
size (MB)=2.477
diagnostics = {os.arch=amd64, java.vendor=IBM Corporation, os=Linux,
source=addIndexes(IndexReader...), java.version=1.6.0, lucene.version=3.1.0
1085809 - 2011-03-26 17:59:57, os.version=2.6.18-348.12.1.el5}
has deletions [delFileName=_6l_hx.del]
test: open reader.........OK [864 deleted docs]
test: fields..............OK [27 fields]
test: field norms.........OK [4 fields]
test: terms, freq, prox...OK [55730 terms; 196164 terms/docs pairs;
117213 tokens]
test: stored fields.......OK [23202 total field count; avg 20.424
fields per doc]
test: term vectors........OK [0 total vector count; avg 0 term/freq
vector fields per doc]
5 of 10: name=_ug3 docCount=2949
compound=false
hasProx=true
numFiles=9
FAILED
WARNING: fixIndex() would remove reference to this segment; full
exception:
java.io.FileNotFoundException: _ug3_10h.del
at org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:292)
at org.apache.lucene.index.SegmentInfo.sizeInBytes(SegmentInfo.java:299)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:446)
at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:898)
6 of 10: name=_1qro docCount=2701
compound=false
hasProx=true
numFiles=9
size (MB)=3.478
diagnostics = {os.arch=amd64, java.vendor=IBM Corporation, os=Linux,
mergeFactor=10, source=merge, java.version=1.6.0, lucene.version=3.1.0
1085809 - 2011-03-26 17:59:57, optimize=false,
os.version=2.6.18-348.16.1.el5}
has deletions [delFileName=_1qro_p0.del]
test: open reader.........OK [1473 deleted docs]
test: fields..............OK [30 fields]
test: field norms.........OK [4 fields]
test: terms, freq, prox...OK [76625 terms; 278909 terms/docs pairs;
143954 tokens]
test: stored fields.......OK [25932 total field count; avg 21.117
fields per doc]
test: term vectors........OK [0 total vector count; avg 0 term/freq
vector fields per doc]
7 of 10: name=_24xf docCount=1645
compound=true
hasProx=true
numFiles=2
FAILED
WARNING: fixIndex() would remove reference to this segment; full
exception:
java.io.FileNotFoundException: _24xf_9l.del
at org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:292)
at org.apache.lucene.index.SegmentInfo.sizeInBytes(SegmentInfo.java:299)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:446)
at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:898)
8 of 10: name=_2czq docCount=681
compound=true
hasProx=true
numFiles=2
size (MB)=0.947
diagnostics = {os.arch=amd64, java.vendor=IBM Corporation, os=Linux,
mergeFactor=10, source=merge, java.version=1.6.0, lucene.version=3.1.0
1085809 - 2011-03-26 17:59:57, optimize=false,
os.version=2.6.18-348.16.1.el5}
has deletions [delFileName=_2czq_3.del]
test: open reader.........OK [465 deleted docs]
test: fields..............OK [30 fields]
test: field norms.........OK [4 fields]
test: terms, freq, prox...OK [25569 terms; 72519 terms/docs pairs;
21076 tokens]
test: stored fields.......OK [4328 total field count; avg 20.037 fields
per doc]
test: term vectors........OK [0 total vector count; avg 0 term/freq
vector fields per doc]
9 of 10: name=_2d00 docCount=1997
compound=true
hasProx=true
numFiles=2
size (MB)=2.564
diagnostics = {os.arch=amd64, java.vendor=IBM Corporation, os=Linux,
source=addIndexes(IndexReader...), java.version=1.6.0, lucene.version=3.1.0
1085809 - 2011-03-26 17:59:57, os.version=2.6.18-371.el5}
has deletions [delFileName=_2d00_1.del]
test: open reader.........OK [1 deleted docs]
test: fields..............OK [31 fields]
test: field norms.........OK [4 fields]
test: terms, freq, prox...OK [57448 terms; 204959 terms/docs pairs;
241754 tokens]
test: stored fields.......OK [44545 total field count; avg 22.317
fields per doc]
test: term vectors........OK [0 total vector count; avg 0 term/freq
vector fields per doc]
10 of 10: name=_2l3x docCount=865
compound=true
hasProx=true
numFiles=1
FAILED
WARNING: fixIndex() would remove reference to this segment; full
exception:
java.io.FileNotFoundException: _2l3x.cfs
at org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:292)
at org.apache.lucene.index.SegmentInfo.sizeInBytes(SegmentInfo.java:299)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:446)
at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:898)
WARNING: 3 broken segments (containing 5459 documents) detected
WARNING: would write new segments file, and 5459 documents would be lost,
if -fix were specified
On Wed, Nov 6, 2013 at 11:07 PM, Uwe Schindler <u...@thetaphi.de> wrote:
> Hi,
> > Hello,
> > I got an index corruption in production, and was wondering if it might
> be a
> > known bug (still with Lucene 3.1), or is my code doing something wrong.
> > It's a local disk index. No known machine power lose. No suppose to even
> > happen, right?
> >
> > This index that got corrupted is updated every 30sec; adding to it a
> small
> > delta's index (using addIndexes()) that was replicated from another
> machine.
> > The series of writer actions to update the index is:
> > 1. writer.deleteDocuments(q);
> > 2. writer.flush(false, true);
> > 3. writer.addIndexes(reader);
> > 4. writer.commit(map);
> >
> > Is the index exposed to corruptions only during commit, or is
> addIndexes()
> > risky by itself (doc says it's not).
> > LUCENE-2610 <https://issues.apache.org/jira/browse/LUCENE-2610> kind of
> > looks in the neberhood, though it's not a bug report.
>
> Hi, LUCENE-2610 is completely unrelated, as this only affects
> addIndexes(Directory...), not addIndexes(IndexReader...). The one you are
> using is using the natural Lucene merging as it is done all the time while
> indexing documents (Lucene internally uses the same code like
> addIndexes(IndexReader) to merge segments). addIndexes(Directory) is very
> different and more risky to have bugs in older Lucene versions (this one
> copies index files around without touching them, but renaming them to have
> new segment names - which is somehow "unnatural"; it also does not
> correctly lock the index directory in older versions).
>
> Why do you call flush() at all? I would leave that out, there is no reason
> to do this from userland code.
>
> To "repair" the index, use Checkindex command line tool with the "fix"
> option. This will delete the segment that is missing (_33gg.cfs). Of course
> this data is lost, but as the file is not there it is lost already. This
> will just remove the metadata of this missing segment from your index. But
> before doing this, you should check what checkindex prints out without fix
> option - the info you posted is not the console output the tool prints when
> ran from command line and run with assertions enabled (-ea JVM option). The
> output looks like the "toString) of the Java API of CheckIndex class, which
> is not so helpful. Please post the full output of the tool executed from
> command line:
> java -cp lucene-core-3.1.0.jar org.apache.lucene.index.CheckIndex
> <options....>
>
> Uwe
>
> > I'll add an ls -l output in a follow up email.
> >
> > Technically the first indication of problems is when calling flush, but
> it could
> > be that the previous writer action left it broken for flush to fail.
> > My stack trace is:
> > Caused by: java.io.FileNotFoundException:
> > /disks/data1/opt/WAS/LotusConnections/Data/catalog/index/Places/index/
> > _33gg.cfs
> > (No such file or directory)
> > at java.io.RandomAccessFile.open(Native Method)
> > at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> > at
> > org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.
> > <init>(SimpleFSDirectory.java:69)
> > at
> > org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.<init>(Simp
> > leFSDirectory.java:90)
> > at
> > org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.<init>(NIOFSDire
> > ctory.java:91)
> > at
> > org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:78)
> > at
> > org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.
> > java:66)
> > at
> > org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentRead
> > er.java:113)
> > at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:578)
> > at
> > org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:684)
> > at
> > org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:659)
> > at
> > org.apache.lucene.index.BufferedDeletes.applyDeletes(BufferedDeletes.jav
> > a:283)
> > at
> > org.apache.lucene.index.BufferedDeletes.applyDeletes(BufferedDeletes.jav
> > a:191)
> > at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3358)
> > at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3296)
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
> For additional commands, e-mail: java-user-h...@lucene.apache.org
>
>
