Lucene 2.3.1 Index Corruption?

2008-03-17 Thread Jamie
Hi There I am getting the following error while searching a given index: java.io.FileNotFoundException: /usr/local/index/_0.fdt (No such file or directory) at java.io.RandomAccessFile.open(Native Method) at java.io.RandomAccessFile.(Unknown Source) at org.apache.lucene.s

Re: Lucene 2.3.1 Index Corruption?

2008-03-17 Thread Michael McCandless
Hi, Can you describe what led up to this? Were there any exceptions when adding documents to the index? Was the index newly created with 2.3.1 or created on 2.3.0 or 2.2? What options are you using in your IndexWriter? Is it easy to reproduce? If so, can you call setInfoStream on your

Re: Lucene 2.3.1 Index Corruption?

2008-03-17 Thread Jamie
As a further followup: The follows files are located in the index: ls /usr/local/index _0.fnm _0.frq _0.nrm _0.prx _0.tii _0.tis _1.cfs indexinfo _j.cfs segments.gen segments_s This problem appears to be intermittant and has occurred on several machines. Is there any incorrect way

Re: Lucene 2.3.1 Index Corruption?

2008-03-18 Thread Jamie
Hi Michael Sorry for the late reply. As you guessed, it missed my attention. Michael McCandless wrote: Hi, Can you describe what led up to this? My application indexes emails. In this particular instance, I had reindexed all emails from their original sources. The error occurred while I w

Re: Lucene 2.3.1 Index Corruption?

2008-03-18 Thread Michael McCandless
It looks like you ignore any IOException coming out of IndexWriter.close? Can you put some code in the catch clause around writer.close to see if you are hitting some exception there? Also, you forcefully remove the write lock if it's present. But are you absolutely certain there isn't

Re: Lucene 2.3.1 Index Corruption?

2008-03-18 Thread Michael McCandless
Yes fdt/fdx hold stored fields. When the first buffered document is added these files are created. The only way they disappear (through Lucene's APIs) is if a writer is opened on that directory, and, those files are not referenced by the current segments file. This is why I'm concerned

Re: Lucene 2.3.1 Index Corruption?

2008-03-18 Thread Jamie
Michael McCandless wrote: Yes fdt/fdx hold stored fields. When the first buffered document is added these files are created. The only way they disappear (through Lucene's APIs) is if a writer is opened on that directory, and, those files are not referenced by the current segments file. Th

Re: Lucene 2.3.1 Index Corruption?

2008-03-18 Thread Michael McCandless
OK, opening two writers at once is definitely a recipe for disaster. Please post back on whether this does or doesn't resolve it. Previous versions of Lucene didn't write the fdt/fdx files until a segment is flushed, so it's possible you escaped index corruption (but, lost documents) before.