Lucene 2.9: IOException from IndexReader.reopen() - Real time search
I am getting IOException when I am doing a "Real-time" search, i.e. I am
creating a Index using the Index Writer and also opening the Index using
Index Reader (writer.getReader()) to make sure the document does not exist
prior adding to the Index file.
The code works perfect fine multiple time indexed across couple of GB data,
however, on a TB it fails with the exception listed below.
Any suggestions appreciated. Thanks. Kumar
Code Snippet:
IndexReader reader2 = reader.reopen();//get
if (reader2 != reader) {
reader.close();
reader = reader2;
searcher = new IndexSearcher(reader);
reader2 = null;
}
Exception:
Exception in thread "main" java.io.IOException: directory
'/Users/Kumar/Dev/Cust/index/values' exists and is a directory, but cannot
be listed: list() returned null
at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:516)
at org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:531)
at
org.apache.lucene.index.IndexFileDeleter.refresh(IndexFileDeleter.java:307)
at
org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:4300)
at
org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:4192)
at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:4183)
at
org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:470)
at
org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:444)
at
org.apache.lucene.index.DirectoryReader.doReopen(DirectoryReader.java:384)
at
org.apache.lucene.index.DirectoryReader.reopen(DirectoryReader.java:348)
Kumar_/|\_
www.saisk.com
[email protected]
"making a profound difference with knowledge and creativity..."
Re: Lucene 2.9: IOException from IndexReader.reopen() - Real time search
Thanks Mike. I think it is something to do with the merge factor. Modified the code to do optimize in the finally block the following error message was thrown. Code Snippet: nameWriter.optimize(); // errors here nameWriter.close(); valueWriter.optimize(); //I am using multiple writers simultaneously. Writing to different directory. valueWriter.close() Index Directory Files: _1o.cfs _1p.cfs _1q.cfs _1r.cfs _1s.cfs _1t.fnm segments_1 segments.gen write.lock Exception in thread "main" java.io.IOException: background merge hit exception: _1o:c55 _1p:c1 _1q:c1 _1r:c1 _1s:c1 into _1t [optimize] [mergeDocStores] at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2854) at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2789) at org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2759) at com.hli.vis.index.VisualSoftIndexWriter.commitChanges(VSIndexWriter.java:46) at com.hli.vis.index.Main.main(Main.java:39) Caused by: java.io.FileNotFoundException: /Users/Kumar/Dev/Cust/index/names/_1t.fdx (Too many open files) at java.io.RandomAccessFile.open(Native Method) at java.io.RandomAccessFile.(RandomAccessFile.java:212) at org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexOutput.(SimpleFSDirectory.java:186) at org.apache.lucene.store.NIOFSDirectory.createOutput(NIOFSDirectory.java:76) at org.apache.lucene.index.FieldsWriter.(FieldsWriter.java:82) at org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:334) at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:153) at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5029) at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614) at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235) at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291) Java Result: 1 Kumar_/|\_ www.saisk.com [email protected] "making a profound difference with knowledge and creativity..." On Thu, Dec 31, 2009 at 6:45 AM, Michael McCandless < [email protected]> wrote: > It sounds like you may be running out of file descriptors -- how many > segments are in your index? > > The reopen logic looks correct (you are closing the old reader). Is > there anything else that may be holding files open? > > Have you changed any of IW's settings, eg mergeFactor? > > Mike > > On Wed, Dec 30, 2009 at 3:21 PM, Kumaravel Kandasami > wrote: > > I am getting IOException when I am doing a "Real-time" search, i.e. I am > > creating a Index using the Index Writer and also opening the Index using > > Index Reader (writer.getReader()) to make sure the document does not > exist > > prior adding to the Index file. > > > > The code works perfect fine multiple time indexed across couple of GB > data, > > however, on a TB it fails with the exception listed below. > > Any suggestions appreciated. Thanks. Kumar > > > > Code Snippet: > > > > IndexReader reader2 = reader.reopen();//get > > > >if (reader2 != reader) { > > > >reader.close(); > >reader = reader2; > >searcher = new IndexSearcher(reader); > >reader2 = null; > >} > > > > > > Exception: > > > > Exception in thread "main" java.io.IOException: directory > > '/Users/Kumar/Dev/Cust/index/values' exists and is a directory, but > cannot > > be listed: list() returned null > >at > org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:516) > >at > org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:531) > >at > > > org.apache.lucene.index.IndexFileDeleter.refresh(IndexFileDeleter.java:307) > >at > > > org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:4300) > >at > > org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:4192) > >at > org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:4183) > >at > > org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:470) > >at > > org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:444) > >at > > > org.apache.lucene.index.DirectoryReader.doReopen(DirectoryReader.java:384) > >at > > org.apache.lucene.index.DirectoryReader.reopen(DirectoryReader.java:348) > > > > Kumar_/|\_ > > www.saisk.com > > [email protected] > > "making a profound difference with knowledge and creativity..." > > > > - > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
Re: Lucene 2.9: IOException from IndexReader.reopen() - Real time search
Identified the problem. reader.close() was not getting called in a specific logic flow. Thank You. Kumar_/|\_ www.saisk.com [email protected] "making a profound difference with knowledge and creativity..." On Thu, Dec 31, 2009 at 11:11 AM, Kumaravel Kandasami < [email protected]> wrote: > Thanks Mike. > > I think it is something to do with the merge factor. > Modified the code to do optimize in the finally block the following error > message was thrown. > > Code Snippet: > nameWriter.optimize(); // errors here > nameWriter.close(); > valueWriter.optimize(); //I am using multiple writers > simultaneously. Writing to different directory. > valueWriter.close() > > > Index Directory Files: > _1o.cfs > _1p.cfs > _1q.cfs > _1r.cfs > _1s.cfs > _1t.fnm > segments_1 > segments.gen > write.lock > > > Exception in thread "main" java.io.IOException: background merge hit > exception: _1o:c55 _1p:c1 _1q:c1 _1r:c1 _1s:c1 into _1t [optimize] > [mergeDocStores] > at > org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2854) > at > org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2789) > at > org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2759) > at > com.hli.vis.index.VisualSoftIndexWriter.commitChanges(VSIndexWriter.java:46) > at com.hli.vis.index.Main.main(Main.java:39) > Caused by: java.io.FileNotFoundException: > /Users/Kumar/Dev/Cust/index/names/_1t.fdx (Too many open files) > at java.io.RandomAccessFile.open(Native Method) > at java.io.RandomAccessFile.(RandomAccessFile.java:212) > at > org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexOutput.(SimpleFSDirectory.java:186) > at > org.apache.lucene.store.NIOFSDirectory.createOutput(NIOFSDirectory.java:76) > at > org.apache.lucene.index.FieldsWriter.(FieldsWriter.java:82) > at > org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:334) > at > org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:153) > at > org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:5029) > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4614) > at > org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:235) > at > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:291) > Java Result: 1 > > > > Kumar_/|\_ > www.saisk.com > [email protected] > "making a profound difference with knowledge and creativity..." > > > On Thu, Dec 31, 2009 at 6:45 AM, Michael McCandless < > [email protected]> wrote: > >> It sounds like you may be running out of file descriptors -- how many >> segments are in your index? >> >> The reopen logic looks correct (you are closing the old reader). Is >> there anything else that may be holding files open? >> >> Have you changed any of IW's settings, eg mergeFactor? >> >> Mike >> >> On Wed, Dec 30, 2009 at 3:21 PM, Kumaravel Kandasami >> wrote: >> > I am getting IOException when I am doing a "Real-time" search, i.e. I am >> > creating a Index using the Index Writer and also opening the Index using >> > Index Reader (writer.getReader()) to make sure the document does not >> exist >> > prior adding to the Index file. >> > >> > The code works perfect fine multiple time indexed across couple of GB >> data, >> > however, on a TB it fails with the exception listed below. >> > Any suggestions appreciated. Thanks. Kumar >> > >> > Code Snippet: >> > >> > IndexReader reader2 = reader.reopen();//get >> > >> >if (reader2 != reader) { >> > >> >reader.close(); >> >reader = reader2; >> >searcher = new IndexSearcher(reader); >> >reader2 = null; >> >} >> > >> > >> > Exception: >> > >> > Exception in thread "main" java.io.IOException: directory >> > '/Users/Kumar/Dev/Cust/index/values' exists and is a directory, but >> cannot >> > be listed: list() returned null >> >at >> org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:516) >> >at >> org.apache.lucene.store.FSDirectory.listAll(FSDirectory.java:531)
