On Tue, May 10, 2016 at 2:30 AM, Ravikumar Govindarajan < [email protected]> wrote:
> Actually there are 2 issues... > > 1. IndexReaderClosedException > 2. HDFS Stream Closed > Likely when the index is closed it closes the underlying indexinputs as well causing the HDFS Stream closed exception. > > Merge completion results in File Deletion & ultimately HDFS Stream Closed > during Search.... > > I use IndexFileDeleter with KeepOnlyLastCommitDeletionPolicy. This blindly > deletes the file, without bothering to cross-check IndexReader.RefCount > > 0. > Hmm. You can see here: https://github.com/apache/incubator-blur/blob/release-0.2.2-incubating/blur-core/src/main/java/org/apache/blur/manager/writer/BlurIndexSimpleWriter.java#L303 That once the new index is available it is swapped into the index ref object and the old one is sent to the index closer. Once the ref to the index are low enough it closes the index. Or at least it should. I will continue looking into the problem but I don't have a solution for you yet. Aaron > > > *Exception(message:Unknown error during rewrite, > stackTraceStr:java.io.IOException: Stream closed* > at org.apache.hadoop.hdfs.DFSInputStream.pread(DFSInputStream.java:1385) > at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:1374) > at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:89) > at > > org.apache.blur.store.hdfs.HdfsIndexInput.readInternal(HdfsIndexInput.java:62) > at > > org.apache.blur.store.buffer.ReusedBufferedIndexInput.readBytes(ReusedBufferedIndexInput.java:167) > at > > org.apache.blur.store.buffer.ReusedBufferedIndexInput.readBytes(ReusedBufferedIndexInput.java:122) > at > > org.apache.blur.store.hdfs.MmapCacheIndexInput.readAndcache(MmapCacheIndexInput.java:24) > at > > org.apache.blur.store.blockcache_v2.CacheIndexInput.fillNormally(CacheIndexInput.java:354) > at > > org.apache.blur.store.blockcache_v2.CacheIndexInput.fill(CacheIndexInput.java:379) > at > > org.apache.blur.store.blockcache_v2.CacheIndexInput.tryToFill(CacheIndexInput.java:297) > at > > org.apache.blur.store.blockcache_v2.CacheIndexInput.readByte(CacheIndexInput.java:151) > at > > org.apache.blur.lucene.warmup.TraceableIndexInput.readByte(TraceableIndexInput.java:62) > at org.apache.lucene.store.DataInput.readVInt(DataInput.java:108) > at > > org.apache.lucene.codecs.BlockTreeTermsReader$FieldReader$SegmentTermsEnum$Frame.loadBlock(BlockTreeTermsReader.java:2366) > at > > org.apache.lucene.codecs.BlockTreeTermsReader$FieldReader$SegmentTermsEnum.seekCeil(BlockTreeTermsReader.java:1949) > at > > org.apache.blur.index.ExitableReader$ExitableTermsEnum.seekCeil(ExitableReader.java:250) > at > org.apache.lucene.index.FilteredTermsEnum.next(FilteredTermsEnum.java:225) > at > > org.apache.lucene.search.TermCollectingRewrite.collectTerms(TermCollectingRewrite.java:78) > at > > org.apache.lucene.search.ConstantScoreAutoRewrite.rewrite(ConstantScoreAutoRewrite.java:95) > at > > org.apache.lucene.search.MultiTermQuery$ConstantScoreAutoRewrite.rewrite(MultiTermQuery.java:220) > at org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:288) > at org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:412) > at org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:412) > at org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:412) > at > > On Mon, May 9, 2016 at 4:42 PM, Ravikumar Govindarajan < > [email protected]> wrote: > > > One extra info we gleaned from the logs... > > > > 1. Merge Starts & is about to complete > > 2. Searcher is opened > > 3. Merge Completes > > 4. Ref-count drops to 0 in IndexReader > > 5. IndexReader closed while Searcher is still open > > > > This seems to be the main pattern for causing the Exception > > > > -- > > Ravi > > > > On Mon, May 9, 2016 at 3:08 PM, Ravikumar Govindarajan < > > [email protected]> wrote: > > > >> Thanks Aaron... > >> > >> Just a quick question. Lucene itself has ref-counting to close it's > >> readers no? Or Blur has it's own logic to handle it? > >> > >> -- > >> Ravi > >> > >> On Fri, May 6, 2016 at 7:56 PM, Aaron McCurry <[email protected]> > wrote: > >> > >>> Likely yes. If have a few minutes this weekend I can look through that > >>> version and see if I can point you in the right direction. > >>> > >>> On Fri, May 6, 2016 at 8:46 AM, Ravikumar Govindarajan < > >>> [email protected]> wrote: > >>> > >>> > Sometimes during an ongoing search we receive an > >>> > IndexReaderClosedException... > >>> > > >>> > We are on an older version of Blur (0.2.2). Has this been fixed in > >>> newer > >>> > versions or we have been using it wrongly? > >>> > > >>> > *stackTraceStr:org.apache.lucene.store.AlreadyClosedException: this > >>> > IndexReader cannot be used anymore as one of its child readers was > >>> closed* > >>> > at > org.apache.lucene.index.IndexReader.ensureOpen(IndexReader.java:257) > >>> > at > >>> > > >>> > > >>> > org.apache.lucene.index.FilterAtomicReader.fields(FilterAtomicReader.java:380) > >>> > at > >>> > > >>> > > >>> > org.apache.blur.index.ExitableReader$ExitableFilterAtomicReader.fields(ExitableReader.java:81) > >>> > at > >>> > > >>> > > >>> > org.apache.lucene.search.TermCollectingRewrite.collectTerms(TermCollectingRewrite.java:52) > >>> > at > >>> > > >>> > > >>> > org.apache.lucene.search.ConstantScoreAutoRewrite.rewrite(ConstantScoreAutoRewrite.java:95) > >>> > at > >>> > > >>> > > >>> > org.apache.lucene.search.MultiTermQuery$ConstantScoreAutoRewrite.rewrite(MultiTermQuery.java:220) > >>> > at > >>> > org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:288) > >>> > at > org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:412) > >>> > > >>> > >> > >> > > >
