Sorry my last reply was nonsense. Even if you reuse TokenStreams, the consumer should close the stream at the end of operations.
And this is for sure done by the DocInverterPerField (finally-block within TokenStream.close()). Maybe the user with the problem has created a TokenFilter or something like that, not passing TS.close() to the underlying TS? We had such a problem with one of the Solr "TokenFilters" which was a subclass of Tokenizer not TokenFilter and missed to override the close/end/... methods. So in 2.9, the Reader is correctly closed, if the TokenStream chain is correctly set up, passing all close() calls to the delegate. ----- Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: u...@thetaphi.de > -----Original Message----- > From: Uwe Schindler [mailto:u...@thetaphi.de] > Sent: Sunday, September 27, 2009 10:48 AM > To: java-dev@lucene.apache.org > Subject: RE: Lucene 2.9.0-rc5 : Reader stays open after > IndexWriter.updateDocument(), is that possible? > > I think I know, what is different than earlier versions: > > The close of the Reader is done by Tokenizer.close(). As in 2.9.0 we did > much work to get all the Tokenizers reusable, there is the following > problem: > > The Reader is only closed, if you call Tokenizer.close(), but not, if you > call Tokenizer.reset(newReader). This was always be so (and has nothing to > do with the new TokenStream API), but in earlier version not all analyzers > were able to reuse the TokenStreams. > > Uwe > > ----- > Uwe Schindler > H.-H.-Meier-Allee 63, D-28213 Bremen > http://www.thetaphi.de > eMail: u...@thetaphi.de > > > -----Original Message----- > > From: Uwe Schindler [mailto:u...@thetaphi.de] > > Sent: Sunday, September 27, 2009 10:42 AM > > To: java-dev@lucene.apache.org > > Subject: RE: Lucene 2.9.0-rc5 : Reader stays open after > > IndexWriter.updateDocument(), is that possible? > > > > How does your test perform with 2.4.1 ? > > > > ----- > > Uwe Schindler > > H.-H.-Meier-Allee 63, D-28213 Bremen > > http://www.thetaphi.de > > eMail: u...@thetaphi.de > > > > > -----Original Message----- > > > From: Chris Hostetter [mailto:hossman_luc...@fucit.org] > > > Sent: Sunday, September 27, 2009 3:34 AM > > > To: Lucene Dev > > > Subject: Re: Lucene 2.9.0-rc5 : Reader stays open after > > > IndexWriter.updateDocument(), is that possible? > > > > > > > > > : Thanks Mark for the pointer, I thought somehow that lucene closed > them > > > as a > > > : convenience, I don't know if it did that in previous releases (aka > > > 2.4.1) but > > > : I'll close them myself from now on. > > > > > > FWIW: As far as i know, Lucene has always closed the Reader for you > when > > > calling addDocument/updateDocument -- BUT -- the docs never promized > > > that Lucene would close any Readers used in Fields. In fact the Field > > > constructor docs say "you may not close the Reader until addDocument > has > > > been called" suggesting that you should close it yourself. > > > (Reader.close() is very clear that there should be no effect on > closing > > a > > > Reader multiple times, so this is safe no matter what Lucene does) > > > > > > That said: If the behavior has changed in 2.9, this could easily bite > > lots > > > of people in the ass if they haven't been closing their readers and > now > > > they run out of file handles. I wrote a quick test to try and > reproduce > > > the problem you're describing, but as far as i can tell 2.9.0 > > > (final) still seems to close the Reader for you. > > > > > > Can anyone else reproduce this problem of Reader's in Field's not > > getting > > > closed? (my test is below) > > > > > > --BEGIN-- > > > package org.apache.lucene; > > > > > > /** > > > * Licensed to the Apache Software Foundation (ASF) under one or more > > > * contributor license agreements. See the NOTICE file distributed > with > > > * this work for additional information regarding copyright ownership. > > > * The ASF licenses this file to You under the Apache License, Version > > 2.0 > > > * (the "License"); you may not use this file except in compliance > with > > > * the License. You may obtain a copy of the License at > > > * > > > * http://www.apache.org/licenses/LICENSE-2.0 > > > * > > > * Unless required by applicable law or agreed to in writing, software > > > * distributed under the License is distributed on an "AS IS" BASIS, > > > * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or > > > implied. > > > * See the License for the specific language governing permissions and > > > * limitations under the License. > > > */ > > > > > > import org.apache.lucene.analysis.KeywordAnalyzer; > > > import org.apache.lucene.index.*; > > > import org.apache.lucene.document.*; > > > import org.apache.lucene.util.LuceneTestCase; > > > import org.apache.lucene.store.RAMDirectory; > > > > > > import java.io.*; > > > > > > public class TestFieldWithReaderClosing extends LuceneTestCase { > > > > > > IndexWriter writer = null; > > > Document d = null; > > > CloseStateReader reader; > > > public void setUp() throws Exception { > > > writer = new IndexWriter(new RAMDirectory(), > > > new KeywordAnalyzer(), true, > > > IndexWriter.MaxFieldLength.LIMITED); > > > d = new Document(); > > > d.add(new Field("id", "x", Field.Store.YES, > Field.Index.ANALYZED)); > > > reader = new CloseStateReader("foo"); > > > d.add(new Field("contents", reader)); > > > } > > > public void tearDown() throws Exception { > > > writer.close(); > > > writer = null; > > > reader.close(); > > > reader = null; > > > } > > > > > > public void testAdd() throws Exception { > > > writer.addDocument(d); > > > assertEquals("close count should be 1", 1, > reader.getCloseCount()); > > > writer.close(); > > > assertEquals("close count should still be 1", 1, > > > reader.getCloseCount()); > > > } > > > public void testEmptyUpdate() throws Exception { > > > writer.updateDocument(new Term("id","x"), d); > > > assertEquals("close count should be 1", 1, > reader.getCloseCount()); > > > writer.close(); > > > assertEquals("close count should still be 1", 1, > > > reader.getCloseCount()); > > > } > > > public void testAddAndUpdate() throws Exception { > > > writer.addDocument(d); > > > assertEquals("close count should be 1", 1, > reader.getCloseCount()); > > > d = new Document(); > > > d.add(new Field("id", "x", Field.Store.YES, > Field.Index.ANALYZED)); > > > reader = new CloseStateReader("foo"); > > > d.add(new Field("contents", reader)); > > > writer.updateDocument(new Term("id","x"), d); > > > assertEquals("new close count should be 1", 1, > > > reader.getCloseCount()); > > > writer.close(); > > > assertEquals("new close count should still be 1", 1, > > > reader.getCloseCount()); > > > } > > > > > > > > > static class CloseStateReader extends StringReader { > > > private int closeCount = 0; > > > public CloseStateReader(String s) { > > > super(s); > > > } > > > public synchronized void close() { > > > closeCount++; > > > super.close(); > > > } > > > public int getCloseCount() { > > > return closeCount; > > > } > > > } > > > } > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org > > > For additional commands, e-mail: java-dev-h...@lucene.apache.org > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org > > For additional commands, e-mail: java-dev-h...@lucene.apache.org > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-dev-h...@lucene.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
