[jira] [Commented] (LUCENE-5931) DirectoryReader.openIfChanged(oldReader, commit) incorrectly assumes given commit point has deletes/field updates

2014-09-22 Thread Vitaly Funstein (JIRA) 

[ 
https://issues.apache.org/jira/browse/LUCENE-5931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14143710#comment-14143710
 ] 

Vitaly Funstein commented on LUCENE-5931:
-

Michael,

Your updated patch definitely fixes the issue. But I just wanted to understand 
why deletes are so special, in that - if I don't have any buffered deletes for 
the segment, but new documents only, the reused reader instance won't pick them 
up, even without the fix in place. This is because liveDocs won't capture 
unflushed doc ids?

 DirectoryReader.openIfChanged(oldReader, commit) incorrectly assumes given 
 commit point has deletes/field updates
 -

 Key: LUCENE-5931
 URL: https://issues.apache.org/jira/browse/LUCENE-5931
 Project: Lucene - Core
  Issue Type: Bug
  Components: core/index
Affects Versions: 4.6.1
Reporter: Vitaly Funstein
Assignee: Michael McCandless
Priority: Critical
 Attachments: CommitReuseTest.java, LUCENE-5931.patch, 
 LUCENE-5931.patch, LUCENE-5931.patch, LUCENE-5931.patch


 {{StandardDirectoryReader}} assumes that the segments from commit point have 
 deletes, when they may not, yet the original SegmentReader for the segment 
 that we are trying to reuse does. This is evident when running attached JUnit 
 test case with asserts enabled (default): 
 {noformat}
 java.lang.AssertionError
   at 
 org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:188)
   at 
 org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:326)
   at 
 org.apache.lucene.index.StandardDirectoryReader$2.doBody(StandardDirectoryReader.java:320)
   at 
 org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:702)
   at 
 org.apache.lucene.index.StandardDirectoryReader.doOpenFromCommit(StandardDirectoryReader.java:315)
   at 
 org.apache.lucene.index.StandardDirectoryReader.doOpenNoWriter(StandardDirectoryReader.java:311)
   at 
 org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:262)
   at 
 org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:183)
 {noformat}
 or, if asserts are disabled then it falls through into NPE:
 {noformat}
 java.lang.NullPointerException
   at java.io.File.init(File.java:305)
   at 
 org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:80)
   at 
 org.apache.lucene.codecs.lucene40.BitVector.init(BitVector.java:327)
   at 
 org.apache.lucene.codecs.lucene40.Lucene40LiveDocsFormat.readLiveDocs(Lucene40LiveDocsFormat.java:90)
   at org.apache.lucene.index.SegmentReader.init(SegmentReader.java:131)
   at 
 org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:194)
   at 
 org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:326)
   at 
 org.apache.lucene.index.StandardDirectoryReader$2.doBody(StandardDirectoryReader.java:320)
   at 
 org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:702)
   at 
 org.apache.lucene.index.StandardDirectoryReader.doOpenFromCommit(StandardDirectoryReader.java:315)
   at 
 org.apache.lucene.index.StandardDirectoryReader.doOpenNoWriter(StandardDirectoryReader.java:311)
   at 
 org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:262)
   at 
 org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:183)
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-5931) DirectoryReader.openIfChanged(oldReader, commit) incorrectly assumes given commit point has deletes/field updates

2014-09-18 Thread Vitaly Funstein (JIRA) 

[ 
https://issues.apache.org/jira/browse/LUCENE-5931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14138564#comment-14138564
 ] 

Vitaly Funstein commented on LUCENE-5931:
-

Mike/Robert,

I have a follow-up question. I have backported the fix to 4.6 and now I believe 
I am seeing another serious issue here. :(

If the old reader passed in to {{DirectoryReader.openIfChanged(DirectoryReader, 
IndexCommit)}} is actually an NRT reader, then it seems that if there is 
unflushed/uncommitted data in the associated writer's buffers, in particular 
deletes, the returned reader will see those changes - thus violating the intent 
of opening the index at just the commit point we wanted, frozen in time. Here's 
my original test case modified to show the problem:

{code}
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertTrue;

import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field.Store;
import org.apache.lucene.document.StringField;
import org.apache.lucene.index.DirectoryReader;
import org.apache.lucene.index.IndexCommit;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.index.IndexWriterConfig;
import org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy;
import org.apache.lucene.index.ReaderManager;
import org.apache.lucene.index.SnapshotDeletionPolicy;
import org.apache.lucene.index.Term;
import org.apache.lucene.search.IndexSearcher;
import org.apache.lucene.search.Query;
import org.apache.lucene.search.TermQuery;
import org.apache.lucene.search.TopDocs;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.FSDirectory;
import org.apache.lucene.util.Version;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;

import java.io.File;

public class CommitReuseTest {

  private final File path = new File(indexDir);
  private IndexWriter writer;
  private final SnapshotDeletionPolicy snapshotter = new 
SnapshotDeletionPolicy(new KeepOnlyLastCommitDeletionPolicy());
 
  @Before
  public void initIndex() throws Exception {
path.mkdirs();
IndexWriterConfig idxWriterCfg = new IndexWriterConfig(Version.LUCENE_46, 
null);
idxWriterCfg.setIndexDeletionPolicy(snapshotter);
idxWriterCfg.setInfoStream(System.out);
   
Directory dir = FSDirectory.open(path);
writer = new IndexWriter(dir, idxWriterCfg);
   
writer.commit(); // make sure all index metadata is written out
  }
 
  @After
  public void stop() throws Exception {
writer.close();
  }

  @Test
  public void test() throws Exception {
Document doc;
ReaderManager rm = new ReaderManager(writer, true);
   
// Index some data
for (int i = 0; i  100; i++) {
  doc = new Document();
  doc.add(new StringField(key- + i, ABC, Store.YES));
  writer.addDocument(doc);
}
   
writer.commit();
   
IndexCommit ic1 = snapshotter.snapshot();
   
doc = new Document();
doc.add(new StringField(key- + 0, AAA, Store.YES));
writer.updateDocument(new Term(key- + 0, ABC), doc);

rm.maybeRefreshBlocking();
DirectoryReader latest = rm.acquire();
assertTrue(latest.hasDeletions());
   
// This reader will be used for searching against commit point 1
DirectoryReader searchReader = DirectoryReader.openIfChanged(latest, ic1);
//assertFalse(searchReader.hasDeletions()); // XXX - this fails too!
rm.release(latest);
   
IndexSearcher s = new IndexSearcher(searchReader);
Query q = new TermQuery(new Term(key-0, ABC));
TopDocs td = s.search(q, 10);
assertEquals(1, td.totalHits);
   
searchReader.close();
rm.close();
snapshotter.release(ic1);
  }
}
{code}

Note, that if I comment out the {{updateDocument()}} call, the test passes. 
Also, if you only have one entry in the index and not enough, then it appears 
that while refreshing the NRT reader, the segment containing just the single 
delete will be removed, making it look like the test passes:

{noformat}
IW 0 [Wed Sep 17 22:32:47 PDT 2014; main]: drop 100% deleted segments: 
_4(4.6):c1/1
{noformat}

This output does not appear when running the code above, unchanged. Hope this 
helps... I can't make further headway myself though.

 DirectoryReader.openIfChanged(oldReader, commit) incorrectly assumes given 
 commit point has deletes/field updates
 -

 Key: LUCENE-5931
 URL: https://issues.apache.org/jira/browse/LUCENE-5931
 Project: Lucene - Core
  Issue Type: Bug
  Components: core/index
Affects Versions: 4.6.1
Reporter: Vitaly Funstein
Assignee: Michael McCandless
Priority: Critical
 Attachments: CommitReuseTest.java, LUCENE-5931.patch, 
 LUCENE-5931.patch, LUCENE-5931.patch

[jira] [Comment Edited] (LUCENE-5931) DirectoryReader.openIfChanged(oldReader, commit) incorrectly assumes given commit point has deletes/field updates

2014-09-18 Thread Vitaly Funstein (JIRA) 

[ 
https://issues.apache.org/jira/browse/LUCENE-5931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14138564#comment-14138564
 ] 

Vitaly Funstein edited comment on LUCENE-5931 at 9/18/14 6:08 AM:
--

Mike/Robert,

I have a follow-up question. I have backported the fix to 4.6 and now I believe 
I am seeing another serious issue here. :(

If the old reader passed in to {{DirectoryReader.openIfChanged(DirectoryReader, 
IndexCommit)}} is actually an NRT reader, then it seems that if there is 
unflushed/uncommitted data in the associated writer's buffers, in particular 
deletes, the returned reader will see those changes - thus violating the intent 
of opening the index at just the commit point we wanted, frozen in time. Here's 
my original test case modified to show the problem:

{code}
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertTrue;

import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field.Store;
import org.apache.lucene.document.StringField;
import org.apache.lucene.index.DirectoryReader;
import org.apache.lucene.index.IndexCommit;
import org.apache.lucene.index.IndexWriter;
import org.apache.lucene.index.IndexWriterConfig;
import org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy;
import org.apache.lucene.index.ReaderManager;
import org.apache.lucene.index.SnapshotDeletionPolicy;
import org.apache.lucene.index.Term;
import org.apache.lucene.search.IndexSearcher;
import org.apache.lucene.search.Query;
import org.apache.lucene.search.TermQuery;
import org.apache.lucene.search.TopDocs;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.FSDirectory;
import org.apache.lucene.util.Version;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;

import java.io.File;

public class CommitReuseTest {

  private final File path = new File(indexDir);
  private IndexWriter writer;
  private final SnapshotDeletionPolicy snapshotter = new 
SnapshotDeletionPolicy(new KeepOnlyLastCommitDeletionPolicy());
 
  @Before
  public void initIndex() throws Exception {
path.mkdirs();
IndexWriterConfig idxWriterCfg = new IndexWriterConfig(Version.LUCENE_46, 
null);
idxWriterCfg.setIndexDeletionPolicy(snapshotter);
idxWriterCfg.setInfoStream(System.out);
   
Directory dir = FSDirectory.open(path);
writer = new IndexWriter(dir, idxWriterCfg);
   
writer.commit(); // make sure all index metadata is written out
  }
 
  @After
  public void stop() throws Exception {
writer.close();
  }

  @Test
  public void test() throws Exception {
Document doc;
ReaderManager rm = new ReaderManager(writer, true);
   
// Index some data
for (int i = 0; i  100; i++) {
  doc = new Document();
  doc.add(new StringField(key- + i, ABC, Store.YES));
  writer.addDocument(doc);
}
   
writer.commit();
   
IndexCommit ic1 = snapshotter.snapshot();
   
doc = new Document();
doc.add(new StringField(key- + 0, AAA, Store.YES));
writer.updateDocument(new Term(key- + 0, ABC), doc);

rm.maybeRefreshBlocking();
DirectoryReader latest = rm.acquire();
assertTrue(latest.hasDeletions());
   
// This reader will be used for searching against commit point 1
DirectoryReader searchReader = DirectoryReader.openIfChanged(latest, ic1);
//assertFalse(searchReader.hasDeletions()); // XXX - this fails too!
rm.release(latest);
   
IndexSearcher s = new IndexSearcher(searchReader);
Query q = new TermQuery(new Term(key-0, ABC));
TopDocs td = s.search(q, 10);
assertEquals(1, td.totalHits);
   
searchReader.close();
rm.close();
snapshotter.release(ic1);
  }
}
{code}

Note, that if I comment out the {{updateDocument()}} call, the test passes. 
Also, if you only have one entry in the index, then it appears that while 
refreshing the NRT reader, the segment containing just the single delete will 
be removed, making it look like the test passes:

{noformat}
IW 0 [Wed Sep 17 22:32:47 PDT 2014; main]: drop 100% deleted segments: 
_4(4.6):c1/1
{noformat}

This output does not appear when running the code above, unchanged. Hope this 
helps... I can't make further headway myself though.


was (Author: vfunstein):
Mike/Robert,

I have a follow-up question. I have backported the fix to 4.6 and now I believe 
I am seeing another serious issue here. :(

If the old reader passed in to {{DirectoryReader.openIfChanged(DirectoryReader, 
IndexCommit)}} is actually an NRT reader, then it seems that if there is 
unflushed/uncommitted data in the associated writer's buffers, in particular 
deletes, the returned reader will see those changes - thus violating the intent 
of opening the index at just the commit point we wanted, frozen in time. Here's 
my original test case modified to show the problem:

{code}
import static org.junit.Assert.assertEquals;

[jira] [Created] (LUCENE-5931) DirectoryReader.openIfChanged(oldReader, commit) incorrectly assumes given commit points has deletes/field updates

2014-09-09 Thread Vitaly Funstein (JIRA) 
Vitaly Funstein created LUCENE-5931:
---

 Summary: DirectoryReader.openIfChanged(oldReader, commit) 
incorrectly assumes given commit points has deletes/field updates
 Key: LUCENE-5931
 URL: https://issues.apache.org/jira/browse/LUCENE-5931
 Project: Lucene - Core
  Issue Type: Bug
  Components: core/index
Affects Versions: 4.6.1
Reporter: Vitaly Funstein
Priority: Critical


{{StandardDirectoryReader}} assumes that the segments from commit point have 
deletes, when they may not, yet the original SegmentReader for the segment that 
we are trying to reuse does. This is evident when running attached JUnit test 
case with asserts enabled (default): 

{noformat}
java.lang.AssertionError
at 
org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:188)
at 
org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:326)
at 
org.apache.lucene.index.StandardDirectoryReader$2.doBody(StandardDirectoryReader.java:320)
at 
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:702)
at 
org.apache.lucene.index.StandardDirectoryReader.doOpenFromCommit(StandardDirectoryReader.java:315)
at 
org.apache.lucene.index.StandardDirectoryReader.doOpenNoWriter(StandardDirectoryReader.java:311)
at 
org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:262)
at 
org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:183)
{noformat}

or, if asserts are disabled then it falls through into NPE:

{noformat}
java.lang.NullPointerException
at java.io.File.init(File.java:305)
at 
org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:80)
at 
org.apache.lucene.codecs.lucene40.BitVector.init(BitVector.java:327)
at 
org.apache.lucene.codecs.lucene40.Lucene40LiveDocsFormat.readLiveDocs(Lucene40LiveDocsFormat.java:90)
at org.apache.lucene.index.SegmentReader.init(SegmentReader.java:131)
at 
org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:194)
at 
org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:326)
at 
org.apache.lucene.index.StandardDirectoryReader$2.doBody(StandardDirectoryReader.java:320)
at 
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:702)
at 
org.apache.lucene.index.StandardDirectoryReader.doOpenFromCommit(StandardDirectoryReader.java:315)
at 
org.apache.lucene.index.StandardDirectoryReader.doOpenNoWriter(StandardDirectoryReader.java:311)
at 
org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:262)
at 
org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:183)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-5931) DirectoryReader.openIfChanged(oldReader, commit) incorrectly assumes given commit points has deletes/field updates

2014-09-09 Thread Vitaly Funstein (JIRA) 

 [ 
https://issues.apache.org/jira/browse/LUCENE-5931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vitaly Funstein updated LUCENE-5931:

Attachment: CommitReuseTest.java

 DirectoryReader.openIfChanged(oldReader, commit) incorrectly assumes given 
 commit points has deletes/field updates
 --

 Key: LUCENE-5931
 URL: https://issues.apache.org/jira/browse/LUCENE-5931
 Project: Lucene - Core
  Issue Type: Bug
  Components: core/index
Affects Versions: 4.6.1
Reporter: Vitaly Funstein
Priority: Critical
 Attachments: CommitReuseTest.java


 {{StandardDirectoryReader}} assumes that the segments from commit point have 
 deletes, when they may not, yet the original SegmentReader for the segment 
 that we are trying to reuse does. This is evident when running attached JUnit 
 test case with asserts enabled (default): 
 {noformat}
 java.lang.AssertionError
   at 
 org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:188)
   at 
 org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:326)
   at 
 org.apache.lucene.index.StandardDirectoryReader$2.doBody(StandardDirectoryReader.java:320)
   at 
 org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:702)
   at 
 org.apache.lucene.index.StandardDirectoryReader.doOpenFromCommit(StandardDirectoryReader.java:315)
   at 
 org.apache.lucene.index.StandardDirectoryReader.doOpenNoWriter(StandardDirectoryReader.java:311)
   at 
 org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:262)
   at 
 org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:183)
 {noformat}
 or, if asserts are disabled then it falls through into NPE:
 {noformat}
 java.lang.NullPointerException
   at java.io.File.init(File.java:305)
   at 
 org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:80)
   at 
 org.apache.lucene.codecs.lucene40.BitVector.init(BitVector.java:327)
   at 
 org.apache.lucene.codecs.lucene40.Lucene40LiveDocsFormat.readLiveDocs(Lucene40LiveDocsFormat.java:90)
   at org.apache.lucene.index.SegmentReader.init(SegmentReader.java:131)
   at 
 org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:194)
   at 
 org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:326)
   at 
 org.apache.lucene.index.StandardDirectoryReader$2.doBody(StandardDirectoryReader.java:320)
   at 
 org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:702)
   at 
 org.apache.lucene.index.StandardDirectoryReader.doOpenFromCommit(StandardDirectoryReader.java:315)
   at 
 org.apache.lucene.index.StandardDirectoryReader.doOpenNoWriter(StandardDirectoryReader.java:311)
   at 
 org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:262)
   at 
 org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:183)
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-5931) DirectoryReader.openIfChanged(oldReader, commit) incorrectly assumes given commit point has deletes/field updates

2014-09-09 Thread Vitaly Funstein (JIRA) 

 [ 
https://issues.apache.org/jira/browse/LUCENE-5931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vitaly Funstein updated LUCENE-5931:

Summary: DirectoryReader.openIfChanged(oldReader, commit) incorrectly 
assumes given commit point has deletes/field updates  (was: 
DirectoryReader.openIfChanged(oldReader, commit) incorrectly assumes given 
commit points has deletes/field updates)

 DirectoryReader.openIfChanged(oldReader, commit) incorrectly assumes given 
 commit point has deletes/field updates
 -

 Key: LUCENE-5931
 URL: https://issues.apache.org/jira/browse/LUCENE-5931
 Project: Lucene - Core
  Issue Type: Bug
  Components: core/index
Affects Versions: 4.6.1
Reporter: Vitaly Funstein
Priority: Critical
 Attachments: CommitReuseTest.java


 {{StandardDirectoryReader}} assumes that the segments from commit point have 
 deletes, when they may not, yet the original SegmentReader for the segment 
 that we are trying to reuse does. This is evident when running attached JUnit 
 test case with asserts enabled (default): 
 {noformat}
 java.lang.AssertionError
   at 
 org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:188)
   at 
 org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:326)
   at 
 org.apache.lucene.index.StandardDirectoryReader$2.doBody(StandardDirectoryReader.java:320)
   at 
 org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:702)
   at 
 org.apache.lucene.index.StandardDirectoryReader.doOpenFromCommit(StandardDirectoryReader.java:315)
   at 
 org.apache.lucene.index.StandardDirectoryReader.doOpenNoWriter(StandardDirectoryReader.java:311)
   at 
 org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:262)
   at 
 org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:183)
 {noformat}
 or, if asserts are disabled then it falls through into NPE:
 {noformat}
 java.lang.NullPointerException
   at java.io.File.init(File.java:305)
   at 
 org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:80)
   at 
 org.apache.lucene.codecs.lucene40.BitVector.init(BitVector.java:327)
   at 
 org.apache.lucene.codecs.lucene40.Lucene40LiveDocsFormat.readLiveDocs(Lucene40LiveDocsFormat.java:90)
   at org.apache.lucene.index.SegmentReader.init(SegmentReader.java:131)
   at 
 org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:194)
   at 
 org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:326)
   at 
 org.apache.lucene.index.StandardDirectoryReader$2.doBody(StandardDirectoryReader.java:320)
   at 
 org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:702)
   at 
 org.apache.lucene.index.StandardDirectoryReader.doOpenFromCommit(StandardDirectoryReader.java:315)
   at 
 org.apache.lucene.index.StandardDirectoryReader.doOpenNoWriter(StandardDirectoryReader.java:311)
   at 
 org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:262)
   at 
 org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:183)
 {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (LUCENE-5912) Non-NRT directory readers don't reuse segments maintained IndexWriter's segment reader pool

2014-08-28 Thread Vitaly Funstein (JIRA) 
Vitaly Funstein created LUCENE-5912:
---

 Summary: Non-NRT directory readers don't reuse segments maintained 
IndexWriter's segment reader pool
 Key: LUCENE-5912
 URL: https://issues.apache.org/jira/browse/LUCENE-5912
 Project: Lucene - Core
  Issue Type: Improvement
  Components: core/index
Affects Versions: 4.6.1
Reporter: Vitaly Funstein


Currently, if you attempt to open a reader into an index at a specific commit 
point, it will always behave as though it's opening a completely new index - 
even if one were to use the {{DirectoryReader.openIfChanged(DirectoryReader, 
IndexCommit)}} API, and pass in an NRT reader instance. What should ideally 
happen here is that the SegmentReader pool managed by IndexWriter linked to the 
NRT reader gets reused for the commit point open as much as possible, to avoid 
wasting heap space.

The problem becomes evident when looking at the code in DirectoryReader:

{code}
protected DirectoryReader doOpenIfChanged(final IndexCommit commit) throws 
IOException {
ensureOpen();

// If we were obtained by writer.getReader(), re-ask the
// writer to get a new reader.
if (writer != null) {
  return doOpenFromWriter(commit);
} else {
  return doOpenNoWriter(commit);
}
  }

  private DirectoryReader doOpenFromWriter(IndexCommit commit) throws 
IOException {
if (commit != null) {
  return doOpenFromCommit(commit);
}
..
{code}

Looks like the fact that a commit point is being re-opened trumps the presence 
of the associated IndexWriter.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (LUCENE-5912) Non-NRT directory readers don't reuse segments maintained IndexWriter's segment reader pool

2014-08-28 Thread Vitaly Funstein (JIRA) 

 [ 
https://issues.apache.org/jira/browse/LUCENE-5912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vitaly Funstein resolved LUCENE-5912.
-

   Resolution: Invalid
Lucene Fields:   (was: New)

Actually, I'll take it back... it looks like 
{{StandardDirectoryReader.doOpenIfChanged(SegmentInfos)}} will read through to 
the subReaders of the current reader, which in case of an NRT reader would be 
backed by the IndexWriter's reader pool.

 Non-NRT directory readers don't reuse segments maintained IndexWriter's 
 segment reader pool
 ---

 Key: LUCENE-5912
 URL: https://issues.apache.org/jira/browse/LUCENE-5912
 Project: Lucene - Core
  Issue Type: Improvement
  Components: core/index
Affects Versions: 4.6.1
Reporter: Vitaly Funstein

 Currently, if you attempt to open a reader into an index at a specific commit 
 point, it will always behave as though it's opening a completely new index - 
 even if one were to use the {{DirectoryReader.openIfChanged(DirectoryReader, 
 IndexCommit)}} API, and pass in an NRT reader instance. What should ideally 
 happen here is that the SegmentReader pool managed by IndexWriter linked to 
 the NRT reader gets reused for the commit point open as much as possible, to 
 avoid wasting heap space.
 The problem becomes evident when looking at the code in DirectoryReader:
 {code}
 protected DirectoryReader doOpenIfChanged(final IndexCommit commit) throws 
 IOException {
 ensureOpen();
 // If we were obtained by writer.getReader(), re-ask the
 // writer to get a new reader.
 if (writer != null) {
   return doOpenFromWriter(commit);
 } else {
   return doOpenNoWriter(commit);
 }
   }
   private DirectoryReader doOpenFromWriter(IndexCommit commit) throws 
 IOException {
 if (commit != null) {
   return doOpenFromCommit(commit);
 }
 ..
 {code}
 Looks like the fact that a commit point is being re-opened trumps the 
 presence of the associated IndexWriter.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org