[jira] Created: (LUCENE-2292) ByteBuffer Directory - allowing to store the index outside the heap
ByteBuffer Directory - allowing to store the index outside the heap --- Key: LUCENE-2292 URL: https://issues.apache.org/jira/browse/LUCENE-2292 Project: Lucene - Java Issue Type: New Feature Components: Store Reporter: Shay Banon A byte buffer based directory with the benefit of being able to create direct byte buffer thus storing the index outside the JVM heap. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Updated: (LUCENE-2292) ByteBuffer Directory - allowing to store the index outside the heap
[ https://issues.apache.org/jira/browse/LUCENE-2292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shay Banon updated LUCENE-2292: --- Attachment: LUCENE-2292.patch ByteBuffer Directory - allowing to store the index outside the heap --- Key: LUCENE-2292 URL: https://issues.apache.org/jira/browse/LUCENE-2292 Project: Lucene - Java Issue Type: New Feature Components: Store Reporter: Shay Banon Attachments: LUCENE-2292.patch A byte buffer based directory with the benefit of being able to create direct byte buffer thus storing the index outside the JVM heap. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-2292) ByteBuffer Directory - allowing to store the index outside the heap
[ https://issues.apache.org/jira/browse/LUCENE-2292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12840379#action_12840379 ] Shay Banon commented on LUCENE-2292: Hi, looks interesting as a replacement for RAMDirectory. This class uses ByteBuffer, which has its overhead over simple byte[], though using the same logic (if you verify it) can be used to improve the concurrency in RAMDirectory (just use byte[[). Your patch uses a sun. internal package. If you want to do something similar to MMapDirectory to release the buffer without waiting for GC, do it in the same way using reflection like in MMapDirectory. From what I know, it was there in all JDKs I worked with (its like sun.misc.Unsafe). Have you seen otherwise? If so, its a simple change (though I am not sure about the access control thingy in MMapDirectory, its a performance killer, and caching of the Method(s) make sense). ByteBuffer Directory - allowing to store the index outside the heap --- Key: LUCENE-2292 URL: https://issues.apache.org/jira/browse/LUCENE-2292 Project: Lucene - Java Issue Type: New Feature Components: Store Reporter: Shay Banon Attachments: LUCENE-2292.patch A byte buffer based directory with the benefit of being able to create direct byte buffer thus storing the index outside the JVM heap. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Updated: (LUCENE-2292) ByteBuffer Directory - allowing to store the index outside the heap
[ https://issues.apache.org/jira/browse/LUCENE-2292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shay Banon updated LUCENE-2292: --- Attachment: LUCENE-2292.patch Attached new patch, does not use sun.* package. I still cache Method since cleaning a buffer is not only done on close of the directory. ByteBuffer Directory - allowing to store the index outside the heap --- Key: LUCENE-2292 URL: https://issues.apache.org/jira/browse/LUCENE-2292 Project: Lucene - Java Issue Type: New Feature Components: Store Reporter: Shay Banon Attachments: LUCENE-2292.patch, LUCENE-2292.patch A byte buffer based directory with the benefit of being able to create direct byte buffer thus storing the index outside the JVM heap. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-2292) ByteBuffer Directory - allowing to store the index outside the heap
[ https://issues.apache.org/jira/browse/LUCENE-2292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12840394#action_12840394 ] Shay Banon commented on LUCENE-2292: By the way, an implementation note. I thought about preallocating a large direct buffer and then splicing it into chunks, but currently I think that the complexity (and overhead in maintaining splicing locations) is not really needed and the current caching should do the trick (with the ability to control both the buffer size and the cache size). ByteBuffer Directory - allowing to store the index outside the heap --- Key: LUCENE-2292 URL: https://issues.apache.org/jira/browse/LUCENE-2292 Project: Lucene - Java Issue Type: New Feature Components: Store Reporter: Shay Banon Attachments: LUCENE-2292.patch, LUCENE-2292.patch A byte buffer based directory with the benefit of being able to create direct byte buffer thus storing the index outside the JVM heap. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Created: (LUCENE-1637) Getting an IndexReader from a committed IndexWriter
Getting an IndexReader from a committed IndexWriter --- Key: LUCENE-1637 URL: https://issues.apache.org/jira/browse/LUCENE-1637 Project: Lucene - Java Issue Type: Improvement Components: Index Affects Versions: 2.9 Reporter: Shay Banon I just had a look at the job done in IndexWriter in order to get an IndexReader with all the current ongoing changes done using the IndexWriter. This feature is very useful, and I was wondering if another feature, which (I think) is simple to implement (compared to the previous one) might make sense. Many times, an application opens an IndexWriter, does whatever changes it does, and then commits the changes. It would be nice to get an IndexReader (read only one is fine) that corresponds to the committed (or even closed) IndexWriter. This will allow for a cache of IndexReader that is already used to be updated with a fresh IndexReader, without the need to reopen one (which should be slower than opening one based on the IndexWriter information). The main difference is the fact that the mentioned IndexReader could still be reopened without the need to throw an AlreadyClosedException. More information can be found here: http://www.nabble.com/Getting-an-IndexReader-from-a-committed-IndexWriter-td23551978.html -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org For additional commands, e-mail: java-dev-h...@lucene.apache.org
[jira] Commented: (LUCENE-1239) IndexWriter deadlock when using ConcurrentMergeScheduler
[ https://issues.apache.org/jira/browse/LUCENE-1239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12579874#action_12579874 ] Shay Banon commented on LUCENE-1239: Yea, it looks like it is my bad, great catch!. While trying to create a better scheduler (at least in terms of reusing threads instead of creating them), I wondered if there is a chance that the current scheduler can be enhanced to support an extension point for that... . I can give such a refactoring a go if you think it make sense. IndexWriter deadlock when using ConcurrentMergeScheduler Key: LUCENE-1239 URL: https://issues.apache.org/jira/browse/LUCENE-1239 Project: Lucene - Java Issue Type: Bug Components: Index Affects Versions: 2.3.1 Environment: Compass 2.0.0M3 (nightly build #57), Lucene 2.3.1, Spring Framework 2.0.7.0 Reporter: Michael Lossos Assignee: Michael McCandless I'm trying to update our application from Compass 2.0.0M1 with Lucene 2.2 to Compass 2.0.0M3 (latest build) with Lucene 2.3.1. I'm holding all other things constant and only changing the Compass and Lucene jars. I'm recreating the search index for our data and seeing deadlock in Lucene's IndexWriter. It appears to be waiting on a signal from the merge thread. I've tried creating a simple reproduction case for this but to no avail. Doing the exact same steps with Compass 2.0.0M1 and Lucene 2.2 has no problems and recreates our search index. That is to say, it's not our code. In particular, the main thread performing the commit (Lucene document save) from Compass is calling Lucene's IndexWriter.optimize(). We're using Compass's ExecutorMergeScheduler to handle the merging, and it is calling IndexWriter.merge(). The main thread in IndexWriter.optimize() enters the wait() at the bottom of that method and is never notified. I can't tell if this is because optimizeMergesPending() is returning true incorrectly, or if IndexWriter.merge()'s notifyAll() is being called prematurely. Looking at the code, it doesn't seem possible for IndexWriter.optimize() to be waiting and miss a notifyAll(), and Lucene's IndexWriter.merge() was recently fixed to always call notifyAll() even on exceptions -- that is all the relevant IndexWriter code looks properly synchronized. Nevertheless, I'm seeing the deadlock behavior described, and it's reproducible using our app and our test data set. Could someone familiar with IndexWriter's synchronization code take another look at it? I'm sorry that I can't give you a simple reproduction test case. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[jira] Updated: (LUCENE-511) New BufferedIndexOutput optimization fails to update bufferStart
[ http://issues.apache.org/jira/browse/LUCENE-511?page=all ] Shay Banon updated LUCENE-511: -- Attachment: BufferedIndexOutput.java New BufferedIndexOutput optimization fails to update bufferStart Key: LUCENE-511 URL: http://issues.apache.org/jira/browse/LUCENE-511 Project: Lucene - Java Type: Bug Components: Store Versions: 1.9 Reporter: Shay Banon Priority: Critical Attachments: BufferedIndexOutput.java, RAMOutputTest.java New BufferIndexOutput optimization of writeBytes fails to update bufferStart under some conditions. Test case and fix attached. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
