RE: In memory index (current status in Lucene)

2013-07-06 Thread Uwe Schindler
- > From: Ramkumar R. Aiyengar [mailto:andyetitmo...@gmail.com] > Sent: Thursday, July 04, 2013 10:14 PM > To: java-user@lucene.apache.org > Subject: Re: In memory index (current status in Lucene) > > Have you tried using MMapDirectory over a RAM disk (assuming you are on > Lin

Re: In memory index (current status in Lucene)

2013-07-06 Thread Sanne Grinovero
There is a decent implementation for a fully in-memory Directory in the Infinispan project: https://github.com/infinispan/infinispan/tree/master/lucene This is however not taking advantage of off-heap buffers but storing the index in the heap itself; the reason being that Infinispan can in this ca

Re: In memory index (current status in Lucene)

2013-07-04 Thread Adrien Grand
On Tue, Jul 2, 2013 at 10:09 AM, Toke Eskildsen wrote: > I wonder if Java's ByteBuffer could be used to make a more GC-friendly > RAMDirectory? For the record, there is an open issue about it: https://issues.apache.org/jira/browse/LUCENE-2292. -- Adrien

Re: In memory index (current status in Lucene)

2013-07-04 Thread Ramkumar R. Aiyengar
Have you tried using MMapDirectory over a RAM disk (assuming you are on Linux)? You can avoid writing to disk (and thus the other ways to get to it persistently as Steven mentions), but still MMap it. On 1 Jul 2013 22:41, "Lance Norskog" wrote: > My current open source project is a Directory that

Re: In memory index (current status in Lucene)

2013-07-02 Thread Toke Eskildsen
On Mon, 2013-07-01 at 16:07 +0200, Emmanuel Espina wrote: > Just to add to this conversation, I found an interesting link to > Mike's blog about memory resident indexes (using another virtual > machine) > http://blog.mikemccandless.com/2012/07/lucene-index-in-ram-with-azuls-zing-jvm.html Testing

Re: In memory index (current status in Lucene)

2013-07-01 Thread Steven Schlansker
On Jul 1, 2013, at 2:41 PM, Lance Norskog wrote: > My current open source project is a Directory that is just like RAMDirectory, > but everything is memory-mapped. The idea is it creates a disk file, opens > it, and immediately deletes the file. The file still exists until the > IndexReader/W

Re: In memory index (current status in Lucene)

2013-07-01 Thread Lance Norskog
My current open source project is a Directory that is just like RAMDirectory, but everything is memory-mapped. The idea is it creates a disk file, opens it, and immediately deletes the file. The file still exists until the IndexReader/Writer/Searcher closes it. But, it cannot be found from the

Re: In memory index (current status in Lucene)

2013-07-01 Thread Emmanuel Espina
Hi Erick! Nice to hear from you again! From time to time my interest in these "Lucene things" returns and I do some experiments :p Just to add to this conversation, I found an interesting link to Mike's blog about memory resident indexes (using another virtual machine) http://blog.mikemccandless.

Re: In memory index (current status in Lucene)

2013-07-01 Thread Erick Erickson
Hey Emma! It's been a while Building on what Steven said, here's Uwe's blog on MMapDirectory and Lucene: http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html I've always considered RAMDirectory for rather restricted use-cases. I.e. if I know without doubt that the index is

Re: In memory index (current status in Lucene)

2013-06-28 Thread Steven Schlansker
On Jun 28, 2013, at 2:29 PM, Emmanuel Espina wrote: > I'm building a distributed index (mostly as a reasearch project for > school) and I'm evaluating indexing the entire collection in memory > (like google, facebook and others have done years ago). The obvious > reason for this is performance c