there is one case where MMAP does not beat RAM, initial warm-up after process
restart. With MMAP it can take a while before you get up to speed. MMAP with
reopen is the best, if you run without restart.
----- Original Message ----
> From: Uwe Schindler <u...@thetaphi.de>
> To: java-user@lucene.apache.org
> Sent: Wednesday, 10 June, 2009 0:24:35
> Subject: RE: Reloading RAM Directory from updated FS Directory
>
> Hi Greg & Kay,
>
> > Lucene (IndexSearchers / IndexReaders) has the notion of cache as well
> > so you need to check if you really want a 100% replication of
> > RAMDirectory / FSDirectory as well concurrently in memory. Have you
> > tested with the FieldCache policies before moving onto the RAMDirectory
> > based backup solution.
>
> As you have so big indexes, I think you use a 64 bit machine and 64 bit JVM.
> I would recommend you in this case to use MMapDirectory instead of copying
> the contents to a RAMDirectory. MMapDirectory works like FSDirectory, but it
> does not read the files via normal RandomAccessFile actions, but it maps the
> file into the 64 bit address space. In principle this is like a swap file,
> the contents of the file are seen like memory. The operating system kernel
> then uses the same strategies to map parts of the index into real RAM, just
> like a swap file. For Lucene it looks like a RAMDirectory backed by a file.
>
> Changes to the underlying index can be seen immediately, and changes can
> reloaded using IndexReader.reopen(), which is much faster than reopening the
> index in complete.
>
> Kay Kay: Instantiating an IndexSearcher on top of a IndexReader costs
> nothing (its just a wrapper around the IndexReader). The costly part is
> opening the IndexReader. So using reopen() and wrapping this always with a
> fresh IndexReader is faster than creating an IndexSearcher using the
> directory path.
>
> Uwe
>
> > Diamond, Greg wrote:
> > > Hi All -
> > >
> > > What is the best way to load a RAM Directory from a FS Directory, and
> > periodically reload the RAM Directory to pick up new documents?
> > >
> > > The scenario I have is I create several large directories which I create
> > to a file system, then load them into ram for faster searching.
> > > They takes several hours to create, so i want to retain a file copy so
> > in the event of a service/server crash or reboot, they can load again
> > > in a few seconds. Once a day I append or recreate the FS Directories
> > with new entries, and then reload the RAM Directories to pick up the
> > > new entries.
> > >
> > > Problem is, what I am doing seems to cause a memory leak. The
> > directories take ~12 GB of RAM to load, but bloats
> > > to 24 GB (the -Xmx setting) after a few reloads and stays there.
> > >
> > > // Map to hold single instance of the IndexSearcher, 1 per RAM
> > Directory, to be reused across requests.
> > > private static final Mapsearchers = new
> > HashMap();
> > >
> > > // Creates the IndexSearcher and stores it in the static map.
> > > public static void load(IndexName index, ...) {
> > > // sync... etc
> > > RAMDirectory dir = new
> > RAMDirectory(FSDirectory.getDirectory(directoryPath);
> > > IndexReader reader = IndexReader.open(dir, true);
> > > IndexSearcher searcher = new IndexSearcher(reader);
> > > searchers.put(index, searcher);
> > > }
> > >
> > > public static IndexSearcher get(IndexName index, ...) {
> > > // sync... etc
> > > return searchers.get(index);
> > > }
> > >
> > > // After an indexing service appends the existing FS Directory, this
> > reloads it.
> > > public static void reload(IndexName index, ...) {
> > > // sync... etc
> > > IndexSearcher searcher = searchers.get(index);
> > > FSDirectory fsDirectory =
> > FSDirectory.getDirectory(directoryPath);
> > > Directory ramDirectory =
> > searcher.getIndexReader().directory();
> > > Directory.copy(fsDirectory, ramDirectory, false);
> > > }
> > >
> > > I've tries some theme and variations on this with the same issue.
> > >
> > > TIA!
> > >
> > > gd
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
> > > For additional commands, e-mail: java-user-h...@lucene.apache.org
> > >
> > >
> > >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
> > For additional commands, e-mail: java-user-h...@lucene.apache.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
> For additional commands, e-mail: java-user-h...@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org
