Will do. On Tue, Feb 7, 2012 at 12:52 AM, Michael McCandless < luc...@mikemccandless.com> wrote:
> You tell NRTCachingDirectory how much RAM it's allowed to use, and it > then caches newly flushed segments in a private RAMDirectory. > > But you should first test performance w/o it (after removing the > commit calls). NRT is very fast... > > Mike McCandless > > http://blog.mikemccandless.com > > On Mon, Feb 6, 2012 at 11:46 AM, Cheng <zhoucheng2...@gmail.com> wrote: > > Good point. I should remove the commits. > > > > Any difference between NRTCashingDirectory and RAMDirectory? how to > define > > the "small"? > > > > On Tue, Feb 7, 2012 at 12:42 AM, Michael McCandless < > > luc...@mikemccandless.com> wrote: > > > >> You shouldn't call IW.commit when using NRT; that's the point of NRT > >> (making changes visible w/o calling commit). > >> > >> Only call commit when you require that all changes be durable (surive > >> OS / JVM crash, power loss, etc.) on disk. > >> > >> Also, you can use NRTCachingDirectory which acts like RAMDirectory for > >> small flushed segments. > >> > >> Mike McCandless > >> > >> http://blog.mikemccandless.com > >> > >> On Mon, Feb 6, 2012 at 10:45 AM, Cheng <zhoucheng2...@gmail.com> wrote: > >> > Uwe, when I meant speed is slow, I didn't refer to instant visibility > of > >> > changes, but that the changes may be synchronized with FSDirectory > when I > >> > use writer.commit(). > >> > > >> > When I use RAMDirectory, the writer.commit() seems much faster than > using > >> > NRTManager built upon FSDirectory. So, I am guessing the difference is > >> the > >> > index synchronization. > >> > > >> > > >> > > >> > On Mon, Feb 6, 2012 at 11:40 PM, Uwe Schindler <u...@thetaphi.de> > wrote: > >> > > >> >> Please review the following articles about NRT, absolutely instant > >> updates > >> >> that are visible as they are done are almost impossible (even with > >> >> RAMDirectory): > >> >> > >> >> http://goo.gl/mzAHt > >> >> http://goo.gl/5RoPx > >> >> http://goo.gl/vSJ7x > >> >> > >> >> Uwe > >> >> > >> >> ----- > >> >> Uwe Schindler > >> >> H.-H.-Meier-Allee 63, D-28213 Bremen > >> >> http://www.thetaphi.de > >> >> eMail: u...@thetaphi.de > >> >> > >> >> > -----Original Message----- > >> >> > From: Cheng [mailto:zhoucheng2...@gmail.com] > >> >> > Sent: Monday, February 06, 2012 4:27 PM > >> >> > To: java-user@lucene.apache.org > >> >> > Subject: Re: Configure writer to write to FSDirectory? > >> >> > > >> >> > Ian, > >> >> > > >> >> > I encountered an issue that I need to frequently update the index. > The > >> >> > NRTManager seems not very helpful on this front as the speed is > slower > >> >> than > >> >> > RAMDirectory is used. > >> >> > > >> >> > Any improvement advice? > >> >> > > >> >> > > >> >> > > >> >> > On Mon, Feb 6, 2012 at 10:24 PM, Cheng <zhoucheng2...@gmail.com> > >> wrote: > >> >> > > >> >> > > That really helps! I will try it out. > >> >> > > > >> >> > > Thanks. > >> >> > > > >> >> > > > >> >> > > On Mon, Feb 6, 2012 at 10:12 PM, Ian Lea <ian....@gmail.com> > wrote: > >> >> > > > >> >> > >> You would use NRTManagerReopenThread as a standalone thread, not > >> >> > >> plugged into your Executor stuff. It is a utility class which > you > >> >> > >> don't have to use. See the javadocs. > >> >> > >> > >> >> > >> But in your case I'd use it, to start with anyway. Fire it up > with > >> >> > >> suitable settings and forget about it, except to call close() > >> >> > >> eventually. Once you've got things up and running you can tweak > >> >> > >> things as much as you want but you appear to be having trouble > >> >> > >> getting up and running. > >> >> > >> > >> >> > >> So ... somewhere in the initialisation code of your app, create > an > >> >> > >> IndexWriter, NRTManager + ReopenThread and SearcherManager as > >> >> > >> outlined before. Then pass the NRTManager to any/all write > methods > >> >> > >> or threads and the SearcherManager instance to any/all search > >> methods > >> >> > >> or threads and you're done. If you want to use threads that are > >> part > >> >> > >> of your ExecutorService, fine. Just wrap it all together in > >> whatever > >> >> > >> combination of Thread or Runnable instances you want. > >> >> > >> > >> >> > >> > >> >> > >> Does that help? > >> >> > >> > >> >> > >> > >> >> > >> -- > >> >> > >> Ian. > >> >> > >> > >> >> > >> > >> >> > >> > I don't understand this following portion: > >> >> > >> > > >> >> > >> > IndexWriter iw = new IndexWriter(whatever - some standard disk > >> >> > >> > index); NRTManager nrtm = new NRTManager(iw, null); > >> >> > >> > NRTManagerReopenThread ropt = new NRTManagerReopenThread(nrtm, > >> >> > >> > ...); ropt.setXxx(...); .... > >> >> > >> > ropt.start(); > >> >> > >> > > >> >> > >> > I have a java ExecutorServices instance running which take > care > >> of > >> >> > >> > my > >> >> > >> own > >> >> > >> > applications. I don't know how this NRTManagerReopenThread > works > >> >> > >> > with my own ExecutorService instance. > >> >> > >> > > >> >> > >> > Can both work together? How can the NRTManagerReopenThread > >> >> > instance > >> >> > >> ropt be > >> >> > >> > plugged into my own multithreading framework? > >> >> > >> > > >> >> > >> > On Mon, Feb 6, 2012 at 8:17 PM, Ian Lea <ian....@gmail.com> > >> wrote: > >> >> > >> > > >> >> > >> >> If you can use NRTManager and SearcherManager things should > be > >> >> > >> >> easy and blazingly fast rather than unbearably slow. The > latter > >> >> > >> >> phrase is not one often associated with lucene. > >> >> > >> >> > >> >> > >> >> IndexWriter iw = new IndexWriter(whatever - some standard > disk > >> >> > >> >> index); NRTManager nrtm = new NRTManager(iw, null); > >> >> > >> >> NRTManagerReopenThread ropt = new > >> >> > NRTManagerReopenThread(nrtm, > >> >> > >> >> ...); ropt.setXxx(...); ... > >> >> > >> >> ropt.start(); > >> >> > >> >> > >> >> > >> >> SearcherManager srchm = nrtm.getSearcherManager(b); > >> >> > >> >> > >> >> > >> >> Then add docs to your index via nrtm.addDocument(d), update > with > >> >> > >> >> nrtm.updateDocument(...), and to search use > >> >> > >> >> > >> >> > >> >> IndexSearcher searcher = srchm.acquire(); try { search ... > >> >> > >> >> } finally { > >> >> > >> >> srchm.release(searcher); > >> >> > >> >> } > >> >> > >> >> > >> >> > >> >> All thread safe so you don't have to worry about any > >> complications > >> >> > >> >> there. And I bet it'll be blindingly fast. > >> >> > >> >> > >> >> > >> >> Don't forget to close() things down at the end. > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> -- > >> >> > >> >> Ian. > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> On Mon, Feb 6, 2012 at 12:15 AM, Cheng < > zhoucheng2...@gmail.com > >> > > >> >> > >> wrote: > >> >> > >> >> > I was trying to, but don't know how to even I read some of > >> your > >> >> > >> blogs. > >> >> > >> >> > > >> >> > >> >> > On Sun, Feb 5, 2012 at 10:22 PM, Michael McCandless < > >> >> > >> >> > luc...@mikemccandless.com> wrote: > >> >> > >> >> > > >> >> > >> >> >> Are you using near-real-time readers? > >> >> > >> >> >> > >> >> > >> >> >> (IndexReader.open(IndexWriter)) > >> >> > >> >> >> > >> >> > >> >> >> Mike McCandless > >> >> > >> >> >> > >> >> > >> >> >> http://blog.mikemccandless.com > >> >> > >> >> >> > >> >> > >> >> >> On Sun, Feb 5, 2012 at 9:03 AM, Cheng < > >> zhoucheng2...@gmail.com> > >> >> > >> wrote: > >> >> > >> >> >> > Hi Uwe, > >> >> > >> >> >> > > >> >> > >> >> >> > My challenge is that I need to update/modify the indexes > >> >> > >> frequently > >> >> > >> >> while > >> >> > >> >> >> > providing the search capability. I was trying to use > >> >> > >> >> >> > FSDirectory, > >> >> > >> but > >> >> > >> >> >> found > >> >> > >> >> >> > out that the reading and writing from/to FSDirectory is > >> >> > >> >> >> > unbearably > >> >> > >> >> slow. > >> >> > >> >> >> So > >> >> > >> >> >> > I now am trying the RAMDirectory, which is fast. > >> >> > >> >> >> > > >> >> > >> >> >> > I don't know of MMapDirectory, and wonder if it is as > fast > >> >> > >> >> >> > as > >> >> > >> >> >> RAMDirectory. > >> >> > >> >> >> > > >> >> > >> >> >> > > >> >> > >> >> >> > On Sun, Feb 5, 2012 at 4:14 PM, Uwe Schindler > >> >> > >> >> >> > <u...@thetaphi.de> > >> >> > >> >> wrote: > >> >> > >> >> >> > > >> >> > >> >> >> >> Hi Cheng, > >> >> > >> >> >> >> > >> >> > >> >> >> >> It seems that you use a RAMDirectory for *caching*, > >> >> > >> >> >> >> otherwise it > >> >> > >> >> makes > >> >> > >> >> >> no > >> >> > >> >> >> >> sense to write changes back. In recent Lucene versions, > >> this > >> >> > >> >> >> >> is > >> >> > >> not a > >> >> > >> >> >> good > >> >> > >> >> >> >> idea, especially for large indexes (RAMDirectory eats > your > >> >> > >> >> >> >> heap > >> >> > >> >> space, > >> >> > >> >> >> >> allocates millions of small byte[] arrays,...). If you > >> need > >> >> > >> something > >> >> > >> >> >> like > >> >> > >> >> >> >> a > >> >> > >> >> >> >> caching Directory and you are working on a 64bit > platform, > >> >> > >> >> >> >> you > >> >> > >> can > >> >> > >> >> use > >> >> > >> >> >> >> MMapDirectory (where the operating system kernel > manages > >> the > >> >> > >> >> read/write > >> >> > >> >> >> >> between disk an memory). MMapDirectory is returned by > >> >> > >> >> >> >> default for > >> >> > >> >> >> >> FSDirectory.open() on most 64 bit platforms. The good > >> thing: > >> >> > >> >> >> >> the > >> >> > >> >> >> "caching" > >> >> > >> >> >> >> space is outside your JVM heap, so does not slowdown > the > >> >> > >> >> >> >> garbage > >> >> > >> >> >> collector. > >> >> > >> >> >> >> So be sure to *not* allocate too much heap space > (-Xmx) to > >> >> > >> >> >> >> your > >> >> > >> >> search > >> >> > >> >> >> app, > >> >> > >> >> >> >> only the minimum needed to execute it and leave the > rest > >> of > >> >> > >> >> >> >> your > >> >> > >> RAM > >> >> > >> >> >> >> available for the OS kernel to manage FS cache. > >> >> > >> >> >> >> > >> >> > >> >> >> >> Uwe > >> >> > >> >> >> >> > >> >> > >> >> >> >> ----- > >> >> > >> >> >> >> Uwe Schindler > >> >> > >> >> >> >> H.-H.-Meier-Allee 63, D-28213 Bremen > >> http://www.thetaphi.de > >> >> > >> >> >> >> eMail: u...@thetaphi.de > >> >> > >> >> >> >> > >> >> > >> >> >> >> > >> >> > >> >> >> >> > -----Original Message----- > >> >> > >> >> >> >> > From: Cheng [mailto:zhoucheng2...@gmail.com] > >> >> > >> >> >> >> > Sent: Sunday, February 05, 2012 7:56 AM > >> >> > >> >> >> >> > To: java-user@lucene.apache.org > >> >> > >> >> >> >> > Subject: Configure writer to write to FSDirectory? > >> >> > >> >> >> >> > > >> >> > >> >> >> >> > Hi, > >> >> > >> >> >> >> > > >> >> > >> >> >> >> > I build an RAMDirectory on a FSDirectory, and would > like > >> >> the > >> >> > >> writer > >> >> > >> >> >> >> associated > >> >> > >> >> >> >> > with the RAMDirectory to periodically write to hard > >> drive. > >> >> > >> >> >> >> > > >> >> > >> >> >> >> > Is this achievable? > >> >> > >> >> >> >> > > >> >> > >> >> >> >> > Thanks. > >> >> > >> >> >> >> > >> >> > >> >> >> >> > >> >> > >> >> >> >> > >> >> > >> > >> --------------------------------------------------------------------- > >> >> > >> >> >> >> To unsubscribe, e-mail: > >> >> java-user-unsubscr...@lucene.apache.org > >> >> > >> >> >> >> For additional commands, e-mail: > >> >> > >> java-user-h...@lucene.apache.org > >> >> > >> >> >> >> > >> >> > >> >> >> >> > >> >> > >> >> >> > >> >> > >> >> >> > >> >> > >> > >> --------------------------------------------------------------------- > >> >> > >> >> >> To unsubscribe, e-mail: > >> java-user-unsubscr...@lucene.apache.org > >> >> > >> >> >> For additional commands, e-mail: > >> >> java-user-h...@lucene.apache.org > >> >> > >> >> >> > >> >> > >> >> >> > >> >> > >> >> > >> >> > >> >> > >> >> --------------------------------------------------------------------- > >> >> > >> >> To unsubscribe, e-mail: > java-user-unsubscr...@lucene.apache.org > >> >> > >> >> For additional commands, e-mail: > >> java-user-h...@lucene.apache.org > >> >> > >> >> > >> >> > >> >> > >> >> > >> > >> >> > >> > >> --------------------------------------------------------------------- > >> >> > >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > >> >> > >> For additional commands, e-mail: > java-user-h...@lucene.apache.org > >> >> > >> > >> >> > >> > >> >> > > > >> >> > >> >> > >> >> --------------------------------------------------------------------- > >> >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > >> >> For additional commands, e-mail: java-user-h...@lucene.apache.org > >> >> > >> >> > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > >> For additional commands, e-mail: java-user-h...@lucene.apache.org > >> > >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > >
