Please review the following articles about NRT, absolutely instant updates that are visible as they are done are almost impossible (even with RAMDirectory):
http://goo.gl/mzAHt http://goo.gl/5RoPx http://goo.gl/vSJ7x Uwe ----- Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: u...@thetaphi.de > -----Original Message----- > From: Cheng [mailto:zhoucheng2...@gmail.com] > Sent: Monday, February 06, 2012 4:27 PM > To: java-user@lucene.apache.org > Subject: Re: Configure writer to write to FSDirectory? > > Ian, > > I encountered an issue that I need to frequently update the index. The > NRTManager seems not very helpful on this front as the speed is slower than > RAMDirectory is used. > > Any improvement advice? > > > > On Mon, Feb 6, 2012 at 10:24 PM, Cheng <zhoucheng2...@gmail.com> wrote: > > > That really helps! I will try it out. > > > > Thanks. > > > > > > On Mon, Feb 6, 2012 at 10:12 PM, Ian Lea <ian....@gmail.com> wrote: > > > >> You would use NRTManagerReopenThread as a standalone thread, not > >> plugged into your Executor stuff. It is a utility class which you > >> don't have to use. See the javadocs. > >> > >> But in your case I'd use it, to start with anyway. Fire it up with > >> suitable settings and forget about it, except to call close() > >> eventually. Once you've got things up and running you can tweak > >> things as much as you want but you appear to be having trouble > >> getting up and running. > >> > >> So ... somewhere in the initialisation code of your app, create an > >> IndexWriter, NRTManager + ReopenThread and SearcherManager as > >> outlined before. Then pass the NRTManager to any/all write methods > >> or threads and the SearcherManager instance to any/all search methods > >> or threads and you're done. If you want to use threads that are part > >> of your ExecutorService, fine. Just wrap it all together in whatever > >> combination of Thread or Runnable instances you want. > >> > >> > >> Does that help? > >> > >> > >> -- > >> Ian. > >> > >> > >> > I don't understand this following portion: > >> > > >> > IndexWriter iw = new IndexWriter(whatever - some standard disk > >> > index); NRTManager nrtm = new NRTManager(iw, null); > >> > NRTManagerReopenThread ropt = new NRTManagerReopenThread(nrtm, > >> > ...); ropt.setXxx(...); .... > >> > ropt.start(); > >> > > >> > I have a java ExecutorServices instance running which take care of > >> > my > >> own > >> > applications. I don't know how this NRTManagerReopenThread works > >> > with my own ExecutorService instance. > >> > > >> > Can both work together? How can the NRTManagerReopenThread > instance > >> ropt be > >> > plugged into my own multithreading framework? > >> > > >> > On Mon, Feb 6, 2012 at 8:17 PM, Ian Lea <ian....@gmail.com> wrote: > >> > > >> >> If you can use NRTManager and SearcherManager things should be > >> >> easy and blazingly fast rather than unbearably slow. The latter > >> >> phrase is not one often associated with lucene. > >> >> > >> >> IndexWriter iw = new IndexWriter(whatever - some standard disk > >> >> index); NRTManager nrtm = new NRTManager(iw, null); > >> >> NRTManagerReopenThread ropt = new > NRTManagerReopenThread(nrtm, > >> >> ...); ropt.setXxx(...); ... > >> >> ropt.start(); > >> >> > >> >> SearcherManager srchm = nrtm.getSearcherManager(b); > >> >> > >> >> Then add docs to your index via nrtm.addDocument(d), update with > >> >> nrtm.updateDocument(...), and to search use > >> >> > >> >> IndexSearcher searcher = srchm.acquire(); try { search ... > >> >> } finally { > >> >> srchm.release(searcher); > >> >> } > >> >> > >> >> All thread safe so you don't have to worry about any complications > >> >> there. And I bet it'll be blindingly fast. > >> >> > >> >> Don't forget to close() things down at the end. > >> >> > >> >> > >> >> -- > >> >> Ian. > >> >> > >> >> > >> >> > >> >> On Mon, Feb 6, 2012 at 12:15 AM, Cheng <zhoucheng2...@gmail.com> > >> wrote: > >> >> > I was trying to, but don't know how to even I read some of your > >> blogs. > >> >> > > >> >> > On Sun, Feb 5, 2012 at 10:22 PM, Michael McCandless < > >> >> > luc...@mikemccandless.com> wrote: > >> >> > > >> >> >> Are you using near-real-time readers? > >> >> >> > >> >> >> (IndexReader.open(IndexWriter)) > >> >> >> > >> >> >> Mike McCandless > >> >> >> > >> >> >> http://blog.mikemccandless.com > >> >> >> > >> >> >> On Sun, Feb 5, 2012 at 9:03 AM, Cheng <zhoucheng2...@gmail.com> > >> wrote: > >> >> >> > Hi Uwe, > >> >> >> > > >> >> >> > My challenge is that I need to update/modify the indexes > >> frequently > >> >> while > >> >> >> > providing the search capability. I was trying to use > >> >> >> > FSDirectory, > >> but > >> >> >> found > >> >> >> > out that the reading and writing from/to FSDirectory is > >> >> >> > unbearably > >> >> slow. > >> >> >> So > >> >> >> > I now am trying the RAMDirectory, which is fast. > >> >> >> > > >> >> >> > I don't know of MMapDirectory, and wonder if it is as fast > >> >> >> > as > >> >> >> RAMDirectory. > >> >> >> > > >> >> >> > > >> >> >> > On Sun, Feb 5, 2012 at 4:14 PM, Uwe Schindler > >> >> >> > <u...@thetaphi.de> > >> >> wrote: > >> >> >> > > >> >> >> >> Hi Cheng, > >> >> >> >> > >> >> >> >> It seems that you use a RAMDirectory for *caching*, > >> >> >> >> otherwise it > >> >> makes > >> >> >> no > >> >> >> >> sense to write changes back. In recent Lucene versions, this > >> >> >> >> is > >> not a > >> >> >> good > >> >> >> >> idea, especially for large indexes (RAMDirectory eats your > >> >> >> >> heap > >> >> space, > >> >> >> >> allocates millions of small byte[] arrays,...). If you need > >> something > >> >> >> like > >> >> >> >> a > >> >> >> >> caching Directory and you are working on a 64bit platform, > >> >> >> >> you > >> can > >> >> use > >> >> >> >> MMapDirectory (where the operating system kernel manages the > >> >> read/write > >> >> >> >> between disk an memory). MMapDirectory is returned by > >> >> >> >> default for > >> >> >> >> FSDirectory.open() on most 64 bit platforms. The good thing: > >> >> >> >> the > >> >> >> "caching" > >> >> >> >> space is outside your JVM heap, so does not slowdown the > >> >> >> >> garbage > >> >> >> collector. > >> >> >> >> So be sure to *not* allocate too much heap space (-Xmx) to > >> >> >> >> your > >> >> search > >> >> >> app, > >> >> >> >> only the minimum needed to execute it and leave the rest of > >> >> >> >> your > >> RAM > >> >> >> >> available for the OS kernel to manage FS cache. > >> >> >> >> > >> >> >> >> Uwe > >> >> >> >> > >> >> >> >> ----- > >> >> >> >> Uwe Schindler > >> >> >> >> H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de > >> >> >> >> eMail: u...@thetaphi.de > >> >> >> >> > >> >> >> >> > >> >> >> >> > -----Original Message----- > >> >> >> >> > From: Cheng [mailto:zhoucheng2...@gmail.com] > >> >> >> >> > Sent: Sunday, February 05, 2012 7:56 AM > >> >> >> >> > To: java-user@lucene.apache.org > >> >> >> >> > Subject: Configure writer to write to FSDirectory? > >> >> >> >> > > >> >> >> >> > Hi, > >> >> >> >> > > >> >> >> >> > I build an RAMDirectory on a FSDirectory, and would like the > >> writer > >> >> >> >> associated > >> >> >> >> > with the RAMDirectory to periodically write to hard drive. > >> >> >> >> > > >> >> >> >> > Is this achievable? > >> >> >> >> > > >> >> >> >> > Thanks. > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> --------------------------------------------------------------------- > >> >> >> >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > >> >> >> >> For additional commands, e-mail: > >> java-user-h...@lucene.apache.org > >> >> >> >> > >> >> >> >> > >> >> >> > >> >> >> > >> --------------------------------------------------------------------- > >> >> >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > >> >> >> For additional commands, e-mail: java-user-h...@lucene.apache.org > >> >> >> > >> >> >> > >> >> > >> >> --------------------------------------------------------------------- > >> >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > >> >> For additional commands, e-mail: java-user-h...@lucene.apache.org > >> >> > >> >> > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > >> For additional commands, e-mail: java-user-h...@lucene.apache.org > >> > >> > > --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
